A DevNetOps Journey – How We Brought DevOps to Our Network Team
There is so much more to infrastructure and network today than ever before. Our Network has evolved and changed from physical routers and switches to a programmable Network (Virtual Machines, Containers, Cloud, IoT, IaC, NETCONF, CIDC pipeline, AI/ML and APIs). This disruption has led to a need to merge Network Engineers with Application Developers to help bridge the gaps between the infrastructure and development worlds and become one team. A DevNetOps team that is a fully engaged and empowered team marching toward one goal and desired outcome – to create new opportunity for the business and enhance the customer experience.
We’ll discuss our journey of recruiting developers to our infrastructure world and creating a win-win situation where our network engineers learn DevOps best practices and think in term of Automation to solve problems, automate workflows, network management and monitoring tasks; while on the other hand, how to help developers see the opportunity to apply their dev skills and DevOps mindset and tools, to ensure infrastructure availability, performance and resiliency.
Want to know more about our DevNetOps Journey - come join us!
Chapters
Full transcript
The complete talk, organized by section.
Hoda D Alshami and Mike Leuzinger
Hoda Alshami introduces herself as a technology product manager from Nationwide. Mike Leuzinger introduces himself as chief architect for Nationwide.
Mike explains that about two years earlier he was leading Nationwide's network services organization and had to update the network strategy for the growing complexity of cloud connectivity, work-from-home connectivity, and increasing expectations of near-zero downtime. They knew the updated strategy needed to focus on automation, infrastructure as code, and DevNetOps practices.
Although the organization already had scripting-style automation, it had stagnated in moving toward a more modern DevNetOps capability. Mike had an open manager role and a couple of open engineering positions that would normally be filled by people with strong enterprise networking backgrounds. Because Nationwide already had many excellent enterprise network engineers, they decided to use those open roles to hire people with more application development background to jump-start and accelerate the DevNetOps effort.
That resulted in hiring Hoda as manager of what was then called the network monitoring team and giving her open positions to fill with developers. Mike says the rest of the presentation will use a question-and-answer format to walk through the journey Hoda and her team have been on for the past 18 months.
01What drew Hoda to the network monitoring opportunity
Mike asks Hoda what drew her to the opportunity, noting that before managing the network monitoring team she had been a developer and one of Nationwide's first Scrum masters and Agile coaches.
Hoda says the opportunity and the challenge drew her in. She had spent more than 14 years in application development and had worked with different AppDev teams bringing DevOps and Agile principles to them. Even in AppDev, those principles were not one-size-fits-all; she had to work with teams so the changes made sense and so people could see the value. She wondered whether the same would be true in infrastructure and operations, specifically networking, and decided to try it.
She says Mike did a great job promoting the job to her. In their first discussion, when she reached out to understand the position and expectations, he helped her understand how the role would help her and what was exciting about the activities needed to adopt DevOps and Agile. That triggered her curiosity. The first thing she did was search for basics like the OSI model, infrastructure as code, switches, routers, vendors, and related networking concepts.
02Assessing the team and defining the capability
Mike says he hired Hoda partly because she was much more familiar with DevOps than he was. He asks how she assessed the situation as she learned networking, and what her vision was for the team and capability.
Hoda says that when she joined, the two things Mike mentioned, team and capability, were not really there. It was more like a group of great engineers. They were SMEs working in silos, with a one-to-one relationship between each tool and the network engineer working on it. They were heads-down doing operational and admin work, and there was no crossing of boundaries between them.
At that time, capability was not a commonly used term. People talked about tools: a tool for network configuration management, a tool for network performance or availability monitoring, a tool for packet capture. Her first activity with the team was to put everyone in one room and write on the whiteboard that they were not going to talk about tools anymore; they were going to talk about capabilities. They laid out the tools they had and the capabilities those tools brought to the table.
That session helped the team see how much overlap existed among tools. It also showed that they had great tools they were not using to their full capability because only one engineer worked on each tool. Features were not implemented because people did not have enough hours in the day to do operations, administration, tasks, and new feature work. From her first month, Hoda's vision was that they needed to be one team working together, streamline their tools, and get the best out of the tools they already had.
03Evolving from monitoring to one team and one backlog
Mike asks how that vision evolved over what he knew would be a long, multi-step journey.
Hoda says it had been 18 months, but it sometimes felt longer. When she joined they were the network monitoring team, responsible for monitoring. They then evolved and moved to a product-centric model, rebranding as network management, with observability and automation in scope.
She worked with engineers to define what automation meant to them, what they needed to automate, and how to build an automation backlog. She reached out to other teams in the network organization, such as campus and firewall teams, and asked them to put any automation ideas or manual processes that technology could help automate onto cards on the board so the team could discuss them. They started gathering ideas from the whole organization about what automation meant.
As the backlog grew, they needed to execute. They had great engineers doing scripting, but they lacked a development mindset. Their scripts were more like serial tasks rather than object-oriented development. So Hoda decided to add the first developer to the team. That was a critical decision because she wanted someone helpful, humble, self-driven, and able to teach, coach, bridge the gap, and be on the same page with the engineers.
That first developer became the backbone for the automation team. As the backlog continued growing, Hoda added more people, including entry-level developers, Tech Elevator candidates, and fresh college graduates: people willing to learn, grow, understand networking, and make a difference. She was transparent with them that the position involved ambiguity and many hats: designing requirements, coding, testing, and not staying in an I-shaped specialty where someone only does development, testing, or requirements analysis. They had one team, one backlog, and everyone worked together to execute it.
04Guiding people through significant change
Mike asks how Hoda guided a team that included very tenured network engineers and newer early-career developers through significant change.
Hoda says it was significant change because she was bringing DevOps, Agile, and different ways of working to the engineers. The first thing she wanted to create was a friendly learning environment. She always said no question was a bad question and no idea was a bad idea. In design thinking, requirements work, and brainstorming, any idea should be put on the table. That environment helped people open up and share recommendations and ideas.
She also wanted to lead people into the learning zone, because everyone, including her, was moving away from the comfort zone. The team would learn together. Active listening was important. Network engineers needed to listen to developers because developers knew new technologies and might have smarter ways to do the same work. The goal was not to automate the same process unchanged, but to optimize it, create the workflow, and then automate it.
Developers also needed to listen to network engineers because those engineers knew the environment, the customers, and which activities were high risk. They could advise where to start with lower-impact tasks. Everyone was learning new things, so Hoda worked to help people acquire the learning they needed to be successful.
Nationwide had a program with a community college, and she was able to send a couple of network engineers there to learn basic DevOps, containers, Python scripting, object-oriented programming, and CI/CD pipelines. After six months, they came back with enough knowledge to talk with developers using the same terminology. Hoda also gave developers a crash course in networking and infrastructure so they could understand discussions with network engineers. The learning environment and training helped people feel they were part of the journey and that the change was not a risk that would take their jobs away. They were all one team.
Hoda says there was pushback. A common objection was that DevOps worked for AppDev, but networks and infrastructure were different and DevOps was not built for them. She tried to avoid killing ideas on day one. She referred to a learning model: try something, give it time, and if it does not work, tweak it so it becomes helpful. The most important thing was seeing value.
For example, they did not apply standups exactly as daily everyone-says-yesterday-today-blockers meetings. They tweaked the format so people saw value in the daily 15 minutes. She also used the ADKAR model to introduce and sustain change: assess the current state, create awareness of why DevOps and Agile were needed, create desire around developers joining and an automation backlog forming, identify what people needed to learn, and then sustain the change so they did not revert to old habits. That included changes in tools, upskilling, and process: work intake, prioritization, presentation, visual management systems, and movement from planning to execution.
05Tools, people, and process
Mike asks Hoda to walk through the tools, people, and processes that changed.
Hoda says DevOps is the magic. Many people think about DevOps from a tools perspective: tools for planning, coding, version control, testing, and operating. That is true, and teams need to learn and master DevOps tools to build the right environment. But DevOps also creates the mindset and culture of shared responsibility, transparency, and faster feedback when network engineers and developers work together and own work across the lifecycle. It is no longer silos; it is one team working toward one goal. That is where DevOps helps most: building the DevOps mindset rather than only adopting tools.
She says the cross-functional team is the magic word. Developers are experts at writing code and are accountable for working with the DevOps team, but network engineers also need help thinking in terms of automation and using technology to address the problems they are solving. Network engineers are the subject matter experts with deep experience and environmental knowledge. They need to work with developers to gather requirements, design, define user acceptance testing and acceptance criteria, build customer feedback, and feed the loop.
Because the team was introducing many changes while the backlog was growing, people still had operational tasks and keep-the-lights-on work. That created ambiguity. Leaders had to set clear expectations and priorities, including weekly priority conversations about which work mattered most because of due dates or urgency. Having a Scrum master or technology delivery professional helped because that person could manage the backlog, groom the work, build metrics and dashboards, facilitate Agile cadences and retrospectives, and help all roles work together as a healthy DevOps team.
06The pipeline toolchain
Mike asks Hoda to talk more about the pipeline toolchain.
Hoda says the pipeline has an interesting story. The first time she introduced CI/CD pipelines, she got pushback: there was no way they could do this because it was networking. People asked why continuous integration was needed. In AppDev, 15 or 20 developers might access the same codebase and need a mechanism to continuously integrate changes and avoid conflicts. But for a switch configuration, a network engineer might access it once a day or once a week. It felt like unnecessary hassle and extra work. Continuous deployment got similar pushback: they were not going to drive network changes daily or hourly; they were not Netflix.
Hoda knew there had to be a way, so she revised the question. She asked what the best way was to introduce the idea because they needed an automated way to audit changes to the environment. They discussed it from the task perspective: what activities happen today to bring a change into the environment? In the current state, a technology engineer designed the change and decided when to implement it. A change management record was created and approved. Then it was up to the engineer to deploy the change.
When production issues happened, it took time to determine who made the change, where the last version of the configuration file was, and what the prior state of the environment had been. A delivery pipeline or change pipeline using DevOps tools could help.
The first step was version-controlling configurations. They put configs in GitHub. Then they looked at the configs and saw they were more like Word documents. They asked whether they could create templates. After researching Jinja and Velocity, they chose Jinja as the standard. They converted configurations into templates and created a mechanism to render data and templates into actual configs.
Once they had actual configs, they added testing: sanity checks and static analysis. They made it a requirement that no config could go to production before passing that step. They then added required peer review because human eyes were still needed and not everything could be caught in unit testing. Peer review also became a knowledge-sharing and knowledge-transfer step: a senior engineer would approve, and a junior engineer would also review and learn.
They looked at the production readiness checklist that engineers used manually and asked what could be automated and placed into the pipeline. Then they used existing tools to automatically deploy changes to switches and routers. After deployment, they asked what should be monitored, what post-validation was needed, and what telemetry should be extracted from the environment to ensure nothing was broken. When things still went wrong despite due diligence, they used the feedback to update sanity checks, peer review steps, or the production readiness checklist. All of that built continuous improvement into what they called the change pipeline.
07Successes and challenges
Mike says the amount of work Hoda and the team have done over the last 18 months is unreal. He asks what she sees as the biggest successes.
Hoda says the toolchain helped them a lot and was a great exercise for looking at the tools they use today and future adoption. Connecting with other technology stacks in the organization let them ask what they could learn and leverage from other areas.
One challenge was showing progress because they were doing foundational work that takes time to build. It takes time to build mindset and time to see the fruit from that work. They needed to show leadership that they were moving and making progress. For example, templatizing documents took months to come up with the idea, choose the tool, and execute, even though a show-and-tell demonstration might take only 15 minutes.
Mike asks what challenges Hoda sees ahead.
Hoda says they need to keep everyone motivated because it is a lot of work. They also need to keep people tuned in. Because she built the team organically, they now have great skills, but if someone changes positions she may need to start over again. They need to keep people motivated, help them see the value in the work, and keep showing progress.
Mike says the team's energy is noticeable, even though people are not in the hallways as much with remote work. Hoda says she now gets many requests from people who want to join the automation team. Mike says that is a good thing, and they close by thanking everyone.