Host Intro (Gene Kim)

The next talk is from Moira Cheng. She is the transformation program manager at Vodafone, a global connectivity and digital services provider that operates in 21 countries. She is part of an amazing initiative called the One Tech Team initiative, which is part of a plan to add nearly 7,000 software engineers to its expanding European-wide technical workforce by 2025 through a combination of recruitment, reskilling existing employees, and insourcing.

What is so interesting about this initiative is the constituency they are targeting. They want to find 2,500 IT operations staff across the enterprise to fill those positions. In an era that is increasingly reliant upon cloud services and platforms as a service, this is often a group that is at risk of being left behind, or perceived as at risk.

There was something about this initiative that appealed to Moira. She attended this conference virtually last year just as she was starting in her new role. I am so pleased that she flew all the way here to tell her amazing story, because it goes way beyond just training and education. It is about creating a community of Ops practitioners across Vodafone to help each other on so many fronts, including ensuring people's relevance both now and in the future. Here is Moira.

Moira Cheng

Hi, good morning, everyone. I am thrilled to be here. It is my first live Vegas Summit, so be kind. It is also my first trip to the US, so say hi. Welcome.

I will tell you a bit about the story. I work for Vodafone Group as an IT Operations DevOps Transformation Program Manager. I am really over the moon to have been asked to come here by Gene. I presented this in our European Summit this year, which was a virtual session, and Gene thought it would be great if I came here in person and shared the story with you.

Let's start with an overview of Vodafone. Vodafone, as you might know, is a global connectivity and digital services provider, and we have operating companies in 21 countries. Vodafone Technology alone has around 34,000 employees across Digital and IT, our network, and other supporting functions. They include our shared services centers in Egypt, India, Romania, and Hungary, and they operate under Vodafone Intelligent Solutions, or _VOIS for short. We will dive deeper into Digital and IT shortly.

Vodafone serves both consumers and business customers. Most of you might know Vodafone as a mobile connectivity provider, but beyond mobile we offer a lot in the consumer IoT space, Wi-Fi, TV, device lifecycles, and other things as well. Vodafone is the world leader in the Internet of Things, or IoT, business. In the business or enterprise division, we serve public and private companies and customers of all sizes, ranging from small office and home office customers to small and medium enterprises and larger enterprise customers. They include national corporate customers, public sector customers, and multinational companies. Across Vodafone Business, we provide fixed and mobile connectivity options as well as unified communications, cloud, security, and IoT products supported by our global network. We provide services to many industries, including manufacturing, financial services, government and education, retail and wholesale, health, and many others, supporting their day-to-day critical business operations.

Within Digital and IT is our Operations, Infrastructure, and Technology _VOIS function. We develop, deliver, and provide operational support for the IT systems and applications that underpin all our products and services for both our consumer and business customers as well as employees. If we narrow our focus just to European IT Operations, it runs in 12 European countries if you include Turkey, and we also have our _VOIS shared service technology centers in India, Egypt, and Romania. This alone makes up circa 2,500 employees. We also have a central group supporting function, and that is the one I am part of.

I will give you a bit of background about myself. I am not an engineer. I am not a developer. I have been with Vodafone for about 16 years, and I came through the acquisition of Cable & Wireless Worldwide. For a decade or so, I spent my time designing services and managing teams who design how we sell, build, and run our products. You can say I am a bit of a service designer, business architect, or IT solution designer, depending on what industry you are in. We have different terms for what I used to do.

I have collaborated cross-functionally across many teams in Vodafone to bring together our technology, both the network and IT components of the solutions, with people and process, ensuring that we can deliver an end-to-end IT or solution through our internal teams and provide great customer experiences.

It was working through new product developments that I started to see a shift to DevOps about five years ago. One unified communications product platform team started to take that lead. In my role then I thought, hang on a second, you want to build it and you want to run it? It was a completely odd thing back then. I thought, what about Ops teams over there? What are they going to do? You also want to do changes, but you do not want to go through the CAB or the change approval process? Okay, I am going to have to speak to our change team over there and see what they think.

In another part of our IT delivery organization, which developed capabilities and applications for our Vodafone Enterprise employees to use in supporting our products, I was also exposed to another team that brought IT delivery, Dev, and IT Ops teams together. I was really impressed by how they began to cross-skill across Dev and Ops, and they matured to become one full-fledged DevOps team over time. I had amazing colleagues and early adopters in Vodafone who were willing to enlighten me on DevOps ways of working, its culture, and its practices.

Organizationally, in April 2021, Vodafone pulled together all of the Digital and IT teams, with every European market coming under one roof. We effectively became one large IT delivery and operations organization. They called us the One Tech Team.

In a press release in October 2021, Vodafone unveiled its plan to add nearly 7,000 software engineers to its expanding European-wide technical workforce by 2025, through a combination of recruitment, reskilling existing employees, and insourcing. It was in May last year that I was offered an opportunity to change course in my career, moving to the IT Operations team to lead a large-scale DevOps transformation program.

The intent for our program was to build on and further accelerate the insourcing and skilling strategy by changing the way we work across DevSecOps. The program was created for two reasons. One was to ensure that we as IT Operations looked at how we worked and the skills needed for the future. Two was for IT Operations to play a key role with colleagues in other functions to shape how we evolve together.

The aim of the program was to educate, share best practices, collaborate with others, and improve our overall understanding of DevOps. With everyone on the same page, we could work to harmonize and transform to DevOps ways of working across all our European markets and our _VOIS shared services team. We had already learned a lot from DevOps success in the digital space, but we had not learned that much from the traditional IT, monolithic, legacy space. It was timely that I attended DevOps Enterprise Summit Europe virtually in 2021. That was my first one, and I was completely immersed in the world of DevOps. I came away completely inspired.

I questioned myself: how am I going to do this? How am I going to drive a program? I delved into three weeks of intensive research: books, videos, websites, everything around DevOps. But there was no magic bullet answer for what I was aiming to do or achieve. I thought I was up for a challenge, and resolved to either find a way or pave the way.

Research and discovery was the starting point for me. All my previous years of experience in service design taught me that I have to know the problem before I dive into the solution. The first thing we did was a lot of interviews with the local markets and IT Ops teams to understand their barriers to adopting DevOps ways of working.

It became apparent that DevOps was perceived to be something that could only be done with cloud-native solutions, and more prominently only worked with digital front-end developments. We struggled to understand how to apply DevOps to hybrid cloud environments or other traditional monolithic legacy IT solution stacks, and importantly how to integrate DevOps into our existing ITSM processes.

Our very rough analysis showed that our consumer segment was clearly leading the way in digital front-end developments, with the majority deployed on cloud-native solutions, followed by our hybrid IT solution stacks. The enterprise or business front end still had about 50% of our solutions deployed on traditional IT stacks. If you look closely at consumer and enterprise backend BSS systems, the vast majority were operating on traditional IT stacks.

Why was this analysis important? With this data, we realized that IT Operations teams are still in the majority serving and running systems based on traditional IT. We had not yet gotten over the curve of modernization into the cloud as a whole. From a broader perspective, that meant we did not have the ideal technical conditions to enable DevOps, by which I mean the ability to use APIs and CI/CD pipelines, or the conditions to deploy small, contained, low-risk cycles into production.

But I thought, why could we not start applying a DevOps culture, encourage collaboration between Dev and Ops teams, and try some DevOps practices?

The other challenge was a lack of involvement in agile projects. If IT Operations teams in the majority are busy supporting, optimizing, maintaining, and patching legacy systems, and in this case still using waterfall delivery methods, there are fewer opportunities for Operations to be involved and collaborate closely with development teams in agile projects. Late engagement, lack of involvement in PI planning, and lack of involvement in agile ceremonies were common themes. Engagement with Operations often still happened just before going into production. How do I build a culture of DevOps collaboration when we are not presented with opportunities for closer collaboration? Do our IT Operations teams even know what working in agile means?

Looking across our organization, there were glimmers of potential. Those working with digital teams had opportunities to evolve to DevOps ways of working. Some markets were more mature than others. Regardless of technology stack and the lack of optimal technical conditions, some teams and some projects did manage to display the right DevOps culture. They were beginning to collaborate better together, but they were not sharing their knowledge widely enough.

How do we leverage the learnings and experiences of those who are practicing DevOps today for markets and teams that have yet to explore DevOps and its potential fully? We began to shape and scope our program in July 2021. We understood it would be best if we brought our teams on this journey with us, and even better if they did not know about DevOps and learned through the program.

We asked for volunteers from across the IT Operations functions, from all the markets and our _VOIS shared services team. Through interest and word of mouth, we found more people who were experienced in DevOps and wanted to share their knowledge, including from the delivery teams. All we asked was that individuals who had interest in DevOps, no matter their experience, commit some time per week to be part of our program.

We were a zero-CAPEX-budget program, spurred on by the buzz and curiosity of DevOps, but with the joint goal of wanting to build, gain knowledge, and evolve to be future-ready for digital and DevOps. We identified seven workstreams: people, culture, and change; team topologies, organizational models, and role evolution; standards and guardrails; measuring success; operations processes and ways of working evolution; tooling and process automation; and security and operations.

We defined OKRs for each stream and understood the outcomes we wanted to achieve. With our first set of 25 keen volunteers, we held design thinking workshops.

In those workshops, we brainstormed and prioritized suggested deliverables. From the market discovery sessions, we sorted through 63 challenges from the markets, mapped them into our workstreams, and prioritized our top 17 challenges based on importance and feasibility. We brainstormed ideas to see whether the team could come up with new ideas or deliverables we had not already thought of. We mapped all our ideas and deliverables back to the challenges and pain points to ensure we could address those pain points up front. To conclude the design thinking workshops, we did a MoSCoW prioritization.

Design thinking was new to the team. Many of the volunteers, especially those from IT Operations, had not thought about design thinking or known about it. It was also their first introduction to the Mural collaboration tool. It was a new way to approach ideation, brainstorming, and prioritization, and it is not something operations teams come across in their usual day jobs. Those volunteers really appreciated being involved in shaping and contributing ideas for the program, and we became invested in the purpose, objective, and potential outcomes of the program.

On the program structure and delivery method, from the start I intended for the program to deliver using the SAFe agile methodology. It would be a big leap of faith because many of our volunteers in IT Operations had never worked in agile projects before. I thought, if I am bringing everyone on a journey of learning DevOps, why not run this program in agile? Everyone will benefit from the practical experience of working in agile too.

My less optimistic inner voice said: you have no fixed team, you are not creating a product, you are not doing software or technical development, this is more of a cultural transformation program. Has anyone ever tried agile with this sort of thing before? But the optimistic and competitive side of me said that if you know your capacity and the hours each person can commit, it does not have to be technical features. You can think of them as enablement features. If you can size it, then you can do it. Match the volunteers to where their interests lie and where the skill sets fit best. Let's experiment and try. What is the worst that could happen? Fail fast and learn. Based on my years of working in agile projects and calls to trusted agile coaches, I had reassuring support, so I was confident it could be done.

First we needed customers and stakeholders. We decided it was best to have an experienced market in DevOps, an inexperienced market in DevOps, and a _VOIS shared services team. We also needed a core solution team: people who helped define and refine the future backlog, create the solution context, and help with launching, publishing, and supporting queries for whatever we produced. We created an Agile Release Train with three scrum teams, and we asked for volunteers that we thought fit best based on characteristics for each scrum.

We defined our features. We spent a lot of time, two hours a day, four days a week, creating the initial feature set. We had about 140 features, and as in any backlog it keeps growing. We had just finished our one-year anniversary and four program increments.

One of the first things we delivered was a survey. We wanted to find out what people on the ground thought about DevOps and DevSecOps. We sent out the survey, and to our amazement about 1,300 people replied, over half of our whole IT Operations team globally.

At a high level, the survey results showed that only about 30% of those in Operations teams were practicing DevOps today. We asked about applying DevOps culture and ways of working to their daily work, and 52% responded that they knew the theory but were not sure how to apply it to day-to-day work. For barriers, 51% selected lack of training opportunities and lack of knowledge, and 21% highlighted lack of opportunities alone.

We also wanted to see how everyone felt about change in general. Over 90% viewed DevOps as an opportunity, which was great, and over 65% felt psychologically safe and empowered to seek help and learn new things. Those results provided great feedback: individuals needed time to learn more about DevOps and DevSecOps but did not have enough time in their working day; they had not had the opportunity to get involved in DevOps, so they had not had a proper chance to apply the culture or adopt the way of working; and they had a positive mindset, viewed DevOps as an opportunity, and were eager to learn.

From the outset of the program, we wanted to deliver a playbook where we could share and learn experiences on DevOps and DevSecOps-related content. We documented guardrails, DevOps principles, and standards. We took principles from operations, our tooling strategy, software engineering teams' engineering principles, and cybersecurity teams' secure-by-design principles, and pulled them all together.

DevOps tooling was very dispersed. In a global environment, many DevOps teams were using various tools. We documented the recommended tool set for use around the entire DevOps lifecycle, to give everyone a clear vision to harmonize and converge on tooling globally over time.

Then we needed to build on and explain Team Topologies and how it applied to Vodafone, taking into account our current IT landscape. We looked at the enablers for favorable and not-so-favorable DevOps ways of working, so people would be able to recognize opportunities to adopt DevOps.

On processes, we started to understand globally who already had a robust change management process fit for digital and DevOps ways of working. We found there were many workarounds to bypass the traditional change management process. Typical workarounds ranged from logging bulk digital changes in advance or retrospectively, to not logging any changes at all.

We also worked on service enablement and service introduction: changing how we feed through non-functional operations requirements. On incident management, we are defining and creating a new target state for how we want to operate in the future, including automation, AI events, monitoring, and better flows.

We did work on measuring maturity and success. We had a software engineering maturity assessment and an IT operations maturity assessment, and we are bringing them together. We are doing the same thing with KPIs. Delivery teams measure KPIs from an engineering and delivery point of view; operations teams have KPIs. We want to unify that KPI set. The view is that with a unified KPI set, we will be transparent and see where we can improve.

For people training, there is too much content everywhere. Where do people start? We had to create a DevOps learning library and a dynamic learning path, which is in progress. The objective is to stop wasting time with people finding things and asking what to start learning first. For success stories, we are collating stories from those who have done DevOps and sharing them out to the wider teams so we can share those experiences over time.

People collaboration and sharing is the most important thing: getting Dev and Ops teams together, sharing knowledge, and spreading it globally.

We have done some foundation work and have some things down on paper. In the last three months, we have gone to market, as product would say, and launched some of the things in our playbook. What we have published internally and marketed since June this year has received over 12,000 views. But the proof will be in the pudding when people use what we have written and apply it.

One area we want to focus on is more experiments on site reliability engineering. For Operations, that is where they can start using SRE techniques and practices to help elevate and change even traditional IT solution stacks so they can more easily apply DevOps practices.

We have put 40 DevOps ambassadors in place. They are going out to share the good stuff we produced and provide a continuous improvement feedback loop. Whatever is used in our playbook, we would like them to share how we can improve it, and we are improving it over time. It is just a start.

In line with the saying "the more the merrier," this is why I am sharing our early transformation journey. It is a letter from IT Operations, if you will. If you are part of a delivery or development team, I hope it inspires you to reach out to your IT Operations colleagues. If you do not know where to start, just say, how can we start improving our MTTR together? I am sure it will open many more conversations.

If you are an agile project manager or scrum master, please seek out your operations team members and bring them into the project from the start. That would be great, because they need to be involved, and operations requirements need to shift left. If you are a project manager, product manager, or product owner, you might start to see non-functional, service-oriented, or platform engineering features coming through. Please include budget for them.

One year on from the start of this program, some personal milestones: we have come a long way in a short period of time, and I am incredibly proud of the volunteers and their valuable contribution above and beyond their day jobs. I was in India last week with the team at one of our biggest shared services centers. We had a first joint DevOps workshop with our software engineering senior global service line heads, software engineering, developers, delivery, testing, and operations, all in the room together.

We agreed to be one team, to put aside our organizational silos and differences, to break down barriers, and to find opportunities to accelerate the transformation to DevOps. We even agreed to share and loan people across Dev and Ops and Ops and Dev for greater cross-skilling. That is a massive mindset shift and a big step in the right direction. The saying "all for one and one for all" comes to mind. I can only be optimistic about what the next year will bring, and how our program will shift from being not just an education program, but one that also has an arm to drive transformation and adoption of DevOps globally.

Last but not least, we did something slightly differently in this program: something more experimental with the use of design thinking and agile, and how we are transforming to DevOps in Vodafone. I would love to hear your experiences. We are still struggling with how to transform our traditional IT solution stacks and legacy, bringing the old ways of working into the new. If you have insight on that, or on how to bring external vendors on this journey, I would be grateful to hear from you. Thank you all for listening.