Etelvina Melo Pires

I'm really delighted to be here with you for this DevOps Enterprise Summit in London. So as Cornelia said, I'm Etelvina Melo Pires.

I'm first a happy mother of three beautiful girls, but I've also dedicated my last 15 years working for Orange. Today, I'm acting as an Agile and DevOps change agent within the DevOps transformation program launched for Orange France IT.

Bruno Meniere

Hello, I'm Bruno Meniere. I'm in charge of DevOps deployment in the whole Orange group. I'm federating the different DevOps journeys in the different business units with the support of local transformation agents, including Etelvina, since two years.

So as you can see, I'm French, so I hope that you will appreciate my French accent.

Etelvina Melo Pires

So we hope that you will appreciate also how we are cultivating DevOps at Orange.

As you may know or not, Orange is a worldwide company, a communication service provider present in 29 countries. And believe me, it was a real challenge for Bruno and I to select the essentials of our diverse journey.

Because even if we are all animated by the same values and the same goal, in each business unit, we have to deal with different soils, different cultures, local markets and business, specific legal constraints, and labor rules.

But before drilling down into the details, let's start with a brief history for you to better understand our context and Orange's ambition for 2020.

So, a brief history. The roots of our company are in French telecommunication research centers, founded in the '40s. Then in the '80s, with the creation of France Telecom, a utility company, we wanted to provide fixed telephony for all in France. Then in the '90s, with the launching of mobile telephony, first in France, then in UK, we used for the first time the Orange brand name.

So as you can see, transformation is in our DNA.

A brief overview about our business. We have 154,000 employees all over the world, serving over 265 million customers all over the world. So we provide fixed telephony and mobile telephony for all, but also a lot of other services like TV services, IoT, education, healthcare, and now bank services.

And in this world of complexity, IT services are more than ever crucial to fulfill Orange's ambition stated in our strategic plan called Essentials 2020, started in March 2015: be a digital, efficient, and responsible company that offers an unmatched customer experience.

For you to better understand what stands behind "crucial," let's take two examples. First, let's go to Africa, in Mali. If IT supporting Orange Money services is down, it directly impacts 30% of the gross domestic product.

Everywhere, if IT supporting mobile services is down, it's immediately several tens of thousands of mobile devices that are partially out of service. Seriously, how long will you be able to survive without your smartphone? One hour? Ten minutes? Maybe less for the most addicted of you, and I'm sure we have plenty in this room.

So clearly, no unmatched customer experience without a modern, agile, and efficient IT. So cloud solutions, agile, and of course, DevOps practices and tools are the three main enablers we've identified to reach that goal. All our CIOs across the whole group are committed today to make them a reality.

Bruno Meniere

So how did we start?

The transformation leaders, the IT managers, and other managers communicate massively and constantly about the why and the what.

About the why: we want to provide top-quality services to customers. We want also to reduce our structural costs, and we want to ensure reliability and security for more than 5,000 IT applications. And also, we want to influence positively the life of our 12,000 IT employees over the world.

But the what: we want to reach a collective agility with everybody using agile and DevOps practices and tools in the group. So for that, we use supporting teams with coaches and skill centers.

But we all know that vision without action plan doesn't make people move. So this is why each business unit manages the how and defines its own roadmap to gradually target the whole IT, often accepting time as an answer.

Etelvina Melo Pires

So, in this journey to DevOps, and while preparing this presentation with Bruno, we wondered, what do we have all in common?

Because in each business unit, taking into consideration the existing culture, we observed at least two kinds of implementation strategy. Huge transformation program led by a strong guiding coalition, as it was chosen, for example, at Orange France, or in other cases, a more viral bottom-up approach.

But we found out preparing this talk that we are all using the same agile cultivation principles. First, engage everybody around compelling reasons to make the change. Then decide where to invest first. And without surprise, we've all started with business-critical and customer-facing applications. And then try and foster continuous learning. And then, finally, expand.

Bruno Meniere

So we are now applying these simple cultivation principles in 10 countries: in Africa, in Europe, in Middle East, and five more by the end of the year.

So it means that we have to face different maturity levels. Some teams are on the road to DevOps. Some others are completely discovering what are agile and DevOps.

So we chose our fight and started small. We selected some challenge projects as diverse as possible, where DevOps can provide an appropriate answer.

At the moment, we have 45 pilots as first seeds where we start DevOps transformation. The goal was to capitalize on them, optimize the learning, speed up the hatching, and be able to pollinate to the other projects.

To speed up this learning, we experiment in three main directions: processes, tools, and organization.

We've experienced new organizational models. We settle such kind of incubators called labs, where biz, dev, and ops teams are joined from the very beginning, experimenting specific agile frameworks and adaptive governance using open space technologies.

Three conditions to be onboarded in these labs. First, motivated teams. It's absolutely mandatory. Involved managers. And the last but not the least, business owners convinced and ready to invest at least a few tens of thousands of euros on transformation actions.

These teams receive very strong support from dedicated agile or DevOps coaches, technical support from our numerous skill centers, but also more informal support coming from our 56 communities of practice that emerge around DevOps.

So clearly, thanks to this virtual cross-functional organization, we've demonstrated that these DevOps product teams are the most efficient model.

Concerning the tools, we let project teams select the tools that they need. For sure, they can reuse existing tools provided by our self-service DevOps store, but they can also introduce new ones when it's relevant. And when we select a tool, it's not a definitive choice. We pivot without mercy when feedback justifies it.

And so project teams can rely on our cloud and API program.

Concerning the process, we have simplified many processes. For instance, in France, for project management and release management, we have made more than 74 evolutions to align practices between DevOps and dev and ops, and to avoid redundancy.

And concerning the CAB, the CAB was considered as an obstacle for continuous deployment because it was a go/no-go decision with the usual process.

So, for the CAB, we made three main evolutions. First, we delegate to the project team the change management authority. Second, we set an API for change records. And the third, concerning the risk analysis, we use now a deep learning tool to help us.

So now we can consider that CAB is no more an obstacle for DevOps, but even leverages DevOps effects.

Etelvina Melo Pires

So these experiments showed us the first traps to avoid, as Bruno just explained. But they also feed us with great success stories. Let's take a couple of examples.

For example, at Orange Labs and Services, our teams are now able to create a completely new platform in our private cloud in less than one hour, deploy an application on top of it in less than 10 minutes, with 100% of the tests fully automated.

At Orange France, where we are reinventing the customer journey in our new smart stores, time to deliver the new version of a mobile app used by our customer advisors was divided by three, while ensuring 99.99% of availability using containers and so on. So we have containers and Docker containers in production.

The last but not the least, we've reduced from three months to three days the time needed to update one of our major and most critical mass-market product offers.

So we capitalize on these wins. We standardize and industrialize reusable solutions and practices when economies of scale, but taking care also not compromising emerging initiatives.

Bruno Meniere

To make DevOps become the rule in the whole IT in 2020, we are now increasing the number of teams who are initiating their DevOps journey.

So to reach this 2020 objective, we need a massive training program. But first, what you have to know is that in our group, each of us has the dedicated time to improve his skills. So it's really useful.

We launched this massive training program to deploy agile and software culture. We trained 20,000 IT and business employees. And also, we trained 1,600 people to DevOps. So we are really proud of this success because it was an objective for 2018, and it was reached by the end of last year.

Also, concerning the operation team, we made a specific effort to train them to the software development tools, to make some close teams between dev and ops. And now our new objective is to have more and more T- or pi-shaped people to empower the DevOps teams.

Etelvina Melo Pires

And in addition to trainings, for teams who need help to initiate their DevOps journey, they also can rely on, well, I think today we are almost 100 DevOps coaches.

Our coaches are used to organize dedicated working sessions involving all the stakeholders, in particular the business owners. First, to assess the ongoing maturity level using great homemade tools such as DevOps Matter, and to make sure that everybody is aware of it. And second, to select and sow the first seeds needed to trigger the step-by-step transformation path.

As an output, these teams define themselves their iterative action plans to reach agreed DevOps level.

But for us to ensure continuous learning, all the teams are also invited to trace and share all reusable and proved solutions as well as all their impediments. This way, we aim at creating a common knowledge base that will allow us to continue capitalizing, but also identify very quickly what are the major issues, in particular when scaling.

Bruno Meniere

Okay. When scaling, funding becomes one of the first impediments. Because when we started pilots, funding was not really a problem because we worked with early adopters. But when we are scaling, for sure, business owners ask for results.

So we have to demonstrate return on investment for each iteration. And also, we have to convince business owners to fund the DevOps actions. For automation actions, for sure, it's not so difficult to explain. But for the other actions, for instance, like refactoring, it's more difficult, so we have to convince.

And also, we need some coaches to help the different stakeholders to forecast the budget allocation to start this transformation.

Each time we need to make some trade-offs with customer value, maintainability, and improvement of the project to make the project happen.

So at the end of the day, when we find these solutions, budget is no more an obstacle. We find some solutions, and it's completely integrated in the new budget plan.

Etelvina Melo Pires

Yeah. And it's always cool when we finally find an agreement with our business owners. But user-centric approach pushed by our business owners is leading us to fund value streams instead of projects.

And believe me, it requires a significant change in our stage-gated, siloed process, in particular the way we manage OpEx and CapEx. And to tell the truth, today we are still wondering how to make it efficient in an agile and DevOps environment.

So if anyone in this room has a brilliant idea to speed up this change, please raise your hand.

Really nobody?

Don't be so shy.

Really? Well, Cornelia, I think we will have to come back next year.

Bruno Meniere

Okay. There is a problem concerning funding, because at the moment for operation, we have some shared resources. And because we consider that for our strategic project, moving to DevOps, we need dedicated operation teams, so we need to find the right budget for this.

So it's difficult, and we have to manage priorities between projects. And also, to find this budget is quite difficult because at the same time, we have some structural cost reduction plan. So, quite difficult.

Another point, because funding is not the only concern that we have. We have also to find the right balance between bottom-up and top-down actions.

In one of our business units, we made an analysis based on the DevOps maturity assessment on the different projects, and we discovered that more than 80% of the brakes are due to structural or organizational causes. So it means under management responsibility.

So with the managers, we made a solution-focused work group to find the right solution. We selected two or three solutions, and then we deployed these solutions with all the management line.

And at the same time, we asked the different managers to support their team in this transformation because it's really necessary.

So it means that in our context, combining top-down decision with bottom-up proposals is really necessary.

Etelvina Melo Pires

And really, we need both to change the habits and anchor them. It takes at least nine months. Easy to remember, especially for a woman: a pregnancy.

So it could be more when teams are not co-located. And to be transparent, it is our routine. So for us, it simply implies to fit out all the equipment and tools to erase the distance. But anyway, willingness to communicate and regular physical face-to-face meetings really remain keystones. Sorry.

Bruno Meniere

So this point is quite simple, but we have three other much more tricky.

For instance, concerning the relationships with our external suppliers, we want to introduce a win-win relationship, and we want to share our DevOps transformation with them. So it means that we have to reshape our contracts, not to decide only on the pricing basis, but also to introduce some criteria concerning the agility and DevOps maturity. So for instance, some measurement concerning the velocity of iterations.

The second point concerns the teams, because in our organization, usually when a product or a service is commercially launched, we dissolve the team. So it's quite difficult to maintain the team and to have long-lasting teams, and we are sure that for DevOps, it's the most beneficial.

So at the moment, we have to find some local solution with local managers with the help of some coaches.

The last point, but maybe the most important for us, is that we have to manage a complex transition phase. Because we have chosen to have an iterative and incremental approach, we have to manage different maturities in the different subsystems.

So when we want to launch a new product, you have to combine these different maturity levels. And for that, we have the support of local coaches, skill centers, self-service DevOps store, and also a strong sponsorship from the management line. So thanks for them.

Etelvina Melo Pires

And as you can see, we have still a lot to do to be resilient to bad weather and even to thunderstorms. And these thunderstorms happen when teams only focus on automation, forgetting fully or partially sharing culture or metrics.

Well, in a technical company as Orange, honestly, automation is the easiest and the fastest part of the game. And once the deployment pipeline is settled, we can create the illusion that the DevOps objective is closed.

But I remember one of our B2C critical applications. Several production streams started working in parallel. Pressured by the business, each production line only focused on its own objective, starting neglecting technical stories and testing effort.

For sure, they increased their delivery rate. But at the end of the day, the ratio of deployments generating major incidents in production was 11% higher than the standard one. So it completely destroyed customers' and ops' trust.

So what did we do? Simple things. First, we made sure that every production line understood the entire system, the entire value stream. We reinforced relationships between biz, dev, and ops, and we made sure to eliminate crappy work.

You're laughing.

So despite all our efforts, till today, features delivery beats DevOps actions, leading the team to only focus on automation to the detriment of the other pillars.

What we observe is when some ingredients are missing, such as involvement, alignment, sharing metrics, in 30% of the cases, it's just a simple failure.

Bruno Meniere

Okay. Maybe some of you are a bit frustrated because we don't speak a lot about tools. But as you can see, tools are not our highest challenge. The highest challenge for us is human, because we don't want to let anybody on the roadside.

So we work with HR department and worker representatives to find the answer to the main questions that we have. One of the main questions is: what will be the impact on my job, on my activities, on my organization?

So we have to find the right answer and to provide them to the teams. For that, we have the support of 100 local ambassadors finding local solutions.

Etelvina Melo Pires

So day after day, teams are gaining confidence and knowledge. Trust and respect are built on a daily basis between biz, dev, and ops teams, working on common subjects in the field with agile leaders supporting them.

All of this fosters autonomy and innovation, and innovation is the final outcome we are looking for.

So DevOps is changing the way we interact, the way we budget, the way we welcome success and failure. And finally, all of this will change our culture. We are sure that this new culture will allow us to retain our best talent and capture new ones.

For sure, it will take time, but the roots of these new trees are deeper and deeper anchored into the ground.

Well, after just 18 months of DevOps journey in so large organization, a lot still has to be done. But we are moving forward step by step, and the more we learn, the more we share, the more we are getting closer to the organic stage.

So now we have to scale what works fine for IT teams to the entire company, because we are all convinced that we are all on the right way to transform, every day, technology into progress, as innovation only makes sense if it serves human beings.

Bruno Meniere

Okay. So before leaving, we would like to send a special thanks to the team helping us to prepare this presentation today, including Stephanie. You are present.

So a new French word: merci for all of you. And a special thanks also to Jean-Pierre Bonafeu, also called alias Hamucho, for those who will search on the Internet. He scribed all the presentation, so a big thank you to him also.

Etelvina Melo Pires

Okay. And because we still have a lot of remaining open questions, we will be happy to discuss with you after the talk. So thank you.

Bruno Meniere

Thank you very much.