Host Intro (Gene Kim)

Good afternoon, everyone. By show of applause, how has the day been for you?

Fantastic. All right, the first speakers of the afternoon are from Deutsche Telekom. Sascha Schärich is a DevOps engineer, and Örs Cseresnyés is a senior chapter coach and vice president. Deutsche Telekom is the largest provider of broadband and mobile in Europe, and to make this happen, they require nearly 10,000 technologists.

I met Sascha at the DevOps Enterprise Summit last year in Las Vegas while in line at the Starbucks the morning before the conference began. I got to meet him and his team and learn about the amazing work that they've been doing inside their organization. I was delighted to hear how he was so influenced by the talk that was given from the team from Virgin Media O2 in the prior year.

One of the reasons I'm excited for this talk is how Sascha and Örs have been able to start changing the ways of working in a company that is famously conservative in work culture, and is engaging scores of teams across the organization to improve their ability to deliver value to their customers. So here's Sascha and Örs.

Örs Cseresnyés

Thank you. All right. Hello. It's great to be here. Today we will share with you our story, our story about changing the minds of about 10,000 people at a large German corporation, Deutsche Telekom. My name is Örs Cseresnyés. I work as the leader of the IT operations and DevSecOps chapters, and I'm joined by Sascha, who is our tireless DevOps evangelist at Deutsche Telekom. Thank you.

If you think about DT, Deutsche Telekom, you would probably think about a German telecommunications company, right? In terms, it's correct, but we are also much more than that by now. It's much more than Germany. Of course, we are represented in many of the European countries and also in the US, and it's much more than a telco. We define ourselves as a technology company. Transforming into a technology company has been majorly contributed by the internal IT of Deutsche Telekom. That's where we work.

If you just look at our numbers, who we are, you would see quite impressive numbers. I always find it practical to explain what we do by just explaining what happens if we mess up. If we mess up, you walk into a shop in Germany and you can't buy the new iPhone. Or if we mess up, the technician who would go to your house to connect your fiber connection can't do his work. Or if we mess up, then you just can't contact our service desk or call agents. As you put it, fundamentally, we can just kill the basic communication services too. You understand it's a lot of really critical systems that we develop and operate.

Our journey begins in 2017. In 2017, it began with this picture. We drew this picture in order to explain our ambitions to our own staff and to our leadership. This was an era where we had everything on prem, big monoliths. There was basically no sign of DevOps yet. In our organization, we set ourselves a couple of goals. Of course, you will find the usual ones: we want to move to CI/CD, we want to decouple, we want to move to microservices, we want to have stability and scalability by leveraging cloud providers, and we would like to have an organization that is flexible enough that supports everything.

Besides all this, for us probably the most important part was that part in the middle, which is the culture. We wanted to have people working for Deutsche Telekom IT who would own what they built, who would share what they built, who would be proud of their metrics because they know where they would like to develop themselves, and we wanted to move into a fully DevOps-minded bunch of people.

Are we there yet? No, we are not there. We have made a lot of mistakes on the way, but I think looking at what we have pulled off in an organization this size and in a legacy culture like Deutsche Telekom is, I think it's really interesting for some other companies too. So on how we did that, I hand over to Sascha.

Sascha Schärich

Thank you very much, Örs. First of all, it's really great to be here and to share our experiences with this community, because for a lot of the things we came up with, I sort of got the idea from exactly this community. It's great to finally be able to give something back.

As you all know, in a DevOps transformation, just a small part is about the technical practices. The much bigger issue is changing the mindset and changing the culture of the company. Here we may have an even bigger challenge than others because, as you heard, we're mainly a German company, which means we have a very German work culture, and that does tend to bring difficulties with it.

As Germans, we love accuracy, we love being on time, we love clear responsibilities, which is great if we talk about German engineering. Just think of the nice cars we are able to build with that mindset. But if you're talking about changing something as fundamental as culture in an IT company, then things start to get tricky. We're just not great with change. We love consistency. For us it's normal to start your career in one company and then decades later go into retirement from that same company. I mean, I'm often referred to as one of the young wild ones, and I've been at Telekom since 1998.

We needed to come up with effective strategies on different levels, and these grew over time, to make sure that we get this culture change going. I'd like to show you what we came up with.

As Örs mentioned, in 2017 we started with our agile transformation by deciding we would do SAFe as an implementation to do our agile and DevOps ways of working and to figure out how to do that effectively. In 2018, we started and set up the first so-called hubs, which are basically a release train according to SAFe, which we chose to reflect what we would encounter in the company overall. With these seven hubs in the pilot phase, we wanted to learn how to do and scale effectively.

The hub I was involved in was responsible for creating new products for our business customers. From the get-go, we had a couple of advantages on our side. First of all, we had really good starting conditions. We were mainly in a greenfield environment, which meant we didn't have to take care of all those crazy monolithic legacy applications. At the same time, these pilots attracted young people, so the spirit was good in that team as well. We had a very high level of management attention on these first seven hubs, which really helped us if we ran into problems to solve them quickly.

Of course, we also had our difficulties in that time. We were still surrounded by legacy processes all over, and they couldn't keep up with the increasing speed we had in those pilot hubs. Then we really didn't have an understanding of what DevOps is yet. If you asked at that time someone what DevOps meant for him, if you asked 10 people, you usually got 12 different answers and 15 reasons why it's a bad idea in the first place. We didn't have any central CI/CD tools, so each of those hubs had to figure out where to get a GitLab from for themselves, for example.

But what we did have in those hubs, at least in the hub I was in, was a system team. I was in the system team. I set up the toolchain for this hub, and then I started teaching the people in the hub how to use this toolchain, so basically how to do the technical part of DevOps. That's where I really fell in love with the topic, because I could immediately see what advantages DevOps brings, not only to the company, but to the individuals and the teams as well. But do remember, I was still referred to as one of the wild ones.

In 2019, we really started scaling. We were satisfied with the results of the pilots and started really rolling out more and more of the hubs. One of the things we had learned in the pilot was, of course, that we would need a central toolchain. One of the first hubs we created was the so-called CI/CD hub, which provided exactly that toolchain. One of the teams in that hub also offered a service, which was teaching the people how to use this toolchain again. Because I already loved talking about DevOps so much, I moved into that hub and started telling even more people about DevOps.

What we quickly learned in those sessions, though, was that at the end, in the Q&A part, people didn't ask about technical stuff. They didn't ask, how can I add another stage to the pipeline? How can I automate this? What they really asked about were things like mindset and culture questions: Why do we need it in the first place? That's where the German engineering started to get in our way. They wanted to know who would tell them it's okay to deploy, or who gives me the approval to put something on production, and who's responsible? What do you mean, team? Which one of us? We need a name.

So we really needed to figure something out and do that. Fortunately at that time, I was already starting to get involved in this community, and I had heard people like Mik Kersten or Mark Schwartz talk and knew that DevOps is not just about the tools. It's more about the mindset. That's why we said, okay, we need to focus on that and do something dedicated for the mindset part. That's why we set up the DevOps coaching team. I got to be a product manager for that team, and we set up a couple of services to help us.

The first service we set up was the DevOps Health Radar sessions. DevOps Health Radar is basically a tool from SAFe, which goes through 16 questions over the whole development life cycle of a product, and where you can see how the team can improve in those areas. But the added value we brought was we did four-hour sessions going with the team through these questions, and getting the teams to talk to each other inside the hub, and getting the people in the team also to talk to each other. That was really valuable. People loved that and kept booking that. We really had to figure out how to do all those health radars.

Another service we offered was DevOps introduction sessions, to get down from the 12 different opinions to maybe five. They were also hugely popular. Suddenly everyone knew, okay, ask the bearded guy about DevOps. I must have done about 50 to 60 of those sessions to sometimes hundreds of people.

But I think the most important service this team still offers today is team consulting. We went out, we talked to the teams, and we helped them figure out what their next steps would be in their DevOps transformation. Then of course we made sure they also put it in the backlog.

In 2021, we decided we'll add another layer of complexity to our overall transformation by introducing skill-based chapters, which meant every employee was assigned to one of roughly 20 chapters based on his or her primary skills. The vision behind this chapter approach was that the chapters would drive the individual topics into the company and also towards the people. I'm, for example, in the DevSecOps chapter, which means we are of course responsible for driving the topic of DevOps into all the hubs.

To be able to do that, we said every people lead has to be what we call a SPOC, or a single point of contact, for that topic. Each one of the people leads has roughly six to eight hubs he's assigned to. That gives the hub a dedicated person to talk to when he has questions about DevOps, which there are a lot of. At the same time, it also gives the chapter organization a possibility to address topics around DevOps to the hubs.

Again, the services the SPOCs and the chapter provide are, of course, demand management, which we try to do in a collaborative fashion. Not just, bring me five DevOps engineers, but more of, what do you need? Maybe you'll be better off with two DevOps engineers who know how to automate better. And skill management, of course. By talking to the hubs, we need to figure out what kind of skills they need and then make sure that we as a chapter can provide those.

But again, I would say the most important thing that we do is helping the hubs to figure out how to do DevOps. Really talking to the hubs, we make sure that they have enabler stories in their backlogs, that we give them impulses to think about, and again, that we make sure that they really put that in the backlog.

Now we have the teams covered by the system teams, by the DevOps coaching team, and we have the hubs covered by the SPOCs, so the management of the hubs especially. But there's still one area missing where they could also sometimes use help, which is, of course, leadership.

In 2022 was a really tough year, as you all know. There were a couple of external influences which really impacted our ability to complete the transformation. One of them, of course, being the war in Ukraine. It made it necessary for us to relocate a significant number of our workforce, over a thousand people, out of Russia as quickly as possible. These were people working on very innovative topics, which were very important to our business as well, and we didn't have the skills or the capacities in other countries to just take that over.

What happens in difficult situations, of course, is people tend to fall back to their old ways of working and to their proven habits. One of the first things that goes over the table, of course, is changing much, especially something as intangible as mindset or culture.

I was already known as the face, or you could say the beard, of DevOps in our company, and many people were already listening to me. But how could I also get the leadership team to listen to me? How could I make sure that I could inspire them as well to come up with new ideas and try out things which I'd heard, for example, out of this community?

Fortunately, a couple of colleagues from Budapest had the same feeling, that we wanted to keep pushing our DevOps transformation, and they were also just as frustrated as me that DevOps kept falling off the table. That's why we did a pitch to the IT leadership team and said, okay, guys, we need the DevOps expert council.

I'm really happy to say they listened, they understood what we wanted to do with it, and they supported the idea and said, sure, go ahead, staff that.

We made sure that this DevOps expert council is strongly connected to the IT leadership team, which you can see because Örs is here as well. But we also made sure that it's connected to the chapter and to the hub organizations as well, because we really need a varied input for our team.

Again, there are a couple of services we provide. One of them being, we are a change agent for all the DevOps initiatives we have out there. They usually talk to me about it anyway, so why not streamline that, put them together, and make sure that they know of each other and so on.

What we also take care of is making sure that we have a constant external exchange with other companies to learn about DevOps. One of the things we did, for example, last year was we set up a joint Lufthansa-Telekom DevOps Day, where we got 25 people from us and 25 people from Lufthansa together for three days to talk about DevOps. That was really cool. We also do regular exchanges with companies like Hermes. We've talked to Siemens, we've talked to the European Space Agency, which is not as cool as NASA, we heard yesterday, but still. We talk to a lot of external companies to get input from them as well.

At the moment we're setting up an SRE pilot to figure out how to implement site reliability engineering, and if that would fit in an organization like ours with all the Germans. But the first service we set up, which is also the one I'm most excited about, is we said we wanted to have a DevOps maturity score.

The idea behind the DevOps maturity score is that we needed really an overview of where we are with our transformation. I had a gut feeling because I had talked to so many people, but we really needed to have something measurable. It was not just enough to count pipelines or use DORA metrics or something because we still have so many legacy systems and legacy applications. They will never probably really have a pipeline in GitLab, so we wouldn't count them, but of course they still need the DevOps mindset.

So we came up with this questionnaire. It's seven simple questions, which all touch on different areas which are part of a successful DevOps transformation, like culture, collaboration, automation, business interaction, or metrics. Each of the seven questions has five predefined answers, which on the one hand immediately show you the way of where you should be going and what you should be thinking of in that area. This makes it easier for the hub owners themselves to fill out this questionnaire and come to the score without having to ask any nerdy bearded DevOps engineers to do it, which we hope will then raise the awareness of what they really should be thinking about and what they should be prioritizing in their backlog.

At the end, you get a score between zero and five, which maps you to one of five maturity levels. I'm happy to say we started the data collection around two months ago. Up until now, 47 of our 127 hubs have responded, and our overall score at the moment is 2.9 over those hubs, which puts us in the defined stage, shortly before measured.

This is great. Apart from confirming my gut feeling that we were on a good way, we can now have something to measure, and now we can come up with ideas how to improve that score even further and repeat the measurement to make sure that we're moving in the right direction. I'm really happy that we have this overview.

Now that brings me to my ask to this community. As you saw, I drew inspiration and ideas from all of you for the setup we have now of the teams and services we provide for changing towards DevOps, but there is one part really missing, which is kind of an important part, which is the business side.

As you know, transformation is not just a thing of IT. Business has to be on board. We are really looking for inspiration on how to do that. If you have a similar organizational setup as we do, if you maybe have your business side aware of your DevOps transformation, maybe you have a dedicated team setup for that, then please do reach out to us and let us know.

We could of course keep sending the book from Mark Schwartz, A Seat at the Table, anonymously to the business side, but there has to be a better way for that.

With that, I want to thank you. I'm very happy that I could give something back to this community. I'd love to hear if this inspires you to set up teams in your company. Please reach out to me over LinkedIn or email, or just talk to one of the people with the magenta-colored stuff. Thank you very much.

Host Outro

Thank you.