Sascha Schärich

Thanks.

Hi, everyone. So how are you doing? Good.

I'm really glad to be here and to talk about what we have learned over the past years on our DevOps implementation journey, which problems we encountered, and how we set up dedicated teams and roles to overcome those problems.

I think we've come up with a really nice setup that could fit other organizations as well, and I'm really happy to share it with you today.

I'm the lead DevOps evangelist for Deutsche Telekom IT, and you have probably all heard of Deutsche Telekom, right? Okay. I mean, we are one of the world's most valuable brands. Number 11, actually, ahead of companies like, or web brands like Facebook, Mercedes, or, sorry, Paul, Home Depot.

Could someone move up the notes a bit? Sorry. Thank you.

And we are the largest telecommunications provider in Europe. To be able to do that, of course, you need some IT, and that's where we come in. I work for Deutsche Telekom IT, and we are the internal IT service provider of Deutsche Telekom. That means we run and develop services and software for all kinds of products which Deutsche Telekom needs to serve our customers.

It also means if we screw something up, then people cannot come into our stores, buy the new iPhone, get their landline connected, or whatever. So we need to make sure that it works.

Here are some nice numbers about us, but I would say the main takeaway on this slide for you is we are roughly 9,500 employees spread across different countries in Europe. The bulk of the colleagues, 5,000 and something, work from Germany. And this makes things tricky because we have, of course, through that, a very German culture.

As you all know, in a DevOps transformation, the technical practices are the smaller part. The much bigger part is transforming the culture. And here, like I said, it can get tricky if you deal with Germans.

Germans love accuracy. They love being on time. They love clear responsibilities. And that's great if you think about German engineering. I mean, just think of the nice cars we're able to build with that mindset.

But if you're trying to implement something like DevOps and establish a new culture, then it will be slow and you will run into problems. We're just not great with change. We love consistency. For a German, it's normal to start the career in one company and then retire, decades later, from that same company.

I'm still referred to as one of the young wild ones, and I've been in Deutsche Telekom since 1998.

So we really needed some effective strategies to come up with. And those things grew over time, and we are trying to focus that on different levels. I'm going to show you today what we came up with.

When we started our transformation, that was in 2017. We decided, okay, we'll use SAFe, Scaled Agile Framework, as our way to implement Agile and DevOps.

In 2018, we started with a couple of pilots and said, okay, we'll take seven, we called them hubs, don't ask me why, which are mainly an agile release train according to SAFe, and test out how to do that and how to then later on scale effectively.

So we took a good mix of what we would encounter later on. The hub I was involved in was responsible for building new products for our business customers. And we did have a couple of advantages on our side.

First of all, we were working in a mainly greenfield environment, which meant we didn't have to take care of any monolithic legacy applications. At the same time, those pilots, they pulled the young ones. So we had a good mindset going already.

Also, we had a high level of management attention on those pilots, which meant we could really react to problems quickly and get escalations and everything which we needed to figure everything out.

Of course, we also had our problems. We had some difficulties. We were still surrounded by legacy processes, and they failed to keep up with the increased speed we were showing them, and just couldn't keep up.

Then there was no real common understanding of what DevOps actually is. If, at that time, you asked 10 people what DevOps is, you usually got 12 different opinions and 15 reasons why it's a bad idea in the first place.

And we didn't have any toolchain. So each of those hubs, for example, the first seven hubs, set up their own GitLab, and everyone had to figure it out themselves. So way far away from DevX platforms or whatever.

But what we did have is we had a system team in those hubs. And as you do according to SAFe, we had the system team set up your toolchain and explain to the people in the hub how to use the toolchain.

That's where I was in and started creating workshops to show the people how to build the pipelines and use those tools. And that's where I fell in love with DevOps, basically, because I immediately could see all the advantages this way of working and this mindset would bring.

But do keep in mind, I was one of the wild ones. Yeah.

So after a while, we were satisfied with the results of the pilots and said, okay, let's scale this thing. Then in 2019, we really started scaling this whole hub thing out. And the number of hubs grew and grew. Now we're at over 120 hubs.

But one of the things we learned in the pilot was that we had a need for a central tooling platform. That's why one of the first hubs we set up was the so-called CI/CD hub, which provides exactly this set of tools.

And this team offered a couple of services, one of them being how to use the tools. And because I already liked talking about DevOps so much, and the people in my hub couldn't hear it anymore, I said, okay, I'll move to the CI/CD hub and teach everyone how to do DevOps.

So we went out, and what we quickly learned in those workshops was the people, at the end, didn't have technical questions like, "How do you add another stage to the pipeline?" or anything, but more of, "Why do you need it in the first place?" and, "What do you mean with mindset and culture?" and so on.

That's where the German engineering, I think, got in our way. They were asking questions like, "Who is going to tell me that it's okay to deploy? And who's going to give me an approval for that change, especially on production? And what do you mean, team is responsible? Which one of us? I mean, we need a name."

So I had already discovered this great community and had listened to talks from Mark or from Mik Kersten and so on, and I knew it was not just about the tools. So I said, okay, we really need to focus on the mindset part and sort of separate that from the technical product.

That's why I became the product owner for the DevOps coaching team. And we said, okay, we'll go out and provide a couple of services for all those hubs.

One of the first ones we did was the DevOps Health Radar, which is based on the DevOps Health Radar from SAFe, from Scaled Agile Framework, which has 16 questions. But what we did is we went into those teams and talked with them through all those 16 questions, and through that got them thinking a lot about what they should be doing and not.

Then, of course, we did DevOps introduction sessions, because we needed to get the mindset into the people. And I must have done about 50 to 60 of those sessions, with sometimes hundreds of people, which was great because it gave me a broad insight into what the German engineers were all struggling with and why they weren't moving that much.

And, of course, we provided team consulting. So that meant we went into the individual teams inside the hubs and helped them figure out their next needed steps, how to put that stuff in the backlog, and make sure that they get it more moving.

But I think the most important thing was we listened to the doubts and the worries of those people, and we provided answers. Some of them I had heard from this community.

In 2021, we decided to add just another layer of complexity to our whole transformation by introducing skill-based chapters. Everyone was assigned, according to his or her primary skills, to one of several chapters. I'm, of course, assigned to the DevSecOps chapter.

The idea behind the chapters was that each one of those chapters drives their individual topic. So for me, DevOps, towards the hubs as well as to the people. And to do that effectively, we said, okay, every people lead in our hub is also a SPoC, or a single point of contact, who talks to six to eight hubs.

That gives the hubs a dedicated person to talk to and ask questions, but also gives us the chance to directly address things towards the hubs.

Again, we set up a couple of services which we would provide, one of them being demand management, of course. So we need to figure out, how many DevOps engineers do you need? Do we have them? Can we send them to you?

But also things like skill management and figuring out the ever-changing need of the skills that the hubs need, and then making sure that the chapter can also provide this.

But again, I think the most important thing we did is we talked to the hubs and we listened to the hubs. We gave them ideas. We gave the business owners inside the hubs new ideas and new impulses to think about doing DevOps better.

So now we have the teams covered by the DevOps coaching team, and we have the system teams inside the hubs doing that as well. And we have the hub level nicely covered by the SPoCs, which also, because they're also people leads, we have the individuals covered.

But what about leadership?

One thing we noticed, for example, in 2022, was there were a lot of external influences which really risked the success of our transformation. One of them, of course, being the war in Ukraine, which made it necessary for us to relocate a significant number of people out of Russia in a very short period of time.

And these were people who were working on really innovative things, which we really needed for the future. And we didn't just have the skills or the capacities to ramp that up in other countries.

Now, if things get difficult, the normal human reaction is you fall back to the old proven habits. You discard changing much, especially something as intangible as culture.

At the same time, we also stumbled over things like having the right metrics or proven patterns, which I have heard from this community, and where I had a possible solution. But nevertheless, we still ran into those problems.

Now, I was already known in our company as the face, or you could say the beard, of DevOps. And many people were listening to me. But how would I get the IT leadership team to listen? How could I inspire them to try out the things I heard in this community?

Fortunately, there were a couple of colleagues from Budapest who also wanted to keep this whole DevOps thing pushing and didn't want it to fall off the table all the time. And we said, okay, we need to do something about it.

So we pitched to the leadership team that we would like to implement what we call a DevOps expert council, and I'm really happy to say they listened to us. They saw the need and said, okay, do that, set that up.

So what we now have is the DevOps expert council. We are strongly connected to the IT leadership team, but as well also to the chapters. And we have people who work part-time in the expert council and part-time in the hubs to make sure that we have a diverse mix of input.

Again, we provide a couple of services. First and foremost, of course, we're the change agents for DevOps. We align all the different DevOps initiatives, or at least try to, because there are many of them, which is good, but still difficult to align.

And we do other things as well. One of the things we do is we foster external exchange with all kinds of companies. For example, in the last year we did Lufthansa DevOps Days, where we got 25 people from Lufthansa together with 25 people from us and spent two days in a nice environment and talked about all things DevOps. That was really cool.

We also do regular exchanges with companies like Hermes or Siemens, and if anyone else wants to exchange with us, reach out to me.

One thing we're also doing at the moment is trying to implement SRE, which seems to be a bit difficult at the moment for us, but we're convinced that SRE will help us. So at the moment, the expert council is also working on doing this in a pilot phase, similar as we did in the hubs, to try this out and see how it goes.

The last service we provide at the moment is we have set up what we call the DevOps maturity score. And I know you're not supposed to measure maturity and so on. You're supposed to measure with DORA metrics, but we're just not ready there yet. Yeah.

So the DevOps maturity score is simple. Seven simple questions which touch all kinds of areas which you should be doing in a successful DevOps transformation: things like culture, collaboration, automation, business interaction, or metrics.

And each of those seven questions has five predefined answers, which makes it easy for the hub owners to answer those questions without having to rely on some nerdy DevOps engineer to tell them what to put into the form.

The idea behind it is that we really are trying to raise the awareness with those hub owners, to make sure that they really are thinking about the right things, and putting it as a priority into their backlogs, and making sure that this happens.

So this is our setup which we came up with. Like I said, we are trying to do this culture thing on different levels. We noticed this will help us probably, and we think it really works.

We even think it works so good that, oh, sorry, that was the maturity score, but I'll skip that. We think it works so good that I have a nice quote which I brought with me from our SVP for Operations in DevSecOps chapters. And he says:

"We aimed to create better software and reduce time to market through DevOps. Yes, we now release software nine times faster, but the profound cultural shift was even more impactful. Being a large telco headquartered in Germany and burdened with legacy is not an excuse on not doing DevOps."

So basically, if we can do it, you can do it.

Thank you.

But, of course, we're never finished. The work of a change agent is never done. So we think we have a setup in a good way. But still, the more we collect information about our organization, the more we still see we have a lot of open issues.

One of them being, for example, the flow metrics. We think they would really help us in finding out more about our organization and the bottlenecks. But unfortunately, I have not managed to excite the critical mass needed to really start implementing this. But I'm working on it.

In general, we do have a big distrust around metrics. So we are currently introducing the DORA metrics as a mandatory measure for the hubs. And quite a few people have not really understood the logic and reasoning behind them and are afraid that these metrics will then be misused in one form or the other.

And, of course, the biggest thing we are still struggling with is the huge gap between business and IT. I mean, sure, we could, of course, keep sending anonymously Mark Schwartz's book A Seat at the Table to them, but there has to be a better way.

And that's why this is my ask to you. If you have a similar organization and are struggling with the same problems, and maybe you have already figured out how to get your business involved much better than we do, and you know how to deal with legacy mindsets, maybe you have set up a dedicated team, as you see, I love setting up teams, then please do reach out to me.

I think this community has helped me so far. A lot of the things I have in our setup came from ideas which were sparked in my brain from going to conferences like this. So I'm pretty sure someone may have a good idea for me.

And with that, I want to thank you for listening. It was an honor to give something back to this community, because I learned so much. And I hope this really helped to inspire and bring someone new ideas to accelerate your DevOps transformation.

Thank you very much.