Hey, welcome. This is, uh, Topo Pal from DevOps Enterprise Summit, 2023 Amsterdam, and we have an excellent guest today with, uh, Asha. Sasha. I don't know, I'm gonna mess up your name anyway, so I'm, that's fine.

Is that fine, Sasha? Yeah. Okay. Awesome.

And, uh, before we start, uh, why don't you just, you know, take a few minutes and, and talk about yourself, like where you are from and what your focus is on what you have been. So if you can start with that. Sure. So, um, I'm, I've worked for Deutsche Telecom it, um, as you can see with the fancy magenta colored stuff.

Um, and I've been working at, uh, telecom since 1998, so for a really long time. Uh, over the, the years I've done operations, I've done system administration, I've done all kinds of stuff. But in, I would say 2017, 2018, I found my real passion, which was being a DevOps engineer. Um, and I found it so good that I had to start talking about it more and more.

And now I'm the DevOps evangelist for Deutsche Telecomm. I t. So, uh, in terms of size, how big is your IT or your impact area? Yeah, so Deutsche Telecom is the service provider for, um, IT service provider for Deutsche, um, Deutsche Telecom.

It is the service provider for Deutsche Telecom. Uh, and we are roughly 9,500 people, uh, all over Europe, uh, mainly in Germany, but also in in other countries like, um, uh, Hungary, uh, Slovakia and so on. So 9,000 engineers. I mean, that covers, uh, just developer operations or Yeah.

Do they include cyber security people too? Some security, sure. Okay. Um, as well.

Um, but yeah, mainly DevOps and, and delivering whatever the, the customer, so Deutsche Telecom needs from us. Yeah. So I saw your talk yesterday. Mm-hmm.

It was awesome talk and, and viewers, uh, you need to actually see his, his talk Yeah. From yesterday, uh, on the, on the recording, it was just fabulous. So I'm, I was very impressed, but on the talk, I have a few questions. One is, uh, what made you go the DevOps route, if you will?

Uh, was there any incident that kind of changed your mind was a pain point? If you can, if you can describe that. Sure. I mean, for, for the company, the decision was made in roughly 2017 to go Agile and to work with DevOps.

And we said, okay, we will use Safe as, as a scaled Agile framework, um, as our way to implement that. And of course, if you do safe, you sort of should be doing DevOps. You can do DevOps without safe, but you can't really do safe without DevOps. So we said, okay, we need DevOps.

So that's the, the, the company view of why we went there. Uh, and then why I immediately wanted to be involved in that is because of the stations of my career I did before. Uh, I noticed, okay, this DevOps stuff is really gonna help us with a lot of problems. I saw the, the strange on-call duty weeks you had as an operations person getting stuff thrown over from the developers and so on.

Um, the end-to-end view on, on topics. And that's why I said, okay, we, we really need this. Not only because we need it for safe, but because it really will make the life of the company and of the, the individuals in the company much better. Awesome.

So, so you are talk yesterday about, was about scaling DevOps, but before scaling, there's a start. Mm-hmm. So, so, so if you can tell the viewers, uh, how did you start, what did it take for you to start? What kind of support did you need across from across the enterprise to mm-hmm.

To start? Yeah, and luckily we, we, from the get go had the, the sort of the support from the management level because they said, okay, let's do DevOps. Uh, and as I said, we decided we'll use safe scaled Agile framework as our framework to work and to figure out how to scale that over the 9,500 people. We have, um, we set up a first set of pilots.

So we, we took seven topics, uh, from our company and put them into what we call hubs, which are mainly sort of release trains according to safe. Um, and tried to figure out how to do that in those hubs in the first place so that then we would learn, okay, what do we need to then effectively scale that? Because now in the meantime we have 127 hubs, which range from sizes of, I dunno, 50 people to 150 or more. Um, and so we really wanted to make sure that before we scale that on that massive level, we start step by step.

And what we found out along the way is that, um, we needed to have some, some stuff in place to, to help us with that. Um, because as is is, as the, this community knows, DevOps is not about the technical practices, it's about the mindset and the culture, and that's where we, we had our biggest problems and that's why we said, okay, uh, we need to figure out how to address that step by step as well to help us with the scaling. So, uh, you talked about, you started on, on selecting few topics. So would you be able to tell us like what exactly was done or was it the C I C D pipeline?

Was it changing the, uh, you know, culture changing, the team construct? No, when the, the topics we chose were basically either applications or topics we were doing for our customer. So for example, I was involved in, in a hub, um, we were doing, creating new products for the business customers. Um, and so the customers, Deutsche Telecomm wanted to have that anyway, and we said, okay, we will try that in an agile way, uh, with all the things you do with MVPs and, and so on.

Um, and tho that's, that was one of the seven topics we chose, um, to then yeah, figure out to do that. And to be able to do that, of course, you need pipelines and you need the structure and, and other things around it. Uh, which is what we found out during the pilot that we would need central tooling. For example, we needed a central supplier for all the, uh, tool chains.

Uh, because with 127 hubs, you don't want all of them figuring out where to get a GitLab and how to run it themselves. Um, and that's where the whole idea of, okay, we need one central instance for this started. Yeah. Okay.

Because the tool selection made you go in that centralized, uh, tructure. Now, the, the initial topics or the applications that you chose, uh, during that time, was there any selection process or did you, what, what are the criteria, uh, used in the selection process? Well, for those, for those first seven hubs, uh, we decided to, to take different topics, um, which sort of represented all of the things we would encounter later when scaling. So we took, um, topics where there was a lot of legacy monolithic applications and put that into one of those seven hubs.

Um, but as I said, with the hub I was involved in, we also took stuff which would we could be able to do on Greenfield, um, where we didn't have to worry about monolithic applications, just about the still surrounding legacy processes. But, um, that was the idea that we, this, these seven hubs sort of represent roughly what we would encounter later on. Yeah. And, and, and what was the duration, um, in the, in the first set of applications where you actually started something and declared victory?

How long and, and, and what did it take actually to get there? Well, the, we, we started with the, the decision to, to create these hubs and as these pilot hubs and to choose the topics was done end of 2017, and then pretty much beginning of 2018, we started to staff them. Uh, and then also those, we, we grew those over time. Uh, like I said, for some of them, we, we used stuff which was already there and which we were already doing, and moved that into a safe structure.

Um, and for others, for example, the hub I was in, um, we, we drew people together and created new teams and, and new ways of as new, new pulled in new people to set that up first. Um, and then we, the, the pilots ran for about, I would say half a year, three quarters of a year, um, before we said, okay, we seem to know what, what we could be getting into if we start scaling. So we were comfortable enough to say, okay, now we really ramp out and create more and more of those hubs. Um, all of the hubs had to follow a process, which we set up also as a learning out of the pilots, uh, where we said, okay, you need to have your value stream clear.

You need to have some documents ready. You needed sort of a checklist, um, before you could, with your application, which was usually in some major department in the past in an organization, say, okay, we move towards a hub. Um, yeah, that's, that's where we really started then with the scaling. Yeah.

Um, I picked up on, on a, on a couple of wards first this checklist. Mm-hmm. So, uh, for the, for the companies that are actually struggling with scaling DevOps, uh, I think they'll be benefited if you can spell out some high level things in that checklist. Mm-hmm.

Like, these are the things to look for. Well, for the, um, for the initial hub creation, um, there were a couple of things. Like I said, one, one I mentioned was the value stream because of course it makes sense to organize a hub or release train according to a value stream. And we didn't have that yet widespread because we were still organized in departments and main departments and with people who got money from the company and the, from, from the customer side, so from Deutsche Telecom and then said, okay, I need that, that many people to deliver this and this project, not really products either.

Um, so there was a lot about that of, of making sure that really what you want to do with the hub is sort of a value stream, which really generates something, um, and which is moving more into a product way and not, not so moving away from the project project side of things. Um, that was one of the things in the checklist as well. You had to, of course, as there was some stuff in there for making sure you had the right financing for your hub, of course. Um, but like I said, these value stream thing I think was the most important, um, to make sure that these hubs were not just copies of all departments, just relabeled as safe.

So the head of the department was named the business owner and so on. So, but really that these were really hubs who, who then work in an agile way, uh, with MVPs, with, with, uh, yeah. Increment planning and, and all the things which you really need instead of just the project waterfall thing. No, uh, from the checklist and the po uh, pilots or proof of concept and all that.

Uh, what was the plan of scaling it out? Uh, um, and, and, and was there any criteria set as as, as successful p o C or pilot, uh, before you moved onto the, the, the scaling stage? To be honest, not really. Okay.

Because the plan was always, okay, we will do safe for the whole company. Got it. Um, uh, so we just took the time to use these pilots to figure out what we would encounter when scaling. Uh, but it was always clear we were gonna scale and move everyone towards safe.

Um, but looking back now, um, I mean now we're, it's four years later, or five years later even, uh, we noticed, okay, maybe we shouldn't have gone with Safe for everything because we still, we were, at the moment we're finding out that for some organizational parts, it doesn't make sense to really use safe, uh, especially if they just, um, deploy services for our customers and so on. So there we are moving a little, a little bit away from Safe, but from the beginning, the idea was, okay, everyone will do safe and then it will all work, which of course it never does in reality. Right, right, right, right. Yeah, I mean, these are, that's why you do pilots to see where you can apply and where you can and, and, and that, that's good.

Now, um, what is the timeline between, uh, the first pilot successful, or first set of pilot successful and scaling out across the whole enterprise? Um, I would say for the, for the, we started the scaling after about three quarters of a year, A year maybe, uh, after we set up with the first pilots. Um, and we sort of finished with the whole scaling, so the last couple of hubs, um, were maybe done two, three years later, so did take quite a while. Um, and now that after we then had all, all the 9,000 feet people sort of sorted away into different hubs, uh, then of course came the phase of, okay, do, does this hub setup really make sense or do we need to merge two hubs together because they're working on the same topic or basically the same value stream?

Uh, or do we need to separate them because they're together in a hub, but they don't really interact with each other and it doesn't make sense that they do that? So maybe we split that up into hub. Um, so that's the phase we're in right now to figure out, okay, first of all, the, the hubs we have, the safe release trends we have now, do they make sense as they are, uh, working together? And like I said as well, um, does it make sense to use Safe for all of those topics anyway?

No. Okay. So, uh, in the, in the scaling process, was there like a standard pattern for scaling out, like, you know, business unit by business unit, or a product by product, or was there any learnings from the bottom up as well as direction from the top down taken into account? Yeah, tha thankfully there, there was both as the, the top up and the bottom as bottom, bottom up and top down approach.

Uh, both was there, um, because we had a lot of teams who were enthusiastic about that and said, okay, we, we want to do DevOps and we want to work in that way. And at the same time we did, of course have the support from management because they understood, okay, if we want to do this Agile transformation correctly, then of course we need DevOps for it. Um, where it got a little bit fuzzy maybe was in the middle, um, because those were the people who were then of course sort of losing control and losing their authority and so on. Because you also, at the same time with doing Agile, you make the whole organization a little bit flatter.

So the ones in the middle, they were struggling a little bit, um, some more than others. But yeah, I think right now we're at a rather good balance of having the, the support from the bottom and as well having sort of the, the wish and the strong urge from the management to, to really do that. Um, Yeah. Uh, uh, I don't know if you'll be able to explain, but, uh, I picked up on, uh, the mill layer losing controls.

Mm-hmm. So can you name some of the controls that that, that they lost or the thought that they lost? Well, first of all, as I mentioned before, we, we had these huge large departments, uh, where the department head was sort of the king of that department. Yeah.

He had the money. Uh, he decided, okay, what will we do? He talked, he was the only one talking to the customer side, and they gave him money to do sp special things or certain things. Um, and then you said, okay, then I'll decide how to do that.

Uh, and if you, if you do agile right, then you lose this control because you want of course, more people to talk to the customer, customer to really figure out what they need. Um, you, you move away from these requirement papers to more doing MVPs and yeah, agile as working in an agile way and agile being able to, to react to changes what the customer wants as well. Um, and they of course don't really like that. Yeah.

So we, like I said, we did see some, some, some tries of making sure that somehow we map the roles of the department to the safe, but those don't really work because of course, the work of a business owner, for example, and safe or of a product owner or a product manager is something totally different than it used to be a head of department somewhere. So, so in, in your transformation, um, you know, the, the, the whole Dave Ops transformation, so it's like Dave and ops then, and then stuff in between. Like did you face any friction from the people in between the Dave and the ops or around Dave and Ops? Sure.

I mean, one example would be we had a lot of testers, uh, who did the manual testing and they were not happy when they heard about, okay, we wanna do DevOps, so that means we will do a lot more automatic testing, uh, testing built into the pipeline and so on. Because they were, of course, understandably a little bit afraid of their job, um, which we sort of started to get rid of when we helped them to explain them, okay, we still need your knowledge on how to do testing. We just don't need you to type it in. We need to explain it to a deaf guy so that he can build it into his pipeline or, and then it gets run automatically.

Um, so, but that was one of the things we were struggling with. And then, uh, what, what I mentioned earlier, the, the thing of, we were still surrounded by a lot of legacy processes, um, meaning they, there was a lot of stuff which we had to figure out how we could do this faster. Uh, workers' council approval, for example. Um, if, if you constantly want to roll out new stuff, then you need to make sure that you get this much quicker.

Um, and they were not used to the speed. We had a time to market of 18 months, and now we were suddenly going down and down and down to six months, three months, and so on. So they, they had to adapt their processes as well. They were just not used to the speed.

Um, so there was also a lot of that we had to figure out to make all the, apart from dev and ops to make all of them aware. But that's where I always kept saying, okay, the DevOps mindset is not just something for the dev and the ops guys. Everyone in the company needs to have this mindset and understand the culture and understand what we're going for, uh, so that they all then can come up with ideas and processes to support this speed and way of working. Yeah.

So, so talking about speed, you, you mentioned like, you know, it used to be 18 months release cycle, and then it got down to, what is it right Now? I think right now we're a little bit under three months, under three months on average. Mm-hmm. Yeah.

So That's a huge improvement. Right. Did anybody got scared about the speed and what did they say? Like, A lot.

I mean, as I mentioned the talk, no, we're Germans, as of most of us, Deutsche Telecom, it are Germans and we're afraid of, of change of instability and of speed and so on, because that tends to break stuff we love, we love when everything is stable. Um, and so increasing the speed made people afraid, of course, because they, what, what do you mean? In three months we can set up a product and put it on production. We, we need to do this first and check this, and this said, yeah, we, we do those checks automatically and then no, get, get approval from workers' consult example while we're doing it already, not just when the product is finished and so on.

Um, yeah. But people were really afraid and didn't know how to cope with that. Yeah. So, So in, in terms of speed, so the, so let's say before all this transformation, you released X amount of features in 18 months.

Mm-hmm. Now is it the same x amount of features being released in every three months or, or is it X by six? It is, uh, unfortunately, I, I don't have exact numbers, but it did increase. Yeah.

That's what we found out, because in these 18 month cycle there, we had big release containers, we had three of them each year, everything was bundled and then we could release Yeah. A certain amount of features. And what we did find out is yes, we're not only releasing more frequently Yeah. But we're also releasing more stuff.

That's true. Because the interesting thing was the idea was maybe, uh, if we, if we de decrease the, the amount of the time we take to release, then maybe we can release sort of the same amount, uh, in inside a hub with, with maybe less people so we can reassign them to other topics. But what we found out is this reassigning didn't happen. The hubs stayed in their size, but they released more stuff.

So, um, yeah. In the end, we, we release more, uh, and faster at the same time. Yeah. But like I said, no exact numbers we could have.

Yeah. So, uh, and, and, uh, last question about scaling and then, and move on the next topic. In the scaling, if, if there are companies who are not doing safe, um, uh, or they have some other kind of structure, what would be your suggestions for them to scale out without the safe construct? Mm-hmm.

In, in general, I think it doesn't matter if you safe or any other organizational setup thing. I think what what did help us was doing these pilots first, uh, sort of like you do the software development in a DevOps way of doing an M V P first and finding out, okay, what works, what doesn't work. As I said in the pilots, we noticed we need some central tooling pipeline because all of those seven hubs had to figure that out themselves. And that doesn't make sense.

Um, and so I think doing a pilot first helps to find this balance of, okay, uh, you need the individuality of each of those groups or teams, uh, to choose what they want to, to see what they want and, and to maybe try out new stuff. But at the same time, you need sort of the, the possibility still to scale and to make sure that they somehow align with each other, uh, so that you don't end up with 127 hubs who all have a different tooling chain. Yeah. Because then how do you get compliance over that, and how do you make sure they align with each other?

Yeah. Right. So, uh, now that you're in a good position, what are your sort of problem areas that you are focusing on these days? Mm-hmm.

Well, one of the things I, I would say is we're, we're still struggling here and there with, um, the value stream as, so what is the value stream? Do we have the hubs cut correctly? Uh, do they really all work on the same value stream? Uh, or do we have everyone who's working on value, one value stream together in one larger team?

Um, that that's something. And at the same time, I think what we're really working on at the moment is, uh, visibility. So making sure we see more where we are with all kinds of metrics. Um, so we're, we're rolling out Dora metrics at the moment, for example, uh, which is also, again, a tough sell for Germans because they think they're being controlled if you give them, give the Dora metrics to someone.

So, um, we need to make clear that it's not about control, but it's about looking at what's going wrong and then seeing what we can do about it. Um, so metrics, uh, and, and the overall view, that's something we're really working on at the moment in which we could probably be a lot better. Yeah. How about, uh, uh, security and, you know, you are regulated company.

Yeah. So in terms of regulators and external auditors, do you have any pushback or are they on, on board? And again, security would be one of the processes I mentioned, which couldn't, can't, couldn't, and is starting to, but really is still struggling to keep up with the increased speed, um, because of course we need, uh, security and we need, uh, compliance and we need all those regulations and, and checks that, that's totally clear. Um, but we need those approvals to be much faster than they were in the past.

Um, because in the past it took you a really a long time to get an approval to put a new product outside on the, in the world. Um, and now we need to make that faster, and there are things going on. So we are, we're, we have some approval processes, for example, where we are starting to do automation for them, where then the idea would be in the end that it's just a part of your pipeline, which then already checks a lot of the security compliance stuff. Did you do this?

Did you use this image? Are you deploying to this in this environment? Because we know that's safe and secure, um, so that we do that. Um, but yeah, that's, that's still something we're, we're figuring out, I would say.

Okay. And, and do you have, in your discussions or in scaling process, do you have, uh, security in the, uh, do they have a seat, uh, on the table, or are they still watching from outside Parts and parts? As I said, 127 hubs. So it, it, it gets brought.

There are some who are better at already making sure we have security directly from the beginning built in, sort of, yeah. And there are others who are struggling more. Um, and especially also the security managers. We have a lot of those still.

Like I said, they also have to figure out, okay, how, how does this new, new, I mean, DevOps is not that new, but how does this work? How can we adapt to that and how can we deliver the speed that these guys need? So, so overall, what are your goals for the next one year and the next three years? Where do you want to be?

After one year and three years? As, as I mentioned in my talk, I, I think the, as of that, we now have this, the, the, the DevOps maturity score as a overview of where we are with our transformation. That's the first step. So I would say, all right, right now my next goal would be for us to really increase the, this, this, um, this overview and this, this observability of where we are with our transformation, with our metrics and so on, so that we can really see where we're struggling.

Um, and then of course, as soon as we have that, and with the maturity score, we have started to do that, then the, the next step, of course will be to increase the score. Yeah. Um, but for that, first we need to get away from, we have a gut feeling of a guy who talked to all kinds of hubs and knows roughly where we are to, okay, we have these and these numbers of where we are with our transformation, and now we can try this and this to see if it increases or decreases the numbers. So, um, a few things around, uh, this conference I have seen you previously in, in, in other, uh, this conference in Las Vegas, uh, last year.

So, uh, how do you like this conference so far, and how does this conference help you in any way, if at all? Uh, it's, it, it's been so great, uh, and, and it really is, is a really great conference. And I mean, this is, uh, my, my fifth summit I'm attending, uh, the first three were virtual, of course, thanks to Covid. But the, the last two being in person is so great because as I mentioned, all, all of the things we came up with to transform our mindset and our culture, they were all inspired by stuff I heard from all of the great presenters, which are here.

Um, so all I usually after those conferences, I come, came back, uh, to my desk at home and said, okay, we need to do this. We need to try out this, we need to come up with this. And then of course, reality sets in of, okay, probably will be difficult to do exactly that in our company. But then you start to adapt.

And then in the end we came up, I think with a pretty good view of how we can address this maturity and, and the mindset on those different levels. And I owe that to this community. That's why I, again, I'm also so happy that I can give back to this community because yeah, it's, it really has helped us and it's, it's really this giving and taking thing which I love about this community. Um, uh, if you have some suggestions for this conference, uh, you know, like what would that be like, how to improve on the content or the format or anything else?

Uh, really, uh, uh, there's not much I would change. Um, maybe because having had the experience that you have to fit in so much of what you learned in 20 minutes, making the slots a bit longer, but I know of course that then gets, gives you fewer slots so that there's always has to be a balance. Sure. Um, but apart from that, I really love the way you can interact with all the speakers.

Um, I, in Las Vegas, it was so great for me to come up to people, to speakers like you yourself and talk to you and, and get direct listen to, to the experts. Uh, and at this conference now, I had the experience of being a speaker myself, and it was so great that people come up and say, Hey, this really, uh, was a great idea and I'll try that out. And, and, and can you explain more on that and that, so this, this interaction between the speakers and the, the, the participants that's really great and that, that we did, they should definitely keep. That's awesome.

Thank you so much. Thank you, Joshua. So, so, so this has been awesome. Thanks, Tim.

I, I look forward to seeing you in the next, uh, version of DevOps Enterprise Summit. Uh, and good luck with your transformation. Thank you, man. It's a pretty good story.

Thank you very much.