Jerome Tassel

So we're going to talk to you about our experience of doing Lean Agile DevOps in a big enterprise for the next 30 minutes. I'll do 15 minutes. Jason will do the next 15 minutes.

For those who don't know about BT, hands up if you don't know about BT. Good. But I'll still go through the slide. So BT is one of those horses company that Gene talked about yesterday. It's not one of the unicorns. It's been around since 1846, so quite a long history. And it's massive. We're big leaders in broadband, in TV, in sport, in mobile, in voice, massive infrastructure, big company, 100,000 employees. And we're very keen to become very flexible and respond to the market more quickly.

So I'm Jerome Tassel. I've been in BT for 21 years. I've been across broadband, TV, networks, program management. I've tried it all, which is one of the great things in BT. And now I've got the privilege of running a team of 200 people who do design, software engineering, and are doing a lot of this transformation. Over to Jason.

Jason McElligott

Yeah. I'm Jason McElligott. I'm the Director of DevOps Transformation in BT. I haven't been there for 20-plus years, so I've been here for two years. I came from an SI background, and I've done transformational stuff in companies like Channel 4, BBC. So that's me.

Jerome Tassel

Cool. So this very busy diagram is not a plan. Going back to what John was mentioning this morning, it's definitely not our transformation plan, but it's what happened. It's the key events that basically we used for our journey. So if you start from the left, the stuff in grey is what I call unsustainable brilliance. We did some fantastic things. We launched Sport, we launched the first 4K channel that was commercially available. We did brilliant things with the Champions League. We've done brilliant things with broadband hubs and Wi-Fi, but God, is it unsustainable. We talk about burnout, we talk about heroes, we talk about infrastructure being really difficult to deploy and maintain. And like every company, we get people surveys as well, and our people were telling us they have no time to learn, they are burnt out, and it's really hard.

So we decided to make a change and we decided to look at two things. One is trying to not do so much infrastructure and go to the cloud, and second, change the ways of working. So looking at very different ways of working using Agile and DevOps techniques. And from then really there were a few key moments, and there's a few there on the slides, and we're going to deep dive in a few of these in more detail. We could probably talk about this nonstop for about two weeks, but we're just going to do two or three of those today, which hopefully are interesting.

So what we did is we started by getting some key new individuals who'd done this before, a bit like Jason. We also spent a lot of time doing a lot of learning. So coming to this forum last year, for example, was one of the pivotal moments for me. We did a lot of looking at papers, presentations, getting examples from companies who'd done it already before. We built a modernization team. We also built a platform to do and start to do CI/CD. We onboarded AWS as a cloud platform. We put lots of training in place for our people in terms of Agile, different ways of working, using the cloud, doing CI/CD. So again, trying to get people with all the skills that they need. We got some coaches in place. We got some external impetus again from people who'd done this before, like Dan North, for example, just to give us that vision of what it could be like and really trying to make it really challenging and really hard, not just pick the easy routes.

And then in the middle of all of this we also had a couple of reorgs. We're a big enterprise; this happens all the time. So we had to face that as well. So we took some of that as opportunities to realign the teams a bit more around, effectively, skills versus products. We also looked at creating product teams and going from project to products. And finally on this one, we're also starting to think about transformational leadership. Again, mentioned this morning, really important and something that we are really looking at at the moment.

What we found when we started to do all of this, especially looking at going from projects to products, is we started to try and create loads of those teams and we found that it was quite hard to get everybody to understand what the outcome was going to be, what we were trying to do, but also to get them aligned to try and achieve similar things. And I don't mean in terms of ways of working, but keep working on what the business wants while you have tens of teams that are individual and trying to be autonomous.

So we, based on some reading up and some external impetus, tried something called quarterly wave planning. It's used in lots of different models. Different companies do it in different ways, and we've just done it for what works for us. And it was two reasons to do that. One was to make sure all of those individual teams are aligned in terms of the goals we're all trying to achieve, but also as an accelerator to the transformation.

The great thing about those planning sessions, and they take two days, is that during those two days you talk about pretty much everything to do with the modernization. So you talk about OKRs, goals, product rather than project, looking at what you're going to do for the next three months, not the big goal of the company and trying to do big projects. You don't try and put a plan together, and you make sure you share what you've learned with the rest of the people you work with over the past quarter. So it's a great example of aligning teams but also accelerating the transformation. And those are not project workshops. They are product workshops. We look at the products and all the demand on those products for the next three months.

So this is how they work. Basically, we take in a lot of the principles for what we're trying to achieve, and we bring some speakers or some information to share with people about what we mean by trying to work differently. We make sure that everybody is there who works in a value stream or a portfolio. So it's devs, it's operations, it's program management, it's the commercial proposition managers. Absolutely everyone who is going to deliver on that value stream or that portfolio. And that's really important, because what we're trying to do is align absolutely everybody, so that when you walk out of the two days, everybody's very clear on what the goals are for the next three months. There's no questions, there is no stone unturned, and people don't go and work on something else for the next three months.

We also make sure that we bring in all the good work that people have done over the past three months. So they bring their results. Have they achieved their OKRs or their goals for the past three months, and how did that go? What did they learn over the past three months? What worked, what didn't? And they share that with the team. And we make that fun as well. My face has been used far too many times on silly videos. People were acting last week to show us some of the results. So we try and make it fun and engaging so it's not a dry, boring two days.

We bring the existing team construct. So we have teams based on what we needed to do for the past three months, and the products that we have. So we bring that construct in as well and use that during the two days. And finally, we need loads of food, lots of drinks, and a few games as well to get people interested.

Then what we do, once we've presented the information, we then talk about the goals. What are the goals for the next three months? And we get our commercial colleagues telling us what's important for them for the next three months. We get the teams to present what's important to them based on what they've done for the past three months. It might be that they need to tackle some technical debt, it might be that they've thought of a feature that could be really cool. So we bring all of that together, and we come out with an agreement on all the things that we all need to do as a priority for the next three months.

And then people iterate during the two days. They go into their teams, we form the teams that we need, and after the two days, we come out with everybody aligned. They've done enough engineering for the two days that they know they have a fighting chance to do it. And we come out with a good baseline of what we're going to do for the next three months.

So what does that look like? We had our last one last week, actually. So I'm using some of the new stuff that we've learned and did as well. The little table on the left there shows you the teams on the left and the goals that we've agreed for the quarter. And this time we've built a very simple matrix of what goals we're going to be able to meet, and what are the things that each team need to do. And some goals we just won't be able to do this time, but that's okay. We know we're working on the highest priority work.

And don't think, again, this is agile, so it's not like this is set in stone for the next... as it happened, this time it was 52 working days. Don't think it's set in stone for the next X working days. But it's a brilliant baseline. It means that everybody is aligned, at least at the start of the quarter. There's a good chance that not that much is going to change, because we need to market those products, we need to launch them. So there are some things we need to follow.

And then the bit on the right-hand side is the structure of the teams. So out of the session comes maybe a modified set of teams. I've blurred them a bit because I didn't want to share too much information. But the teams will change depending on the demand. So we might need to do something we didn't know or didn't think about before, so we might need to create a new team for that. So the team and the structure evolves with the demand, as and when it's needed. But we still keep in mind the fact that we have got product teams and they need to sustain what they're doing.

Here's a couple more pictures from the day. So one thing that worked really well this time, on the top left there, is having an impediment board. We found that the teams were working together, trying to fix things in terms of how they work, and they got stuck by just looking at it themselves. So we said, "Fine. If you're stuck, go and put it on the board and I'll pick it up." And so I spent most of the second day picking up the things that people tried to fix and couldn't, and I just went and talked to various people, just try and help them to remove those impediments if they couldn't remove them themselves. And that's worked really well.

Visualize. You know when you go to a big meeting for two days and you come back to the office and people tell you, "So what did you do for two days? Was it good?" So what we've started to do now is actually visualize the two days graphically. So we create those little one-pagers with the key moments of the two days. And when people ask us those questions, we can just talk around that. And it's really powerful because it shows people are bought into it, and that it's really, really valuable. And again, visualizing things is really important.

We have cakes. Everybody has cakes now. I think I heard the cake name at least four times every day for the past couple of days. So we do cakes as well. It's great. And we've also started to share all of this with the rest of the business by writing a little white paper that explains why we're excited about it, and how we do it, and why it adds value. And we've shared that with the rest of BT. So we are an element of BT. We are broadband TV. We don't do everything. It's about 1,000 people, and we're basically trying this out and sharing it with the rest of the business to show what works.

So what makes it work? One of the big things for me is to make what doesn't work visible to people, and for them to know that these workshops, these sessions, are really to try and tackle the things they're not happy about. So then they come because they want to change the way they work. Again, like we heard this morning, really important to have senior support and leadership showing that it's important. This is not yet another meeting. It is the meeting. It's the meeting where you're going to be able to work with the team, set direction, work out what you want to do, and give them support. So if you don't turn up, you're missing out on the big meeting that really moves the organization forward.

It's making it what the teams need. What I found over the past three is that actually this time there was far less structure and the teams just got on with it. They knew what they wanted to do, and there was no need to do loads of reporting back and big agenda. They just got on with it and did exactly what they wanted to do, which was great.

Get some external impetus to get people to believe what you say, because people don't believe what you say if you come from the business. They like to hear it from other people, like people like you and people from here. That works well. And really think about starting with something small. So we started with IPTV. We've now expanded it to our big data and analytics. We've expanded it to BT Sport as well is doing it. We've expanded it to our media and broadcast business and so on and so on. So just start with something small to show it works and get people excited about it. And let people know that it's going to be fun and engaging as well. It's not going to be dull.

And it really affects the modernization change, I really believe. It's because we talk about everything that matters. We talk about products, we talk about empowering the teams, and we do it in the moment. It happens. They make decisions, they get support if they need to. We talk about outcomes and not tasks. So we talk about goals for the business, goals for the squads. We record OKRs for the next three months, and we're getting better at that as well. And we do a lot of retrospection and learning during the day. People present and share with each other what they learned, what failed and what worked. And then people can try and deep dive a bit more and try and do it, see if it works for them. And it creates the social network as well that is going to deliver.

And finally, this is not just for software engineering. I've been doing it for team leadership. So we talked about transformational leadership this morning. We're finding this is really massively important and we're doing more of this now. We're using the IT Revolution paper on transformational leadership and the capabilities there. And I actually run quarterly planning for my team. So every three months we get together as a leadership team and we decide what we're going to do in terms of leadership for the next three months. And we have a board and we have stories and we've got OKRs for our leadership. So it's not just about software engineering. Massively important. I've also shared that with our HR teams, our finance teams, and they see the value and they're trying bits of this as well. So again, agility for everyone, not just for software engineering. Jason?

Jason McElligott

Amazing. Dead on time.

Right. So some of the stuff that Jerome's been talking about is lean practices. Some of them are capabilities, and you've probably seen some correlation between the Accelerate stuff that Jez Humble and those guys have been working on.

So another lean practice that we've been looking at, a little bit of history as to how this happened. I came in, we built a continuous delivery pipeline, a cloud services platform, and all the technical bits. We talked about doing some DevOps. We went round and we were doing roadshows around all the different BT offices. There's an interesting Gartner thing which talks about what do people perceive to be the biggest risks or issues with transformation, modernization. And so what I did is I basically ran the same survey internally while we went round and did all of these things, all these roadshows.

And I think the results were very interesting. We were busy building technical stuff, did the survey. Not one person said technical issues would be a challenge. So BT is full of very technical people. We know we're going to nail it in the end. So everybody, you can see the breakdown there. In effect, 60% thought process was going to be the issue, and 40% thought people transformation was going to be the issue. And that really stopped us in our tracks and we thought, okay, we're going around talking about the technical stuff. How do we think about process?

And at the same time, we engaged with Dan North, who came in, and he's a big thought leader in this space, and he does a lot of value stream mapping. He helped us with a quarterly rolling wave plan and stuff. And so the value stream mapping stuff is really interesting. So when you do value stream maps, you look at any process. It could be path to live, it could be fuzzy front end, it could be provisioning a server. And you basically work out all the blocks, you work out how much execution time and how much wait time there is, and then you look at where you can fix the wait time. And so it's typical for processes, especially complex processes in large companies like BT, to be less than 5% execution time. So all the automation work that you're doing is pretty much focusing on the 5%, not the 95%. So unless you look at your processes, you're going to struggle.

And so this was really a revelation at that point. And so BT, 146 years, was it, Jerome? We've spent 146 years to work up our processes. They don't get simpler, they get more complicated. So value stream mapping is a really interesting exercise where you don't need to do any of the tool... You have to go away and do your cloud and pipeline stuff. But actually you could start right now, look at your processes, get together, do a value stream mapping, work out where your biggest constraint is, and go after that constraint.

And so we were halving our lead time on some of our processes just in one session. So you tackle the biggest constraint, then you keep doing this continuous improvement, doing value stream mapping, and you go round and round and round this cycle. And so you can see that it drives real, tangible benefits. And I think also when you're moving to a DevOps-type model, you've got guys like the PMO, really useful, intelligent people. Rather than being the police of some of the processes, they can make a material difference to the lead time to deliver stuff.

And so if you think about squads or if you're doing development, most people's preconception is, if you want more velocity, you have to add more people. So actually, you should go through your value stream mapping exercise and you'll find lots of constraints which aren't people. They're handoffs, they're silos, all that kind of stuff, right? Or gating processes. And you need to go through that cycle until you get to the point where your builders or your developers are the constraint. And that takes quite a while. But most people, the first thing they do is they add more people. It's actually the process, it's the flow that you need to look at.

So we started doing this. And I think if I move forward, what it turns out is everybody talks about DevOps, right? And so I'm not going to read out all of the constraints and stuff that DevOps fixes. But I think what's interesting is if you went through that value stream mapping exercise, you would come up with a common set of constraints. So there are a common set, and things like silos and handoffs and ticket queues and all those things are the common ones. Some of them are business process, so we may have processes which are our own construction, and we can fix those, too. But really, it turns out that DevOps is more of a prepackaged set of constraint killers. So it's like an accelerator. So if you take DevOps ways of working, you're going to fix a whole load of constraints, which are common constraints that people have. And everybody has pretty much the same set of constraints.

And so it's a nice way of doing it. I think the other thing I just wanted to... God, defining what DevOps is. I think this is what we think DevOps is. So DevOps to us is, it's agile working practices. It's building on continuous delivery capabilities. And inside that, it's all continuous, right? You can see the word is continuous. That's what we're after. And it also includes the architecture aspect.

And what we see sometimes is we see even large consultancies that come in, and they'll say, "Okay, agile, that's over here. That's a different thing. Culture over there. Let's have a working group on culture. Let's have a working group on agile. And DevOps, that's automation. Let's put it over there." We don't see it like that. It's the whole way of working. It's the culture. It's all of it together.

And for us, DevOps is underpinned by the cloud and continuous delivery. I'm not going to read the constraints out to you there. But you'll see that we've built, and I don't know if anybody knows about GovCloud. So Jez Humble did a lot of work on GovCloud. When you need to be continuously compliant, you can't walk away from that compliance, right? You have to go through that compliance. So the best way to do it is to build like a pyramid system where you inherit the compliance of the underlying components in your platform.

So if you're a product squad sitting at the top working on your product, you inherit the compliance of your cloud service platform. You inherit the compliance of Kite-marked modules that you use in the pipeline. And so you just need to do a delta. You just need to get approval for the function that you're operating on at the top. And so this is what we've done. And also, we've tried to simplify everything. We want people to be able to self-serve this stuff. So we have an abstraction layer, so people don't actually have to know how to do Terraform or Ansible or any of that kind of stuff. It's just defined in the DSL. They write it out in XML. That's it. Away they go.

So how have we gone about doing this? We had a couple of goes at this, and not all of them worked. And so really the Accelerate stuff has really, really helped us. So the capabilities map in Accelerate, with if you want to be a high-performing business, then you need to drive all of these markers, and that's like software delivery performance and culture and all those kind of things. And you do that by affecting the system, like with culture. You can't change culture by telling people that the culture's changed. You actually have to affect the system that they're using. So you have to introduce lean practices, agile working practices, continuous delivery, and that will change your culture.

And so the way that we've gone about this is, if you look at the left-hand side, there's all those things around that we know develop a high-performing organization. So Jerome was talking about transformational leadership. And it turns out that most leaders are like, "Go and do transformation. I want more for less," and actually don't realize that they need to transform themselves. So we've got a big piece of work going on, basically just trying to become transformational leaders across the business. So it's not that I'm a transformational leader. All of our leaders should be transformational leaders, and there's a set of attributes and skills which they should have, and we're working hard to develop that.

So the other entities we've got is Lead Tech Tribe. These are the guys on the PO for that, and we're in there building out the tooling that we need, the pipeline, the technical stuff that we need to underpin continuous delivery. And then we have the chapters, which is the historical line management structure. So we still have developers in developing land and QA testers, and they're all in their own chapters. And so some of the performance stuff or some of the capabilities that we need are chapter-level capabilities, right? So shifting left on testing would be something that you could push into the chapter and say, right, as a group of testers, you need to push the boundaries for modern testing and take that forward. And so we incentivize and focus at the chapter level, too, to try and get those different working practices and capabilities done.

And so then the other thing we have is the modernization tribe. And so this is a distributed center of excellence, really. So it's pulling people from all over the business, the key stakeholders, the key influencers into a group, and it's their job to help move forward the things that aren't technical. So we do a lot of stuff around lean and agile management, which if you know Accelerate, you'll know that there's basically two bits, which is lean and agile management is the day-to-day execution. The squads, how you go about it, whether you do Scrum, whether you do Kanban, all that kind of stuff. And where lean products and process development, that's the stuff that we've been talking about here. So that's about doing value stream mapping, about doing our quarterly rolling wave plan and stuff.

And that's the area where we really struggled. Unless you get the right people into the modernization tribe who can influence... So what we did is we went out with the key heads of each one of the departments, had a trusted proxy who came into that squad so they could then take that information backwards and forwards. It's really difficult to get anywhere. So that's the kind of entities that we use.

And so the other thing that we make sure that we do is we eat our own dog food. So all of the cloud service platforms that we have, the pipeline stuff that we do, we use all of our own working practices. This is an example of the pipeline guys' board. So we build and operate and they run that, and they're responsible for doing both. And so you'll see on the board that there's a set of reactives and different issue types for things that break, different issue types for requests that need to be done. And then actually below that are a set of features which they're building at the same time. And so all of the work that that squad do are all on this board. Done for them is in prod. So they go through that, and when a feature is done, the feature's in prod, and that can be 12 to 15 features a week going into prod to be used and consumed by everybody else.

You probably can't see it, but the other thing is all of these, at a tribe level, we do quarterly rolling wave planning, as Jerome was talking about, and they have a set of product lines, and we get together, the cloud guys, the tool guys, and the pipeline guys, and we talk about what we want to achieve in the next quarter. And if you have a look at that board, you can see that all of the stories that are on the features side of it, they all map back pretty much to a quarterly objective. And so we have a traceability all the way back through. So we all know if there's anything on the board which doesn't have a Q1-type epic in it, then that's not something that contributes to our objectives for the quarter.

So what we found, and when we say we eat our own dog food, I think the other thing is even from a leadership team perspective, we're running quarterly rolling wave planning for the leadership team. So we're defining what we want to achieve in this quarter, and that's actually right now to change ourselves into transformational leaders. So we're going through what we need to do to do that. And so we're adopting these lean practices all across the business and it's working out really quite well.

Jerome Tassel

So loads of things we've tried, and in terms of outcome, I guess. We've now got serverless solutions built live, being used for our big events on BT Sport, for example, and our TV services. We've seen some teams who were only able to release twice a year; they now release every two weeks and they're really keen to do it weekly and then to go daily and many times a day, but it's a massive difference. The engineers in our team are doing bits of design, bits of dev. They think about operations. We're getting there. None of this is perfect. There's plenty more to do, but people are really trying hard to work out how they're going to do it, and we see some now doing it.

The learning piece has really transformed. So if we look at our people survey from probably a couple of years ago now, no time to learn, and now people actually proactively book time in their diaries to learn and to block everything out. It's fantastic. We see book clubs appearing, books in the office, papers doing the rounds, Knowledge School brown bag sessions. People are just hungry now for continuous learning and it's brilliant. It makes a massive difference.

On the CI/CD pipeline, just to give you some examples, 1,301 changes through the pipeline over the last 19 working days. Again, we had none of this before, so it's a massive difference to what we've achieved.

Personally as well, I've done many talks in my 20 years. I've never done a talk about ways of working. It's always been about technology, IPTV or R&D, quality of service or the internet. And that's been a massive transformation to really put, if not more effort, actually more effort definitely in ways of working than the technology and let the teams get on with that and spend time learning and working out how the teams want to work, remove impediments, think about vision and goals, and really try and work out how we're going to do this Agile stuff. And that's been transformational. And I have to say this: IT Revolution, the papers, the books, it's just absolutely brilliant to do that. So I can't wait to talk more to all of you.

And then finally, the quarterly rolling wave stuff. We've now done about 12 across different areas. It's really affecting the change. It's really getting people together to do this in the right way and align on the goals, align on the learning, and again, making a massive difference. So we've still got a massive way to go, but it's been really exciting. We've really looked at what needed to be fixed, tried it, succeeded with some, failed with others, but we carry on and I'm sure it's going to be a very exciting journey ahead.

Jason McElligott

And I think one thing that we haven't really spoke about is resulting technology change. So at the end of doing all of this, in there you can see that we've got services as a platform solution. This is for us. People talk about serverless and everybody thinks about functions, but actually when we're delivering our solutions, most of the solutions that are on the cloud are either serverless or managed services. So they're not all functions, but that's wrapped around API gateways, DynamoDB. And so our architectural principles are to deliver services which don't require any hosting. They're all managed services and if we can't do that, then by exception we'll build other services. But it's a pretty dramatic technology change from a cost perspective and from an operational perspective as well. And that's one of those things. Look at that, five seconds.

Jerome Tassel

And we've also made the backend stuff for all of this a bit more sexy. So before, especially product line, we'd all always go to the apps because they can see what happens. It's colorful, it's what people use. But now they come and talk to us and we're not the bottleneck anymore like they used to think. It's really transformed the way we work.

Thank you very much. Any questions? Are you allowed questions?