DevOps as Driver of Organization-Wide Change
Beyond organising our teams' work better, DevOps establishes the core structure and practices that sustain transformation of whole business operating models - this session relates case studies from several global enterprises that have remodelled their business structures and practices on a 'DevOps' core.
Chapters
Full transcript
The complete talk, organized by section.
Finbarr Joy
There's a question mark there. So that's very much one of the reasons that I was interested in being here today. This would be a discussion I'm interested to have with the community. I would suggest that, arguably, DevOps is a driver of organizational change.
Some of you have probably seen this already done to death, but I always think it's good to not take yourselves too seriously. This is a particular definition. Let's face it, that's one of the struggles we have, defining DevOps. I think this is an excellent summary to just set the tone for us.
Currently with Lebara. Lebara exists to make migrant lives easier: communications, entertainment, financial services. Telco background, so very much the calls provider. You'll see us at the airport, you'll see us at the newsagents. But we're right in the mix of a lot of those changes that are happening in the broader telecommunications industry, and inevitably threatened by those newer digital players: the WhatsApps, the Vibers, the WeChats.
Again, in the context of today's conversation, I think material to, how do we flip a different model? How do we operate in a completely different way? Because certainly the traditional telco model is not fit for purpose for the digital channels going ahead.
We operate across eight countries, six million customers, about 250,000 retail points, about 1,000 employees. I've only been there a very short time, so I'm still very much getting up to speed. But it's highly relevant for the context of today's presentation, in that I'm certainly viewing the DevOps model, if you like, as how we operate across the entire business.
I'll be relaying some experience as well from my previous role as Group CTO at William Hill, also Head of Technology at the phone book and the directories business at BT. Operated as a consultant through the 2000s. Very much spent that entire decade with this problem. Anybody who tried to do any material e-commerce deliveries through the 2000s certainly headed into this problem every single day. And I go all the way back to the beginning of my career, the pre-dawn era of Netscape even.
Why do we care? Why does any of this matter? I think the founding point, the context point for all of this topic, is very much around how much the world has changed. And we all work in organizations, most of which predate this era, many of which have their foundings in industrial times, if you like.
There's something going on. This illustrates relative market-cap value swaps, if you like, over a three- or four-year period. I think it's a snapshot from around a couple of years ago. And it tells us that those companies that are driving forward with software seem to be making a material difference to their marketplaces.
If software is eating the world, very trite analogy by now, absolutely, but I would say if software is eating the world, then for me, it's profoundly driven by the DevOps mechanic.
Yet many of us are rooted in organizations that are not necessarily yet taking that fully on board, or necessarily yet embedding the consequences of that mechanic. So it's very often the case that we do have these completely distinct teams, and we really, truly are squaring a circle.
It's not unusual, and I'm sure many of you work in organizations where these teams are geographically distributed. They may literally be separate organizations, set up with often contrarian objectives. The guys on the right are usually told, "You must deliver 100% uptime." The way they manage change management is typically change prevention. The guys on the left are told, "We want rapid delivery, we want lots of new product, and that's how you'll be incentivized."
I know many organizations despair at why this IT function can't seem to get it on. You're left at the beginning of every single endeavor standing, looking at the bottom of that mountain, thinking, "How the hell am I going to do that?" And I hope you can see the little figure there at the bottom left-hand corner.
Certainly my experience through the early 2000s and mid-2000s was all of those efforts that existed to try and bridge that gap. There was a lot of tools investment. There was certainly a lot of team investment. And of course, many organizations have been working with Agile since then. But my experience in those early days of Agile was it made this problem, this domain, profoundly worse.
All that seemed to happen was we spun the dev teams up to get faster and faster and faster, and we made the problem even more pernicious for those organizations. I don't think we really did anything, or any of the frameworks did anything, to help.
And certainly, there was a flocking to tools. I'm sure there are lots of you who were working with Java during that period. I remember the time when Ant was the great savior and the tool set that was going to automate everything for us. And we found ourselves very quickly buried under the scale of that. So you were yet again still not making much progress, finding yourself at the bottom of that mountain at the end of every project.
Who remembers the application lifecycle management days? We still see an awful lot of that now, but certainly it was the case that the machinery of, "We can automate your lifecycle," was seen as the panacea to address this problem. And that is supposed to be a production line, by the way. I'm aware it looks slightly like a tank, so there's not some subliminal message going on there, even though I am from a development background. I will acknowledge that.
Profoundly was the case that while there was an awful lot of investment in that, and certainly in my days in the consulting company, I somewhat guiltily acknowledge we got an awful lot of business from implementation projects around that tooling. Because guess what? The tooling was extraordinarily complex. It was extraordinarily brittle. It was very difficult to extend that out across multiple teams. It was certainly very difficult to distribute it across an organization.
So we made some progress. We started to make some way up the mountain, but it was by no means material. And we were still, for me, not really addressing, I think, what later became the more significant elements.
Around about that 2008, 2009 period, I found myself in BT, responsible for the directories IT operation, the phone book IT operation. And probably like many of you in this room, fascinated to see this new model that was starting to be promoted. The Flickr "10 deployments per day" deck. I'm assuming most people in this audience have probably seen it, but if you haven't, if you Google "Flickr 10 deploys" and "DevOps," it was one of those first early signals around, there's a different model for doing this.
And we started to look at applying those Lean techniques around flow, around feedback, and you could actually congeal these teams together.
I was really, really fortunate to have a particular experience then, which was quite a profound learning experience in terms of how you give yourselves a fighting chance of making this model work. That was an example where we had two different organizations on either side of the fence.
The context was an acquisition. Guess what? It was a funky digital web company that were the dev house. And of course, it was the incumbent BT operation that was ops. And you can imagine how that went on first encounter. It wasn't straightforward by any means.
What was very, very lucky: while we spent a good six months living that regular corporate experience of Gantt charts forever shifting to the right, showing green, green, green, green, green, and in the last quarter of the project, red, every single time. And why has that gone red? And you'd have the project manager sitting there tutting, shaking their head, saying, "Deploy let us down. We were looking great. We had it all in line. Deployment had it in their schedule. Deployment let us down."
And of course, the war rooms start, and everybody gets into the war rooms. And deployment, of course, are saying, "Well, nothing that's being handed over is fit for purpose." And certainly even at that time, even though this is several years ago, I certainly found myself thinking, "Is this really going to be my life? I seem to have spent my life in this conversation."
So it was good to start to look at a different model.
What helped us in that context was because it was an acquisition, of course, we got an awful lot of sponsorship to make this team model work. It was fundamental, actually, in the end, because we were allowed to create a combined team with a combined identity. And I know that if you wind back six months prior to that, certainly in either of those organizations, you wouldn't have been allowed to do that.
But because tens of millions had been spent on an acquisition, which must work at all costs, people were quite prepared to try anything. And we certainly got that benefit. So literally from creating a combined team, from creating a new identity in the team, it gave us a completely different dynamic.
It took a significant amount of time also, and I think that's where, again, I must acknowledge, extraordinarily lucky. We probably wouldn't have been given that amount of time but for the fact that this had to work. Yeah. We've made this acquisition, it has to work. So we had a good year to put it together, and I would say it took a good year before you started to see material improvements.
But those improvements were significant. At that endpoint, we found ourselves getting out from one release per quarter to certainly a release every week. And you could actually draw a line between the end-of-year financial performance, which had released a bunch of new features, a bunch of new pricing mechanisms for the product, which gave the organization a 40% uplift in revenue.
So whenever there are those anguished conversations around, "If you build this lifecycle, if you build this working model, I can understand how it's better, but what business value does it bring?" I think in the right product context, it brings material business value. You get straight to that revenue all the quicker.
So we were a little bit further up that mountain. We're making genuine progress here now, but we're still only able to see the summit. Why are we not all the way at the top? Because, again, my experience and the reason why I started getting interested in the organization-wide implications of this are that as that team coalesces and builds its identity and starts to get more confident, we then started to see that other parts of the organization became the problem.
And it is classic flow, really, for theory of constraints. The problem moved somewhere else. Our next problem became the contact center operation. The rate of release was no longer being sustainable without keeping them involved an awful lot more dynamically.
And the traditional project controls, if you like, weren't dealing with this at all. So while we'd combined those two IT teams, that was still seen as an IT delivery, and they were still certainly liaising with the business, in inverted commas, at one arm removed.
Which is where I started to get more interested in, if we've got this pattern right for these two units who were separate units, maybe it can help us adopt this pattern for the rest of the organization. And as you zoom out, you start to zoom out and consider that. And if you take that systems-thinking view of the world, then why not include more of these elements?
The next encounter for me was, lucky enough to be at William Hill at a time when they were making very significant changes. And the learning point here was, okay, in what contexts do you apply this new model? In what contexts do you explore it?
I think we all inevitably, those of us that are leading teams, we all inevitably look for those opportunities that mean, how do we do this as an experiment to make sure we don't get it wrong? There's an awful lot of fear around how different the model is to an incumbent enterprise team especially. Therefore, we have to tread slowly, tread softly.
I think what was interesting here was how big the bet was made. And if a gambling company doesn't bet, who's going to?
It was interesting in the sense that we go off on later in the presentation to look at the cultural elements around DevOps. They're all really, really difficult organizational problems, which, with the best will in the world, a bunch of people in an IT team, if you like, are never going to be able to solve. How do you get the sponsorship for that?
I think one excellent way of getting sponsorship is by giving your endeavor, if you like, and using this model in the most mission-critical thing that the organization is doing, because that does buy you an awful lot of support. Albeit some people would see the word support in inverted commas, as in, you're absolutely on the hook.
But it was profoundly material in the level of progress we were able to make in setting up the culture and the environment. The flow through from there, and again, the scale was fairly significant. Whereas at BT, we'd gone as far as about 50 or 60 people incorporated in the model, here now, it's the main dot-com delivery for the whole organization. It's the main revenue model for the whole organization.
The operations that you'd call DevOps in our terminology were spanning some 350, 400 people across several different locations: Gibraltar, London, Leeds, even parts of Tel Aviv. Spectacularly distributed operation.
And again, we had to take our time to get there. This was fairly relatively recent. I mean, this was only like three years ago. If you think about three years ago, there were already a bunch of really mature toolkits. There were already lots of practices out there, lots of organizational learning. There were already lots of professionals and practitioners, and of course, we hoovered up lots of them. We hired all the practitioners we could find.
There were already lots of consulting companies with the expertise, and of course, we engaged the consulting companies. My experience has been none of that accelerates you. It might underpin you. Yeah. It might give you some assurance. But the acceleration is down to how quickly the teams create their own environment to get to work on this together.
So we're looking at a good 18 months before you started to see a genuine pattern emerge, where you could say, "This is clearly a different organization now." And the proof points were really, at the beginning of that cycle, it was literally four releases per year. And at the end of that cycle, they were literally releasing on demand. Something like five, six, seven releases per week.
Real telling point as well is there were certainly some good revenue impacts, but a real telling point is, off the back of that, the main app ended up in the top 10 of the iTunes charts. And it was very clear that the responsiveness of the team to the feedback was making a material difference to how successful the app was now. Whereas you wind back six months previous, it was way off any of that kind of curve. You could spot the material difference it was making, being able to move quicker, and that mechanism.
You carry on zooming out. Why not apply this throughout the whole of the organization? There's some pretty arbitrary acronyms in here because I couldn't fit them on, but marketing should be in there, sales should be in there, finance should be in there. Why not adopt this model and treat the organization as a network?
Start to blend those practices the same way that we do in the technology teams. So I was starting to acknowledge there's no reason why a developer can't deploy. There's no reason why a so-called ops person can't write code in a world of infrastructure as code. Then maybe there's no reason why a tester can't be the guy who makes the product call, because that's how well they know the product.
What's the difference between a product manager and somebody who knows a product inside out, can make exactly the same call? Maybe a lot of these delineations are completely artificial. And if we empower the team, they can make an awful lot more progress.
Because I'm only three months in, I can be brave enough to declare this is the model that we are exploring with Lebara. I don't think there's actually any boundary to the scope of DevOps once you're comfortable with the model. Feedback and flow should relate to every part of the organization. Given how material and fundamental the delivery of digital product is to most organizations now, why shouldn't that be driven by the technology function?
Eventually, somebody will say, "That's all well and good." So that's collaboration at a grand scale. We are that traditional corporate now. We have got those divisions. We have got completely different matrices in place and incentive models in place. And somebody is going to mention the C word and say, "Well, what about that whole cultural element?"
And then you start to get into real trouble because, of course, over the last two or three years, everybody overlays that with digital, and we start to see this fog emerge. And there's a risk we can lose the plot. Before you know where you are, you're seeing plans appear to say, "We've got to start putting foosball tables in here and making sure we've got beanbags, and are we giving the developers pizza regularly enough?" And it's all far too cosmetic.
So instead, I would suggest it's time for those material conversations. I'm sure lots of you read The Phoenix Project, about a name call-out with Gene's conference. But those really, really difficult encounters that have to occur between the team members to make collaboration happen. Alchemy, chemistry, biology, it's all in there.
I don't see any way of accelerating that. I only see the responsibility and the obligation for the organization to sustain it and create the environment for that to work its way out. So I don't know how many of you have come across The Oz Principle, the book. But again, in my experience, it's been the best way of ensuring that the cultural change happens.
You can message all you like. You can send the cosmetic signals as much as you like, but the culture won't change until people have lived a different experience. And in most corporate environments, that's the most difficult thing to do.
So how do you give them those new experiences? It means it generates a new set of beliefs, and off the back of those new set of beliefs, people start acting differently. And when we start acting differently, we get those completely different results.
For me, that's profoundly the biggest element of that. How do we change the culture through action? How do we stop top-down, if you like, structuring, and how do we verify that through the bottom-up culture elements as well?
I think the art is to parallel those initiatives. You do need profound executive support to pull this off. You do need to create that momentum. But, in my experience, you have to underpin that with work that you do know is going on at the ground level of the organization to make it true. And the last thing is to, if you like, deliver to the teams, "This is the model we are going to use for you."
So where to start? Certainly in terms of hard lessons learned, I certainly am by no means prescribing what works, but I certainly learned a few lessons in terms of what doesn't.
When we do that zoom out, when we look at the wider organizational implications and we look at, imagine everybody was pursuing the DevOps model of feedback and flow. Imagine we are being a lot more entrepreneurial, a lot more aggressive around, if you like, questioning some of these role demarcations. Then it does beg questions around how we're currently structured.
It impacts the HR function. It impacts finance. We start to think about how we incentivize and motivate our people differently. Maybe those brittle definitions of, is it something like, was it literally dozens of job descriptions, I think, in the BCS SFIA portfolio? Maybe they're no longer relevant. Maybe those demarcations are no longer relevant because practices are much broader now, and maybe it's those broad practices that we need to be encouraging.
I've had experiences where people believed the broadening was career-threatening because it could no longer sit within the narrow guidelines of the next level of certification that some arbitrary IT authority had laid down. And if you think about how constraining that is, it's a complete cart-and-horse moment, isn't it?
It should be the experience of being able to do more, broaden skill sets, certainly do more as a team, and be more empowered, should be an awful lot better than stepping through rungs of some third-party certified career ladder.
Creating those multidisciplinary teams is extraordinarily difficult to do. At some point, you've got to break out of the cover. We can, to a certain extent, do what we like under the IT hood, and maybe the business barely even notices. But to really make this model work, at some point we've got to be clear. Come tell the business the dirty secret: do you realize these two elements were running completely in isolation, and we've had to work on bringing them together?
But if that's true for this team, it's got to be true for an awful lot of the rest of the organization as well. And reflecting on how you give a team the ability to be able to work in this model, then the most difficult element, without any question of a doubt, is a mature, existing P&L, successful organization having to get comfortable with a mechanism of autonomy. Because it's almost an adverse principle of everything they've learned in their entire management careers.
And last but not least, I'm not massively dwelling on tools here, but acknowledging the spectacular improvement in certainly tools and frameworks over the last few years. Make sure the team does have the freedom and entitlement to plug into the communities that are generating those. And certainly my experience has been it's the open source communities that are going further and faster with all of this.
So yeah, out of those functional silos, we start with our own teams. We break out of the pattern. We create the true multidisciplinary teams, which allows ops and dev to start merging, but also starts to drag in maybe some of the other roles start merging.
Another element relating to around the whole flow constraints model and theory of constraints: it's very interesting how, given every IT department out there has got a spectacular problem with demand management, and that the business has this expectation it can throw everything in at the top of the hopper, and if we shout at it loud enough, things will come out at the bottom of that machine eventually. But we all think it's all far too slow, and I wish somebody would just sort out this blob of IT.
When you bring the delivery this close to the individual P&L, then all of a sudden prioritization is free. Because you've got a constrained resource. You're no longer throwing this at a pool. You're throwing this at a constrained resource, and actually, the P&L owner now starts to become responsible for the management of this.
It's no longer a supplier-and-buyer conversation, which an awful lot of corporate IT conversations are. We're no longer in that mode. We're both accountable and responsible for delivery.
One of the points we're wrestling with at Lebara is how do we triangulate P&L and product and tech, and willfully make it, if you like, a fuzzy thing. Not allow one to be able to point the finger at the other, because one dies, they all die. Do you see what I mean? One sinks, they all sink, but one flies, they all fly. Yeah?
Similarly, recognizing the power of this model to be able to pivot the organization at a completely different mode of delivery. Lots of us are working in industries which are mortally challenged by new digital models. Certainly, in telco we are.
I would suggest it's highly unlikely that people are going to second-guess what's going to be the product that, if we put capital investment in for the next two years, is going to make us successful in five. Entirely unlikely that's going to be true on the basis that it's unlikely we know what Google, Facebook, Apple, Samsung, and the rest of them's plans are. We're entirely dependent on their ecosystem.
So how do we build a model which means that this organization is brilliant at experimentation? It no longer has to second-guess. We no longer have to make the multi-year epic plans. We can find the winning initiative and get there quickly.
It's at the core of DevOps principles to be able to release quickly, be able to get that mechanism out there. So all of a sudden, we're making a material difference to the commercial success of the organization.
Which brings you right into, if you're running all of those experiments, you're going to get very good at failure. And I think here again, there's been another element where it's certainly been a struggle in most large organizations to recognize the reality of comfortable with failure, and the conditions of failure. Recognizing that that attempt to prevent things from failing, if you like, is one of the constraints that prevents them also from releasing innovative product.
So again, think about those core mechanics around recovery, configuration management. They're all in there. This is DevOps machinery that we've got here. And when we apply it to the company operating model, we're starting to make a material difference. Failure isn't a big deal if you've only got a two-minute rollback to occur. And we're only going to be able to do that if we apply DevOps tool sets and processes.
We've talked about fast failure, and we've talked about DevOps. Somebody is going to jump up and say, "Great, let's go for digital as per model X, Y, and Z." And they'll pull down all the latest literature off the web.
You've got to avoid this cargo cult behavior. It's highly likely that whatever model is working, especially a pure dot-com consumer internet company, is unlikely to be lifted and shifted into a regular corporate organization.
So I appreciate I'm probably becoming highly boring on this point, but got to allow the team to develop the model that works for the team. There's lots of learnings we can apply, and we can infuse the team with those learnings. There's lots of benchmarks we can take. So that's what good looks like.
But think about the nuances in each of your organizations. It's highly likely you've all got really idiosyncratic stuff going on in your teams. Nobody's going to be able to impose a model on the top of that and say, "Try this template, it works." And the only people who are going to figure it out are your domain experts and your specialists. So we must give them the time to do that.
And that's why I really do highlight that benchmark element. The benefit of looking at other organizations is to understand the benchmark. How do we measure against that benchmark? What is our cycle time, cost efficiency, flow, quality?
Momentum is everything. Somebody in your team will already be pursuing this. Pair up with the business. Stop using that terminology of a business. Recognize that where we've created DevOps in our team, so we've stopped using the development and ops terminology, we've got to get into that habit across the entire organization. This is no longer a buyer-supplier relationship.
Think about those people you already have in your team that are probably going to be brilliant facilitators for this. Your architects already know what's where, and who works on what, and how it fits together. They're the perfect facilitators to get this underway.
Profoundly, I would say, get going. I've seen a lot of initiatives stall on trying to reorient very large programs into a new model and reorient change management. Those of us that own teams, we have to have the guile to get this going under our own steam and create that environment. Two to three people to get going. Two to three people to give you the confidence to head into the executive discussion, because you can see the proof of it. Therefore, it's not such a risky discussion after all.
And then, reflecting on, there are always reflections in these kind of scenarios around, but what does that mean for IT? And I would suggest if we're playing true to the principles of DevOps, and we take this throughout the entire organization, then we no longer talk about IT alignment.
IT alignment still assumes we're trying to line ourselves up with this business partner that's driving the business. And it's more about convergence. So what was formerly IT is now converged entirely with the business as technology practitioners.
And we're at the top of the mountain. Had to finish with a glib picture.
So in summary, how do we mobilize that company-wide change? I strongly believe it can. And I've seen a few interesting examples, and certainly nothing that would deter me from suggesting there's no reason why we shouldn't continue to do this.
It does take as long as it takes. The hardest part, harder than the tools, harder than the skills, harder than all of that, is absolutely the sustenance of the team to figure out what it needs to be able to do. And culture over tools every time.
The help I'm looking for: there must be others who are seeing this occur. It's absolutely the case that this pattern is beginning to be locked in, so I'd be fascinated to learn that, because that would help me in my conversations.
I think we're all still getting bogged down whenever we begin this journey with any organization, and are spending too much time debating what DevOps is. So let's try and get clearer on that.
What about when the team's geographically split? I completely subscribe to the principle of co-location. I've never seen it yet because I've never been lucky enough to work in an organization that was small enough to do it. So the reality is, we are distributed. Let's not make that a restriction.
There's one here. I don't know if there's any of you here from telco, but I'm a fairly recent entrant to telco. So having been a DevOps evangelist and so used to infrastructure as code, it's brought me up close around how far away from that telco is. So that's a particular pain point, and if any of you are in the telco technology world, I'd certainly be interested to learn that, because it seems to me the suppliers are way off.
And yes, more community outreach. That's my overwhelming ask of the community, if you like. Let's get in amongst all of those other forums. Make sure people appreciate these mechanisms, therefore the difference it can make to the business.
And if there's a risk, let's make that risk not be that it becomes too much of a cult, if you like, which I think the word can give you. It sometimes leads to far too heated debate too quickly. And recognize these strengths. Why don't we work with lots of other forums that are not branded DevOps, but nonetheless, the machinery is clear it's coming from the DevOps community.
And with that, thank you very much.