Lessons from 50 Enterprise DevOps Transformations
Over the last few years, we have worked on over 50 DevOps transformations, in many instances with large, global, traditional enterprise organisations.
During this time, we have gained hard won experience in how to be successful in modernising organisations to DevOps—changing working practices, re-structuring organisations, and re-platforming legacy technology stacks to benefit from infrastructure as code and other DevOps practices.
In this talk we will talk about our experiences and hard won lessons of how to be successful with a DevOps transformation, with many real world case studies referenced.
Chapters
Full transcript
The complete talk, organized by section.
Ben Wootton
Good to be here. My name's Ben Wootton. I run a company called Contino. We are a DevOps consultancy. We were set up with the explicit aim of helping large enterprises adopt DevOps. And today I'm here to talk about some of the lessons that we've learned doing that over the last few years.
Sorry. Keynote's at 2:45. Sorry. Okay, cool. We all need to race over there as soon as you...
Can't talk. Hello. You good? That better? What? Yeah, it is. Soon as they're quiet at the back. Yeah. Too quiet? Hello, testing. We're good? Thank you.
So, over the last few years, we have been working with lots of large enterprise clients on DevOps adoption. Some very big organizations, some international organizations with all of what that implies, regulated industries, some people with a lot of history and legacy. So I'd like to think we've learned quite a lot around how to do this through hard-won experience.
And these experiences have given me a different view of what a lot of DevOps advocates say. So, I personally would quite like to see something more like an ITIL or a Scrum for DevOps, like a template. I quite like the idea of DevOps teams, which everybody else in DevOps seems to hate. I quite like DevOps as a job title, as a concept, like DevOps engineering, which everybody else seems to hate. I hate the idea of DevOps as a culture. I always found that a little bit fluffy and inactionable when we needed a lot more concrete advice around how to do it.
And I've always quite liked the legacy vendors as well. I won't name names. They're not as fashionable as the Puppets and Chefs and Dockers of the world, which I also really like. But I think that we've got a lot to learn from those guys around what the enterprise wants from their IT software factory. So probably not the most fashionable DevOps person around, but this is a result of my experience.
So the aim for this was, I wanted to cover some of these hard-won lessons, go through them pretty quickly at pace. I actually thought I had 45 minutes to speak. I've only got half an hour. So by necessity, I'm going to go through them pretty quickly.
And I also want to put them into a historical context. So look back at our experiences in the enterprise DevOps world over the last few years. Because I think the lessons and the challenges that we're going through today are different to those of 2013, 2014 when we started doing this.
So looking back to 2014, when we started a business and we started helping people with enterprise DevOps, that year for us it was characterized by workshops where a lot of people were saying, "What is DevOps? What does it mean to us? Where do we get started first? What are our competitors doing?" A lot of confusion and people looking for guidance.
So there wasn't a lot of people doing DevOps. They weren't investing in proof of concepts or trialing the tools or trialing the new way of working. There was just a lot of confusion. And I felt sorry for those guys. This still exists even today, the sort of complex landscape.
Xebia put the Periodic Table of DevOps Tools together. It's quite cool. But, if you're working within an enterprise, how are you supposed to find time to make sense of all of this stuff? All of the vendors crowding around you shouting, "Buy our tool and you'll be more DevOps mature," can be quite intimidating from the outside.
There's all the thought leaders who are there shouting over the rabble about how, "Our view of the world and our DevOps is the right thing to do." And we were quite dismissive of enterprise, saying, "It's not fashionable, and we don't need to think about those enterprise concerns," when in reality we think we do. So it's a very challenging picture.
But at the same time, when I first started engaging with enterprises, we realized how far they have to go on this journey.
Typical enterprise IT, very siloed place. Lots of different development teams, different levels of maturity. Lots of operations teams, middleware teams, network, database, all of that stuff. And the common state within enterprise IT is handovers and raising tickets to each other.
So, my development team going to my support team and my database team, my middleware team, because they're not empowered to do any of this internally within IT operations. You've got teams raising tickets to each other because they're not empowered and they don't understand enough of the stack. And the whole thing just ends up feeling like orchestration through Jira and through tickets.
And that was my experience as a developer within enterprise IT. I was just raising tickets all day, and it's pretty depressing and ineffective.
And this was one of the things we did on our first project to map out an incident. It was with a bank. A website was running slow, and we literally just mapped out the resolution of that incident. And this shows, very simple incident, the amount of handovers and wastage and information loss and people who aren't empowered to get the job done, skills sitting in the wrong place. So this is a typical enterprise IT picture.
And again, I started because I was originally a developer. I moved into this DevOps consulting area. And you get, for the first time, the complexity of an enterprise architecture, enterprise IT shop, all of that legacy, all of the interconnections, all of the divergence and variance in the technologies. So it's an incredibly complex challenge which we have to solve here.
And the other early thing we realized in 2014, there was very little guidance of how you do DevOps in an enterprise. So if you look at Scrum, it's not perfect, but it's a template for how you do Agile. It talks about roles, it talks about iterations, it talks about the kind of outputs and the measurement in terms of velocity, but there's no similar thinking around DevOps.
Nobody's really saying, "This is how you set up your team," or, "This is how you do a release," or, "This is how you do change management," let alone the more complex considerations like financing. How do we do financing, and how do we do the new sort of KPIs in a DevOps world?
And I looked at stuff like ITIL and Scrum, and scaled Agile was just coming through, and these things aren't perfect and they're easy to detract from and throw abuse at. But again, it's a template which we don't have in DevOps. Even now, we're rubbish at bringing our ideas together around how you do it in an enterprise setting. So that was another theme back in 2014.
And another drum I used to bash a lot and got a lot of abuse over: everybody used to say DevOps is a culture and it's all about empathy and understanding and working together. And again, I found it quite woolly and inactionable.
I remember one of my first meetings with a CIO where I lectured him on how DevOps was a culture and he said, "Oh, that's great, but what do we do tomorrow?" And I was like, "Well, I don't really know."
At the time, people used to throw things around like, we'll get development and operations together, buy them some pizza and beer and DevOps will emerge. Or maybe we'll do a hackathon. This is a picture of a hackathon. And it just felt a bit lightweight.
You're never going to turn around a 10, 50,000-person organization with a hackathon or with pizza and beer, right? You have to get into deep restructuring, deep process change, deep human change management, deep technical re-architecture. So we need a lot more depth and a lot more advice for companies of how to do this.
So then moving into 2015, I think what happened was a lot of people worked small POCs into their plans for 2015. They got a little bit of budget. They said, we're going to find one team. We're going to try all cross-functional working, stuff like that. Oops.
But what we found then is it was really hard to get some of those sort of POCs off the ground, if you like. There was a bit of a will, but there was a lot of challenge to get them on the book of work whilst people are busy running their day-to-day jobs.
And for us, this was where we came into contact with a lot of change agents. So a lot of individuals in these organizations, they were a bit odd sometimes. They were sort of mavericks, and they wanted to do things differently. They wanted to break down the silos in their organization.
We teamed up with lots of people in early 2015, those change agents who said, "Look, we want to do DevOps, but how do we sell it to the rest of the organization and get it off the ground?"
So at this period, we were doing lots of work on business case and helping to articulate why something which we intrinsically know is good and makes sense into how does that save money for the organization.
So, don't need the detail, but during this year, we've spent a lot of time understanding how to go from DevOps as a concept to DevOps as a business case. Like, what are all of the efficiency levers? What is the return on investment of those? How do we play those efficiency levers back over the old book of work and come up with hard cost savings out of the bottom? How do you model the headcount changes and stuff like that?
And this continues even now in 2016, as people are looking to roll DevOps out even further. But there's a lot of lessons there around articulating the business case with hard numbers against it.
I think related to the business case was, last year, people were realizing how DevOps enabled a lot of this stuff: security, audit, governance, and stuff like that. So it's not just about time to market, it was also about efficiency and allowing you to hit these other challenges and hit them more efficiently.
So, using more automation, driving security into a process, and probably removing headcount as a result of that. So there's a picture of an audit.
On to 2016. So again, it's quite interesting in our experience, every year is where we see a leap in adoption, and I think that is enterprise budgeting cycles and planning cycles and stuff like that. So quite excited to see what happens in 2017, actually.
And here we've got more lessons. I think these are more valuable because these are current challenges which we see out in the enterprise.
So, we have seen a real rise in adoption, a more aggressive adoption, from early this year. A lot of those POCs were proven, and this year it's more about how do we take it into more areas of the organization, where do we go, et cetera.
And something we were talking about yesterday is that a lot of those lessons from last year were not necessarily applicable this year because a lot of those proof of concepts, maybe they were relatively greenfield, the system we were building. Maybe we'd put a team together temporarily, so we hadn't changed any reporting lines. We'd just built a quick virtual team. Maybe we were working within the cloud.
So a lot of those proof of concepts were pretty ivory tower in nature. The real challenge with enterprise DevOps comes with when you want to merge those proof of concepts and those early engagements, those early proof points back into enterprise. That is a point where you hit your financing and your reporting lines and your governance and your information security who say, "There's no way we're going to put that live," et cetera.
So, one lesson we've learned is, in a way, you're best to take on the hard challenges first if you want to do enterprise DevOps. So that might mean work on the existing infrastructure. It might be have that conversation about reporting lines early and try to make a real progress rather than stand up a kind of fake environment just to get some proof points which don't translate.
Because I've seen a few people who have done this. They've done the proof of concept. It's gone well. You've gone to the broader rollout, and it's kind of failed because it hasn't moved quickly, and the whole thing loses initiative. So there's a balance there.
I think people are also realizing this year that DevOps is a sort of reorganization and the people and process implications of doing it. So last year, we kind of tackled the easy stuff. A lot of people were putting Puppet and Chef in, and we were working on the cloud, and we were doing automation, and maybe we did these small virtual teams. But to really get the benefits, you have to do the reorg as well as adopt the tooling.
So, a lot of the lessons we've learned are around how do we move to that sort of end state for DevOps. So, this includes moving to more cross-functional teams, where for the first time we're putting developers, testers, and IT operations together.
And then this role of the DevOps team. So, these people who are responsible for building out your path to production, building your CI/CD, doing your test automation platform, stuff like that. Again, it goes contrary to what a lot of people say, but we've found that these sort of DevOps teams can be a source of efficiency and scalability. But you have to do them in the right way.
Avoid them becoming a silo, and make sure they have the right mentality, the right coaching and training. Their responsibilities need to be correct and articulated, and stuff like that. So this is a big focus for us this year with people who were doing automation last year, they're now willing to take on the sort of bigger organizational challenges.
And then it's how do you scale these concepts through the enterprise. So, what happens to that traditional IT Ops function as headcount kind of moves into DevOps teams and into product-aligned teams? If it's a really big organization, how do you federate this, so you've got DevOps teams at different levels so we can get that balance of governance but also flexibility? So this is a big topic for us right now.
I think the thing that kind of kills it and the challenge sometimes is the people element of this. I draw all of these sort of diagrams. I do it a lot. I'm drawing org charts, and I'm working out headcounts on a spreadsheet and stuff like that. But you do have to remember, at the end of the day, it's real humans who you're talking about, and they're in their job and stuff.
And what we've realized is it is a sort of human change journey as well. And there's a lot of science behind how you support human change and how you successfully move people from A to B, from a mentality, from a cultural perspective, from a skill set perspective and stuff like that.
And although this is softer and although I'm not a fan of DevOps as a culture, this is the important stuff which will help you be successful. So you can't come in like this guy, dictatorial and putting people through permanent restructures and dumping process change on their head. You have to manage them through that process.
Another kind of common thing we bump into a lot is, still related to people, in any sort of technical change, we have to focus on the users and the adoption of that technology.
So I work with lots of people who maybe they've built a CI/CD pipeline or they've built a private cloud or they've built a public cloud or a monitoring platform, which they want different people to adopt.
I think what we have to do when we're building technology is think about the user and build it in a very customer-centric way. So, we advise people, don't build technology in a silo. Don't have a big ta-da moment where you give it to your user community and say, "Here it is." You have to do it very iteratively, lots of engagement with the customer, and stuff like that.
So I see this quite a lot where, from a DevOps perspective, you go, "Cool, we've built a platform as a service. We've built a private cloud. We've built a CI/CD pipeline, or we've installed GitHub Enterprise." But then the users don't show up because it wasn't built with the right level of engagement.
So, I think even internally within enterprise IT, we need to be much more customer-centric and customer-focused, and work in a similar way than you would if you were a digital company building out something for your customers.
And from that perspective, I often see how the tooling and the people side of things, they have to evolve together. We see a lot of people who they adopt a disruptive technology. So, in the old days, it was Puppet and Chef. Then it was cloud, and now it's Docker, and it's containers and stuff like that.
But they don't evolve the way of working, and they don't evolve a skill set to take advantage of it. So then all we're left with is the same old legacy process with all of that waste and handovers. But they haven't made the changes to sort of really take advantage of that. So they introduce some automation, but they don't get to DevOps.
So what I think people really need to do this year is move towards that more integrated model where we're changing people, process, technology, culture, way of working. And that is when I think organizations will really start to see a return on investment from DevOps.
Quite a few lessons here.
And the final point is, I marvel every day about how difficult enterprise DevOps is. So this picture: another week doing enterprise DevOps.
Because what I like about it is, it's some kind of weird combination of people, process, technology change. What has there ever really been where you've had to do that from a transformation perspective?
We're trying to work out how do we evolve these complex legacy interconnected systems. We're trying to change process, which is very complex in a big enterprise organization where you've got different departments handing off. Not only do you have to change process internally in those groups, you have to change it end to end.
We're trying to do organizational restructures, and that involves moving people and upskilling and bringing new people in, changing the commercial model. So it's incredibly complicated.
But, as we all know, DevOps is an incredibly big prize at the end of it in terms of speed, efficiency. So it's happening.
This is how I often feel on a Friday afternoon. I'm pretty tired today, actually. It's been a week like that.
And I do think, Gene Kim talks about this a lot, how you need to be a hero to make this happen. You have to put your head above a parapet and almost go outside of process and find people to come behind you and join you on that journey. So you are putting yourself at personal risk and career risk.
So I think that anybody who's doing this within their organization or helping organizations do it, they are a bit of a hero, really, and I really respect a lot of the people I work with who have successfully moved their organizations forward.
And that, I believe, is the end. So, thank you.