Gene Kim

We are here to present the long-awaited 2017 State of DevOps Report findings. I just learned this morning that I get to present with you. I get to present a couple of the early slides.

Let me first start by saying this is the fourth year that I've been involved with this amazing collaboration with Puppet, where over four, now five years, we've spanned over 25,000 respondents, 27,000 respondents if you add them all up. And I think it's given this unprecedented ability to understand what high-performing technology organizations look like, and what are the practices and factors that enable it.

Many of you, if I understand correctly, got an email in your inbox at 9:00 a.m. London time with the actual final report. I actually had some people ask me questions about it, so this is freaking awesome. I'm so happy with the way this report turned out.

Can we get the timer going, please, Jez?

The only person who has contributed to this that... Well, many people contributed to this, but another person that we need to call out is Alanna Brown from Puppet, who's been doing the State of DevOps Report from the very beginning. That was really her brainchild in 2012. We're now going into the sixth year of this. And Nicole, your first year was?

Dr. Nicole Forsgren

2014.

Gene Kim

2014. Awesome.

First, number one is the demographics. And the point of all this is to say that with an N of 3,800...

Dr. Nicole Forsgren

Just over 3,200 this year.

Gene Kim

3,200. Not enough prep time.

These organizations really do look like the organizations in this room. 25% of these organizations are definitely enterprises with 10,000 employees or more, spanning across every industry vertical. So that's the first thing that we always look for: is this a representative sample of organizations that we're trying to characterize?

I guess I'm really here to present two things before I turn it over to Nicole. One of the most exciting findings for me is around transformational leadership. Really, the punchline for this is that leadership matters, and it even matters more than we initially thought.

Dr. Steven Mayner presented on transformational leadership earlier this afternoon. It turns out that leadership significantly amplifies the ability for organizations to become a high performer.

Before I describe what the impact is, let me describe the model that we used. We used a slightly different model of transformational leadership than what Dr. Mayner described. Instead of four categories, we used five.

The first one is vision. Does the transformational leader understand the organizational direction, and can they understand how to position their teams to contribute to that, to become relevant, to become important?

The second one, the factor that I just love, is intellectual stimulation. In other words, does the leader have this relentless need to challenge the status quo? Just because we've done it for 20 years doesn't mean that we should be doing it the same way now. Challenge basic assumptions of our work.

The third one is inspirational communication, the notion of now that we're part of this team becoming relevant, can we get people on board, get people excited to be a part of the effort?

Fourth is supportive leadership and personal recognition. I think these are the sort of classic servant leadership behaviors. And what's interesting about servant leadership versus transformational leadership, there's this pithy line in the report that I love, which is the servant leader often focuses on the needs of the followers. The transformational leader also focuses primarily on the needs of the organization. You need both.

So what we found was that leadership matters. We asked in the survey, to what extent does your leadership exhibit these 15 characteristics? The teams with the least transformational leadership characteristics, in other words, the bottom third, were one half as likely to be high performers.

And when we flipped it the other way, leaders can't do it alone. You take the teams at the top 10% of transformational leaders, they performed no better than the median. In fact, technically, they did actually a little worse. So it just says that leaders without culture, technical practices, great people, talent, and so forth, they don't do anything. So leaders do a lot, but they can't do it alone.

Is this interesting?

Yes? Cool.

And I guess my last thing I want to point out here that made me so excited about this was that I think these are clearly the characteristics that are being exhibited by the leaders presenting at the DevOps Enterprise Summit.

In fact, one of the precursors to this work is that Steve Mayner, Nicole, and I did this test. We administered a battery of questions that asked the DevOps Enterprise leader community, to what extent do you self-identify with these? And it was off the charts. For a sample of N of about 110, I think, they all clearly self-identified as transformational leaders.

So the next step was then to integrate into the State of DevOps Report. We can now ask, to what extent does your leadership exhibit these? So that's a very exciting finding for me.

This matters more than ever since half of CIOs, according to Gartner, who have not transformed their team's capabilities will be displaced from other factions, including the digital leadership teams.

The last responsibility I have to discharge before I turn it over to the team is that high performers continue to achieve both better throughput and stability. This has now been validated for four years in a row, and I think almost all the anxiety has been out. We know that high performers are both faster and more reliable.

So this year, what we call affectionately the boom statistics, is high performers are performing 46 times more frequent change deployments. And they continue to deploy far quickly. From a change commit to version control through integration, test, and deployment, they're now 440 times faster than their lower-performing peers.

And the reliability outcomes are still decisively faster. Their mean time to repair is 96 times faster, and the change failure rate is one-fifth as high. So you'll note that the statistics have changed even over the ones that we presented to you yesterday, and it's not because the high performers are getting worse. I think we can actually say, in absolute terms, it's because everyone else is getting better.

Dr. Nicole Forsgren

What do you think about this?

Some people will come back to me, and they'll notice year over year, and they'll ask me, "What does this mean? Does this mean the low performers are getting worse, or the high performers are getting better? What's the difference there?"

It's not that the high performers are changing. What we're noticing is the difference is relative. It's the difference between the high and the low performers.

So what this means, Gene, if you want to click over, the high performers are consistently maximizing their throughput. They're staying really, really good, and they're also consistently maintaining that stability, which is really exciting.

Although, then take a look at the low performers. When we get these boom stats, we're actually just comparing the high and the low. The low performers have gotten better at their throughput, but they're sacrificing on their stability.

What this tells us is that it's confirming that the high performers, if you're doing DevOps well, you're able to get this throughput and stability together. Trade-offs are not necessary. You can have it all, and we've seen this four years in a row.

However, our low performers, this is the first time we're seeing a little bit of evidence of trade-offs. They're trying to get this throughput, but they're really seeing it at the expense of stability, and it's not working very well at all.

Nigel Kersten

I think we've seen automation has always been a really important part of the whole DevOps movement. If you go back to the really early days, Damon Edwards and John Willis coined CAMS: culture, automation, measurement, and sharing. And then Jez later came along and added L for lean. Automation has always been a big part of it.

It's automation that lets us create those contracts between teams. It's what lets us push out code reliably. It's what lets us just have CI systems that actually work, rather than someone packaging software up by hand. And we've seen this over and over again for years with the DevOps Report, and just anecdotally from everyone, that automation is really at the core of what we do. But it's not just about automation.

What we decided this year is we wanted to dive into four key areas about automation. We wanted to find out what's the difference in automation rates between the high performers and the low performers for configuration management, for their testing, for their actual deployments, and for one of the things that's actually really hard to automate, which is the change approval process itself.

So we went through, and this is pretty clear across the board: we see that the highest-performing teams are much more automated. But I think we saw, jumping into some of the stats from earlier, that it's not always just going to be about automation. We're going to talk about that a little bit later.

And here we see the difference. Now, if you look at these stats, you'll notice that we haven't actually asked people how much of your work is automated, because you can't actually ask someone that and expect to get a really honest answer. But what you can ask people is, how much manual work are you doing?

We did this last year when we dove into planned work versus unplanned work. You've got to ask the inverse question of people. People have a pretty visceral awareness of how much manual work they do, but automated work, by its nature, kind of fades into the background and isn't necessarily something people have a great awareness of. "60% of my work is actually automated."

So here, if you have a look at the amount of work that's being done manually by different groups, you can see that it's pretty solid. The high performers are doing much less manual work across the board. And you see that the stats are pretty solid, again, apart from change approval process.

I think if you read a bunch of people, Rob England with the IT Skeptic blog, there's a lot of people talking about what change management is going to look like in a fully automated, fully embraced, DevOps glorious future world. And I think it's pretty clear it's going to ultimately just fade into the background and be baked into all of these processes.

But this is some of the hardest stuff to do in big enterprises. If you look at these stats, there's something kind of interesting here. The medium performers are actually doing more manual work than the low performers. And this wasn't something that we actually expected to see. It was hinted at a little bit with the unplanned versus planned work from last year, but this is what ends up being called a J curve.

We've got a few explanations for why we think this is. You start off automating things. You're picking the low-hanging fruit, the quick wins, the things that you go, okay, that process is a nightmare. If we just automate it, it's relatively simple. We'll get a big win.

But as you start automating these things, most of your processes and systems, especially in enterprises, are actually entangled with each other. You're starting uncovering more technical debt. And a lot of that complexity, as you uncover it and you fold back the curtain, people start putting in some more manual processes to make up for it.

But what we can see here is that if you actually power through it and go, okay, let's start automating things, let's start automating the hard things, and you keep going all the way through, performance will radically improve.

Dr. Nicole Forsgren

We've also found year over year, DevOps applies to all organizations. Gene referenced this earlier when we take a look at the types of organizations that apply.

So let's dig into some of the science. This is my favorite part. This is the research model for this year.

Let's talk about how to read it. This is my favorite piece. Anytime you see an arrow, you can read this as predicts, drives, impacts. And by the way, this is what makes this particular research report kind of special and kind of different in the industry, because we actually follow academically rigorous research design. It really is a research report that gets into prediction. We go a step beyond correlation.

And if you want, you can dig into it in the book, chapters two through four.

Jez Humble

Page 28.

Dr. Nicole Forsgren

Page 28.

It goes through the different steps that we follow. So here's how we can read it.

As Gene was mentioning, transformational leadership underpins, influences, and amplifies all of the pieces that go into DevOps, all of the technical work, all of the cultural implications, and all of the process work that flows into everything else. That's why you can see all of those arrows that flow all of the way through the rest of the process.

All of the pieces that flow into transformational leadership: transformational leadership is comprised of personal recognition, supportive leadership, intellectual stimulation, inspirational communication, and vision.

It then predicts and helps support and drives and impacts all of our technical processes, which are test and deployment automation, continuous integration, trunk-based development, shifting left on security, loosely coupled architecture, and empowered teams. Jez is going to touch on that in a minute.

These all predict and impact continuous delivery. Now, transformational leadership also predicts and impacts lean product management, which is comprised of team experimentation, working in small batches, gathering and implementing customer feedback.

Now, that dotted line, this is 2016. Last year, we looked at how lean product management predicted IT performance. This year, we kind of flipped the model. We took a look at how IT performance predicts lean product management. Jez is going to take a look at that also.

This year, we took a look at continuous delivery in and of itself, and how that predicts IT performance and reduces deployment pain. This is exciting because we're seeing validation of our results in another year.

Now, IT performance, again, is predictive of organizational performance and non-commercial performance. This is super exciting for me. This is so great for me this year.

So what do we mean by this? High-performing organizations are twice as likely to achieve or exceed goals. We've said this, and by the way, we've said this for four years in a row. When I first started working with the team in 2014, we were one of the first studies to find this, which was huge. For years, and I'm talking decades in the research, we've never seen a link between investments in IT and the bottom line. This was big.

And what we mean by organizational performance, we've always studied profitability, productivity, and market share. High performers are seeing a 2X improvement here, twice as likely to exceed these goals.

But for years, at least for the last couple of years, people have come up to us and they've said, "Okay, but what about me? What if I work in the government? What if I work in not-for-profit? Or what if I work in a for-profit, but we really have a strong push for non-commercial measures? What if we really care about corporate social responsibility?"

And so this year, we added additional measures like non-commercial goals. And do you know what's fascinating and super awesome? We see the same multiplier. Not only is it predictive, but we see that high-performing organizations are twice as likely to exceed these objectives: quantity of products and services, operating efficiency, customer satisfaction, quality of products and services provided, and achieving organizational mission goals.

And by the way, these measures come out of very strict, highly refereed external journal articles coming out of the accounting literature that have been shown to be robust to economic cycles. The org perf measures on the left are highly correlated with ROI. We've been very careful to make this as academically rigorous as possible.

So I'm excited to see that IT perf is predictive of now commercial and non-commercial org perf. Yay, DevOps!

Jez Humble

A couple of years ago, we looked at architecture, and we looked at all kinds of different architectures, whether people are using greenfield or brownfield services and applying DevOps to those, whether they're using COTS, commercial off-the-shelf software systems, or custom-developed software. And we found that actually, DevOps works in all of these different kinds of architectures.

DevOps applies just as well in a brownfield situation. If you're using mainframes, you can absolutely apply a lot of these principles. There's a talk by, who was it?

Dr. Nicole Forsgren

Rosalind?

Jez Humble

Rosalind Radcliffe from IBM.

Dr. Nicole Forsgren

Rosalind Radcliffe.

Jez Humble

About applying DevOps principles in a mainframe context, and the data supports that. There's no reason that you can't use DevOps just because you're working in brownfield systems, you're using mainframes. It absolutely works.

So we wanted to find out a bit more about architecture and what works and what doesn't work in terms of architecture. Obviously, there's a lot of people in this room who are probably experimenting with Kubernetes and Docker and a bunch of these new technologies.

What we found is that doing architecture right is the strongest predictor of continuous delivery. And we listed a whole bunch of things earlier in the slide on technical practices: continuous integration, trunk-based development, test automation. The biggest predictor of being able to do continuous delivery effectively was architecture done right.

Now, what do we mean by architecture done right? Do we mean using Kubernetes and Docker and so forth? Well, actually, it turns out the architectural outcomes are the most important things.

So ask yourself, can you answer these questions for your team? Can your team make large-scale changes to the design of its systems without the permission of someone outside the team or depending on other teams? Can my team complete its work without fine-grained communication and coordination with people outside the team?

These are measures of how decoupled the teams are from each other. Can my team get its work done from design through to delivery on its own without having lots and lots of collaboration, seeking permission and approval from different teams?

So having cross-functional teams, so you have everyone required to do that work within the team rather than having a bunch of different functional silos, allows you to do well on these first two measures.

And then the last three measures are technical architecture measures. Can I deploy or release my product or service on demand independently of other services the product or service depends on? Do I need to do a big-bang monolithic release where everything goes out at once, or can I deploy my service or product independently of other ones?

Can I do most of my testing on demand without requiring, for an example, an integrated systems test environment where there's a whole bunch of things that need to be in place? I need to have Salesforce and a bunch of other things before I can do any real testing.

And can I perform deployments during normal business hours with negligible downtime? This is really the fundamental continuous delivery test, in my opinion.

Actually, you can answer. Who in this room has to work evenings or weekends in order to do deployments?

Okay. That's a sign that something's wrong with your architecture. The whole reason me and Dave Farley wrote the Continuous Delivery book is because in 2005, we found a way to take a weekend deployment and change our architecture so that we could deploy during normal business hours at the push of a button. And we wrote that book so no one would have to do that ever again at the weekend or evenings.

And obviously, we failed. So I'm a bit sad that here we are, seven years later, and there's a bunch of us still doing that. That's not something anyone should accept.

Gene Kim

It wasn't a good enough book, Jez.

Jez Humble

Yeah. I'll actually try harder.

Gene Kim

I'm kidding. Did I say that out loud? Inside voice, Gene. Inside voice.

Jez Humble

Crikey.

The data bears that out. It's not about whether you're using... Who remembers service-oriented architectures and went through a service-oriented architecture re-architecture in the early noughties? Okay, lots of you.

And instead of focusing on these outcomes, we were focused on, do we have the right SOAP and WSDL packet configuration and so forth, instead of, did we actually achieve the outcomes?

So if you're re-architecting or implementing a bunch of new stuff, the technology is not as important as, did I achieve these outcomes that we care about?

Dr. Nicole Forsgren

And I'd like to point out we've been very careful to design the research this way, to make sure that the results and the findings and the report are very technology agnostic. So you can apply it to everything and anything, because our exploratory research in 2015 was very clear that good outcomes were highly correlated regardless of technology platform.

Jez Humble

You can achieve these results in any organization. I spent last year working in the U.S. federal government doing this stuff with very complex legacy systems and in a very highly regulated environment. You can do this stuff in any environment.

But just because you're working in a new environment with shiny technologies, you could still end up with a horrible big-bang orchestrated deployment that happens to be on Kubernetes if you do it wrong.

These two outcomes, the organizational ones and the technology one, are related by what's called Conway's Law, one of the most important laws in software delivery, in my opinion. Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations.

So what our research bears out is what's sometimes called Reverse Conway's Law, which is that you should design your technical architecture to enable the organizational architecture that you want, which is small teams, cross-functional, loosely coupled, that can design and build experiments and push stuff to production without these very complex fine-grained communications that slow things down.

We also looked at lean product management in a bit more depth. Last year, we looked at the first three things on this list: working in small batches, making the flow of work visible through the process, having Kanban and obeya, visual management boards basically, to model the flow of work through the organization. Gathering, broadcasting, and implementing customer feedback.

Those three things last year were how we modeled lean product management. This year, we added a new one. Can teams try out new ideas? Can they update specifications during the development process? Are teams able to say, "Well, listen, we tried this idea out. We did some user testing. We found out this idea is bad. Let's rip it up. Let's write a different story instead."

And we found that actually these four capabilities, with the extension this year of the ability of teams to rip up or change stories based on customer feedback, predicted organizational performance in those...

Dr. Nicole Forsgren

We're going straight to money.

Jez Humble

...for-profit and non-profit measures that Nicole described earlier.

So what we're looking at this year is the technical practices enable lean product management. You can't run experiments in production if it takes you a month to get your code deployed. You can only run experiments in production if you can get new ideas and do A/B testing with lead times of hours or minutes. And that, in turn, is what allows you to avoid building features that don't matter, which is what allows you to make money faster than your competition, which is why companies like Amazon are eating the world.

Nigel Kersten

One piece of advice.

So this is where we all get to say one thing which we think is advice to you all based on what we found out this year. We're a few minutes ahead, so I'm going to take two.

If you look at the stats, and we saw that speed of deployment, the gap between the high performers and the low performers actually shrank, but quality actually got worse. Don't just adopt automated deployment tools that push crap to production faster, because it's still just going to be crap.

We've got to remember, the DevOps movement started as an ops movement. If you're just going to let people not worry about operability concerns, about what maintenance in production actually looks like, actually taking that sort of ops mindset and making it part of the development teams, things aren't going to get better. It's going to happen more quickly, but it's not going to get better.

And the second one: if you have a look in the report, we've got a really good section, I think, on commercial off-the-shelf software, because we hear this all the time from people. They're like, "We don't write our own software. I'm mainly running commercial off-the-shelf software. I can't take advantage of all these practices of re-architecting everything."

You can't do everything, but there's an awful lot you can actually do. And the one bit of advice that I think we're all really behind is, it's a whole-systems approach. It's not just the tech. It's your people. It's the processes.

Don't treat your processes that may have existed for 10 years inside your organization as ossified and immutable, and you can't change those, and customize the hell out of your commercial off-the-shelf software just so that you can actually facilitate those processes.

Keep your off-the-shelf stuff as vanilla as possible. Adjust your processes to fit where you can. You ultimately get the big wins. If you don't need to customize that software very much, you can throw it through a CI system way faster. Developers can deploy copies of it to do experimentation on much easier.

And all of these other benefits that Jez went through about different kinds of architecture, you can kind of apply a whole bunch of them even if it isn't software you wrote.

Dr. Nicole Forsgren

I'm pretty sure you all can guess what mine's going to be. Measure.

I love that you all just came back to this for me. Thanks. This is pretty, and this is awesome. I love that I just called my own work awesome. Thanks.

These are also final results. We get surprised often by what works and also what doesn't work. There are a lot of things that we measure and hypotheses that we have that don't pan out.

The same is going to be true in your own organizations. Measure your own transformation journey. Keep track of what's going on. Take an experimental and hypothesis-driven approach, because if you don't measure what you're doing, you won't know what's not working, and you will continue to grow and transform and build features and deploy features that just aren't working.

And so I think a data-driven approach is really important.

Jez Humble

One thing I would like to do, which I always want to do, is just reiterate the overall conclusion we've had for the last four years, which is that high performers do better at throughput and stability, and as we showed last year, also in quality.

It's not a trade-off. That's why we're all up here, and why DevOps Enterprise is a roaring success, and why The DevOps Handbook has sold north of 30,000 copies since it came out, is because this stuff works and it's transformational. It's not about a trade-off. It's about getting better at all the things.

We're reproducing what happened to the manufacturing industry in the last century, where Toyota didn't win by building crappy cars faster. They won by building higher-quality cars faster and cheaper. And this is what this stuff allows you to do. It allows you to build high-quality software faster and cheaper than the competition.

But you've got to be very disciplined about the way you do that, and you've got to invest in it. And in particular, if you look at this, and we don't have the numbers on it, but what the data says very clearly is it's not about moving everything to Kubernetes. It's about investing in architecture and culture. Those are the things that you ought to focus on in order to get this transformation to work.

And those are hard, and they require singleness of purpose over years in order to get right. If you don't have that at all levels of the organization, from leadership downwards, you won't succeed. This stuff is hard, and it takes time. But it's worth it.

Gene Kim

I don't actually have advice, but maybe more of an observation.

A mentor, Dr. Tom Longstaff from Carnegie Mellon University, said, "The goal of science is to explain the most amount of observable phenomena with the fewest number of principles, confirm deeply held intuitions, and reveal surprising insights."

The one thing I just love about this year's report is around transformational leadership and architecture. The whole Ruly Visbok story from HPE, about he saw work should be done in a certain way. His leadership team resisted. He went to go talk to the developers doing the work and said, "The work is meaningless to me. I don't understand what we're doing. Ruly, you are the worst manager I've ever had in my life. Get me off this team."

He reorganized his teams. Suddenly, the teams were able to actually independently deliver work. Productivity went up. Employee engagement went up. And he was able to achieve his goals.

And I think this structural equation model actually explains both the downward bad state as well as the ideal better state that he actually achieved within six months, which is essentially the heart of that story. So super cool.

Actually, one quick question. Where is employee Net Promoter Score in this?

Dr. Nicole Forsgren

Employee Net Promoter Score doesn't belong in a structural equation model because it's not a scalar variable.

Gene Kim

Nerd. That's what I thought.

Dr. Nicole Forsgren

But we do find that the high performers are most highly correlated with employee Net Promoter Score.

Jez Humble

And what's not in this diagram, because we didn't actually measure it this year, but we measured it in previous years, is culture.

Dr. Nicole Forsgren

Team culture.

Jez Humble

Team culture measured using Westrum's model. And we found that continuous delivery predicts culture, which predicts both IT performance and organizational performance.

My favorite paper on cultural transformation is by John Shook. He talks about lessons from transforming NUMMI. So if you Google NUMMI, John Shook in MIT Sloan Management Review, you'll find this article where he talks about lessons from changing a culture at NUMMI.

And what he says is the way you change the culture is by changing what people do and how they behave. And the way you do that is by giving them the tools and the skills and the ability to build quality into their work using these technical practices.

So implementing the technical practices and the leadership is what changes the culture, and that's what creates the IT performance and organizational performance outcomes.

Dr. Nicole Forsgren

Yes. And lots of research suggests that transformational leadership would be highly correlated with and predictive of changing culture, but we have to keep the size down, because no one's going to take a 25-minute survey. They are already mad at me at 20 minutes. So we did leadership management instead of team management this year.

Gene Kim

Awesome. Anything else?

Dr. Nicole Forsgren

Yes. No.

So our final point is thank you so much for all of your input and ideas over the years. Gene's request to all of us who speak is, if we have any final questions or requests from the audience, it would be: please continue to engage with us throughout the year, because research design starts for us in November and December.

So if you have any pressing concerns or questions, you are the ones who help us shape our future research. Thank you again for always reaching out.

And you can download the State of DevOps Report here.

Thank you.