Gene Kim

Let's start with the State of DevOps Report findings.

I shared some of the findings in the earliest part of the session, but I want to invite now Dr. Nicole Forsgren, who has worked with us for the last three years, to really walk you through all of the key findings and share with you the process of how we actually made those findings, because I think it's valuable ammunition that you can bring to bear as you help build the business case around DevOps.

So, Dr. Nicole Forsgren.

Nicole Forsgren

Thank you. Thanks so much for a few minutes.

As he mentioned, I'll be talking through a few of the key findings from the report and the lengths that we go to to make sure that the report findings are valid and reliable.

First, I'll do a quick intro. So who am I? I spent my early career as a software and hardware engineer at IBM, also as a sysadmin supporting all of those systems. Then I went and got my PhD in MIS. I have a master's in accounting. Now I'm a director of organizational performance and analytics at Chef.

So in short, I rub science on things.

In 2003, Nicholas Carr told the world in Harvard Business Review, "IT Doesn't Matter." But I think we've seen over the last couple of days now that it absolutely does. We're really delivering value to the organizations where we are, and really the last few years of the State of DevOps Reports and other studies coming out of MIT have told us that IT really does matter.

The 2014, 2015 study showed us that high-performing IT organizations are twice as likely to exceed profitability, market share, and productivity goals. We also saw a 50% higher market cap growth over the previous three years out of the 2014 report for companies, or for teams, that could provide us with stock ticker symbols. This is huge, and these were statistically significant results, which was really exciting.

As we move forward into the 2016 results, we see higher-performing organizations really, really outstripping their lower-performing peers.

When we take a look at the speed, and I like to point out that when we look at our IT performance, we look at both speed and stability because we joke that if you only look at one metric, it's really easy to game one metric. So we look at both speed and stability.

When I look at speed, I see 200 times more frequent deployments, high versus low performers, and I see 2,555 times shorter lead times. That's code commit to code deploy. Think about what this means in terms of your own organization: putting that code in, being able to get features to customers, being able to pivot quickly when the market changes. This is really exciting.

Then let's take a look at stability. That top right: 24 times faster recovery from failures. When something does go down, we can recover really, really quickly. And then a three times lower change fail rate. If we do introduce a piece of code that causes any kind of impairment at all, we can recover from that really quickly, and those changes that introduce impairments are really, really rare.

Moving forward, security is addressed at every stage. So we include InfoSec in design. We include security in our automated testing all the way through that software development pipeline. And what happens is our high performers spend 50% less time remediating security defects. So baking in security early, we're seeing the results.

Something else that was really exciting: employees in high-performing organizations are 2.2 times more likely to recommend their organization as a great place to work. This is our employee NPS, net promoter score. We know this is also worthwhile. We've seen research from other organizations come out showing that employee NPS is also related to revenue and organizational performance. So IT performance also matters here.

Now, this one's also exciting. Building quality in matters.

Because high performers build quality in fast, they have good automated tests. They're looking and catching defects early. They spend less time on rework than their low-performing peers: 21% of their time versus 27% of their time.

Also, take a look at what that means in terms of new work, delivering new features. They get to spend 49% of their time, almost half their time, on new work compared to low performers that are only spending 38% of their time. These are all statistically significant differences. Think about what this means in terms of sheer cost that you have to spend.

Gene, I'm going to pass it back to you just for a minute.

Gene Kim

Awesome. And by the way, it is so fun to be working on the State of DevOps Report.

Before, it was with Puppet, with Jez Humble and myself, and three years ago we brought in Nicole, and I would say the level of rigor and the excitement then was just increased phenomenally.

I'll tell you, I want to repeat one of my favorite findings, though, from last year, which I think shed a lot of light on the deploys per day metric. What I love about this finding is that I think what it shows is that deploys per day is actually hiding a more important metric, which is deploys per day per developer.

So the graph that I love is this graph. On the Y-axis, it shows the number of deploys per day, and the X-axis shows the number of developers. And what we found was that in low performers, as you increase the number of developers, the deploys per day went down. In medium performers, the deploys per day was constant, whereas in the high performers, the deploys per day increased linearly as you increase the number of developers.

So this is the opposite of what Frederick Brooks taught us in The Mythical Man-Month, right? In other words, in general, our common experience is if we double the number of developers, we double the code integration effort, we double the testing effort, we double the deployment effort, right?

And that's why breaking The Mythical Man-Month is so important. And what this shows us is that under certain conditions, with the right architecture, the right technical practices, the right cultural norms, we can actually break The Mythical Man-Month. We can scale developer productivity linearly with the number of developers. And there's no technology leader who doesn't care about that, whether we're dev, test, ops, or even information security.

So that's the highlight for me. Not that I don't like the other findings either.

Nicole Forsgren

They're all good.

This slide I threw in here because I think it's important to note that these findings apply to this audience as well. We had strong responses out of Europe this year as well, and we typically do. So these findings apply to us as well.

Now, one note I just wanted to mention at the end: this report is different from many of the vendor reports you find out there, and it's because I mentioned I have a PhD. For years, I was a tenure-track professor. So when I first joined the team, I was slightly selfish. I wanted data so that I could get peer-reviewed academic articles. I still do, actually. So we have a few publications off of this. I still do peer review off of the data in this.

So the report is based on academic, theory-based research design. And before I even start looking at how the data relates to each other or relates amongst itself, I go through a very, very rigorous process. And the team, we feel very strongly that we want to make sure that the results and the report are absolutely rigorous and valid and reliable.

So we go through a multi, multi-step process. We have rigorous validation of the data itself. We test several biases in the data, and then we go through multiple steps to test the measurement model, which means I check all of the constructs, all of the data. I set all of that up before I even start looking for correlations, prediction, and prediction is only tested if it meets a certain standard.

And if you want to come meet with me later, we can play statistics bingo, and I can share with you all of the very exciting things I do. I kind of nerd out, but I know I only have a certain amount of minutes here. So thank you very much for your time.

If you want to find more out about the report, you can go to the website there, get your own copy. It's free to download at the Puppet website. The report is put out this year in cooperation between Puppet and DORA. DORA is me and Gene and Jez Humble, and you can find out more about the report.

There's going to be a forthcoming ROI whitepaper. The very end of the report this year has an ROI summary. I'll be breaking out in much more detail the methodology behind that in a few more weeks. I need some sleep.

And we have some of our peer-reviewed research up on the website as well.

Gene Kim

Awesome. And before you go, Nicole, what help would you be looking for?

Nicole Forsgren

I'm always interested in hearing what your next concerns are, what your pain points are, what you might like to see in next year's survey, and as a researcher, I'm also interested in data.

Gene Kim

Awesome. Thank you so much, Nicole.

Nicole Forsgren

Thanks so much. Thank you.