Dr. Nicole Forsgren

I'm Nicole Forsgren, CEO and Chief Scientist at DORA. And what I'm going to be talking about is some of the data that we like to talk about so that we can really verify some of our intuitions.

Because we work in these organizations. We've heard stories from so many people and so many of our peers, sometimes here at DevOps Enterprise Summit, sometimes at other conferences. Sometimes we hear them from our peers as we're trying to do these transformations. And so often our intuition is right, but sometimes it's not. And so what we really need is data to verify it.

So much of what I'm going to be talking about today comes from the State of DevOps reports. I'm the lead investigator on these reports. The DORA team is myself, Gene, you all know Gene Kim, Jez Humble. The last few years we've done it with Puppet, and this year we're doing it with Google. So let's dig in and see what we find.

I'll mention that I've summarized all of this work, me and Gene and Jez, in Accelerate. Some people ask me, "Why?" Why did I do that? I don't like writing. I really should have thought about this before the PhD. What I would much rather do is sit around and drink Diet Coke all day.

But so often people would come up and they would ask, "What did you find? What's most important? What do I really need to do to help transform my organization?" And so, again, that's what we'll cover today.

So let's start today's story with Harvard Business Review. Harvard Business Review is really where so many people in organizations, and particularly management, read to find out what's happening and what's happening in research. So I have a PhD. I read research papers for fun, which might be a weakness.

But research papers can be really difficult to get through. Harvard Business Review is great because it presents them to us in easy-to-read snippets with key findings. Now, some managers use this as the only place to get their ideas, although I know this is no one in here. But some people just read it for their ideas.

Well, in 2003, Nicholas Carr wrote a paper titled "IT Doesn't Matter." The title of it was "IT Doesn't Matter," and so many people in leadership and executives read this paper, or read the title of the paper, and then just stopped reading and decided that they were just done. They were done investing.

And so that became a real challenge, because for those of us that are doing technology, we were like, "But wait, we know this is important, so this paper can die in a fire." This was really, really frustrating. And so we really needed to move forward and find ways to have data and prove that what we were doing was making an impact.

So today's talk will be going through some of the highlights that I've found, some of my favorite surprises running this research over the last four years.

So first is technology matters. Going back to Nicholas Carr, he actually had a point. He had a really, really good point. And what this point is, is the way that so many people were doing technology up until the early 2000s, coming out of the '80s and coming out of the '90s, was we were acquiring technology, and we were just plugging it in, and then we were walking away.

If I can do that, if it's that easy, if I can buy a solution off of a shelf, and if I can plug it in and I can walk away, my competitor can do the exact same thing. It no longer brings me differential value. It no longer helps me get ahead, because if someone else can do the exact same thing, they can catch up. The only thing that helps me is if I buy it sooner or buy it faster, which might be a few months.

So in that case, IT might not matter, unless it's saving me from doing something by hand that I shouldn't be doing by hand anyway. And so if this is the case, then, yeah, you're doing it wrong.

And it's really interesting to me, and by interesting I mean laughable and I roll my eyes all the time, when we see so many vendors come to the plate now and say, "I'm going to sell you DevOps in a box." I'm like, "Really? This isn't going to work."

If it's only a technology solution that is really easy that you can buy and walk away from, we're just going back to the '80s and '90s again, when we saw these large initiatives, if not fail, not deliver substantial value to their organizations.

Although we all kind of know this, right? But the pushback that we get when we're trying to get resources so we can make smart investments tells us that we need some more evidence and proof.

So if we look back over some of our history, one of the great talks that really helped set us off was "10+ Deploys per Day: Dev and Ops Cooperation at Flickr." So Allspaw and Hammond were telling us about this early on, Velocity 2009.

But what they were telling us were what a handful of other companies were also discovering. It does take those technical practices, but it also takes lean process. It also takes culture. It's difficult. It's hard. We do need to share these stories, but when we do it right, it drives performance outcomes. And it drives performance outcomes along a few different areas.

Now, again, intuition may fall short, though, and I saw this myself when I was in software development. I would go to my management team and I would say, "Listen, I've got this really great idea, and so many other teams have done this." And leadership would say, "I just can't allocate resources because we're not that team. You're not that team. I need better evidence. I need a better business case." Right? Who here has heard that? So many times. And so we need data.

Well, this is what we find. DevOps, this mix of technology and process and culture, is good for delivering technology for the business. By the way, this is how we measure it when we run the State of DevOps reports.

Software delivery performance is developing and delivering software with both speed and stability. I think it's important to notice I said speed and stability. So how we measure speed is deployment frequency, or when the business demands. This does work for things like packaged software. Lead time for changes. Now, here we're focusing in on code commit to code deploy because this is the most predictable and consistent part of the pipeline that we have.

Now, when I talk about stability, I'm talking about MTTR, mean time to recover, mean time to restore, and change fail rate.

We see that by investing smartly in developing key capabilities across tech and process and culture, it improves software delivery performance across all of these measures and all of these measures together. High performers are doing well at all of them. Low performers are really struggling at all of them. Medium performers are hanging out in the middle. And these are all at statistically significant levels. And we see it four years in a row. Oh, five years in a row. The fifth year's data is in.

Here's what we see. High-performing DevOps teams are more agile. We see 46 times more frequent code deployments. You can push code multiple times a day as opposed to a week or less. By the way, these are median numbers. I see high performers moving much faster. I see low performers moving much slower.

This is also the difference in lead times. 440 times faster lead time, the time from code commit to code deploy. It's the difference between less than an hour and more than a week.

Some people say, "Why do I need to get my code through the pipeline that fast?" Well, we may want to delight our customers. We can pivot when we need to pivot. We can keep up with compliance and regulatory and security changes. Or we can get that emergency fix through our regular pipeline and not through an emergency pipeline that means that we've skipped our tests, which is probably not the case we want to be following when everything's on fire.

We also see that our high-performing teams are more reliable. We see 96 times faster MTTR. Now, this stat scares me just a little bit, because my high performers are recovering in less than an hour instead of several days. Low performers are still down for several days.

We also see that high performers are only a fifth as likely that a change that they introduce into the system will fail. By the way, this isn't a big, huge, high-impact failure. This is any change that requires any type of response.

Now, again, some people might say, "Why do I care about this? Why do I care about delivering code? It's fine." It's because DevOps is also good for organizations. We see that high performers are twice as likely to meet or exceed commercial goals, productivity, profitability, market share.

And in today's environment, these capabilities also help us deliver more than just commercial goals. They also help us with things like effectiveness and efficiency, customer satisfaction. And for bonus points, we were able to collect data and see that our high performers achieved 50% higher market cap growth over the previous three years. So it really is delivering on the bottom line.

One thing I really want to mention that I hope all of you get out of this. Software delivery performance is both throughput and stability, and both are possible without trade-offs.

For years, we were told that in order to be stable and be safe, we had to slow down. This doesn't appear anywhere in the data. High performers are maximizing and optimizing on everything. Low performers are having a really difficult time at everything.

This is where we need data and research to help us test and confirm our intuitions. And if anything, it may be that slowing down makes it even harder. Actually, it probably does make it harder to be stable. We see really good evidence of this in the data.

But how do we get there? How do we actually get better?

Let's talk about maturity models. Maturity models are things that someone, sometimes consultants, sometimes a center of excellence, sometimes someone, we make up. We invent these to describe how you go from sucking at technology to being amazing at technology.

So on the off chance you haven't heard of a maturity model, these are basically what maturity models are. It's just a rubric. They usually go to level five. I don't know why everyone likes level five. This is the essence of a maturity model.

Now, here's the challenge with maturity models. They don't capture the complexity of the way we work. And they don't account for changes in the system and the environment. So let's cover this with an example.

Who here has played World of Warcraft? Who here has heard of World of Warcraft? Has a friend who plays? Awesome. Okay. We all have friends who've played World of Warcraft. I had a friend who needed to stop playing World of Warcraft so she could finish her dissertation. It was great.

So when World of Warcraft first came out, it went to level 60. And there were two lands. So if we are inventing ourselves a maturity model, because again, they're all invented, which is fine, we are creating a rubric to talk about sucking at World of Warcraft to being amazing at World of Warcraft.

So one level is good, and let's call it we get to 40 and we conquer one land. Let's conquer the left one, which I'm not even going to try to pronounce because someone will come tell me I did it wrong, especially on the internet.

And then level great, we get to 50, and we conquer half of the second land, Eastern Kingdoms. I'm sure someone will still tell me I said that wrong. And then to be amazing, we get to level 60 and we conquer both lands. Welcome to my maturity model. I will now charge you massive amounts of money to help you get there.

Well, here's what happens. We wait a few years, we dedicate resources, we get there, and things are amazing. And we check it off our list, and we collect our fat bonuses, and we are good. Everything's happy. We move on to the next project.

The challenge is that several years later, the world has changed. The landscape has changed. World of Warcraft now goes to level 110. And there are additional tooling and technologies available. There are pets. There are additional characters you can roll, you can become, you can inhabit.

And so if we had relied on our original maturity model, getting to amazing level, level 60 and two lands, the rest of the world has passed us by. And if we think about it, this sounds a lot like technology, right? The technology landscape changed. The business landscape changed. What customers expect of us and expect of technology and service has changed.

We didn't keep up, right? We weren't talking about Docker or Kubernetes or containers five or 10 years ago, not the way we are today. Customers were not expecting to deposit checks on their phone, not even expecting, demanding, five or 10 years ago. But if we no longer offer that capability, we are probably out of business. And that's for a bank. That's not for a technology organization. But so many companies that are not in technology are fundamentally technology organizations now.

And so these are some of the limitations. Maturity models tell us that we're done once we reach a destination, and the rest of the world passes us by. The best, most innovative organizations understand that they must keep improving. They must keep reassessing the climate, the context, the environment, the competitive landscape, their customers, compliance and security and regulatory risks, and they must keep improving.

Maturity models tell us and executives to stop dedicating resources once we've arrived at that end state. Now, resources are more than just money. It can be time and attention. What happens when our technology has arrived and we're done, and we no longer get time for improvement or training or anything else?

How often do you get to done and decide that your business is no longer going to do accounting? That doesn't happen, right? We don't move off of our HR or accounting initiatives. Why would we move off of our technology initiative? That's bizarre, when we know that technology is driving our business.

Maturity models also tell us that we all follow the same path to success. So often they follow the exact same lockstep linear process to success. But how many of us in here are in large organizations, what you would consider a large organization with more than 10 teams building software?

Almost all the hands. Think about those teams. How many of them look exactly the same, and you would tell every single one of those teams to improve in the exact same way? It doesn't always happen that way.

We quite often see, particularly in enterprises, teams at very different stages and very different phases, particularly as we get to really large organizations and very mature organizations like we see here in Europe, that have mixes of legacy software and brownfield and greenfield, because maturing those technology stacks will necessarily look very different. We expect that to.

Now, maturity models also tell us that technology is a checklist to be completed and not an exciting journey to continually explore and improve. Again, this isn't something that we just check off the box and then be done, like, "Finally, I'm over it."

This should be fun. One reason I love the World of Warcraft analogy is everyone gets so excited when a new release comes out. There are more lands. There are more things to conquer. There's more to do. It's not like, "Really? I have to invest again?"

No, we get to find more exciting, amazing ways to deliver content to our customers. We get to find more ways to make their lives amazing. And if we want to be capitalistic, yeah, we get to find more ways to make money. Or if we're in government or not-for-profit, we get to find more ways to be amazingly efficient and effective and make people's lives incredible, leveraging software. And once we've done that, we can move on to the next thing and make even more things amazing.

This shouldn't be something that we're super bummed out about. And if it is, maybe we should do something else. Okay.

Another thing that I found that was really fun and kind of exciting as we were going through the research over the several years: architecture matters. Technology and technology stack, not so much.

Here's what we found. Yes, it's true that low performers are more likely to be working with outsourced software, and they're more likely to be working on a mainframe. So for all you all working on mainframe systems, that's cool. Also, but, there's a giant but in the middle of the screen. Everyone sees this?

However, there's no excuses here because when I did more analysis, there was no statistically significant correlation to performance. I don't see legacy systems showing up only in low-performing groups, nor do I only see startups and greenfield systems appearing in high-performing groups.

Working on a mainframe system isn't correlated with performance. Working on greenfield and brownfield systems isn't correlated with performance either. I really want to put a big sticker across the slide that says, "No excuses."

We've seen DevOps teams working in mainframes. We've seen DevOps teams working on firmware. What's important is how you architect your system.

Architectural outcomes matter. So can your team change the design of its system, test the system, deploy the system, without having to do fine-grained communication and coordination with groups and people in other teams? Because that's where we get a lot of friction. That's where we get a lot of time lost. Comes down to architectural outcomes.

Now, granted, it will be more difficult to implement in very old legacy codebases and mainframes. It's going to be easier if you're starting from scratch. But I've seen plenty of poorly architected monolithic apps built in the last couple of years. If we don't think about this from a smart architectural standpoint, we need to focus on these things.

Which sounds a lot like Conway's Law, because I hope you noticed that that first point was also about communicating with your team. I think this wins me DevOps bingo, right? You have to talk about either Conway or Deming. I'm halfway there.

Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations.

We need to make it easy for teams to do their work, communicate their work, test, and deploy.

Now, another thing that we found is that leadership matters. Which, yay, leadership matters.

Which was interesting to test, because traditionally and historically, DevOps was a very grassroots movement. And so we were getting very genuine inquiries about, now that DevOps looks like it's really important, our organization is investing in leadership. How do we do this? How do we do this right? Does leadership actually matter?

And here's what we find. It is important. And we measured this with a particular model of transformational leadership using five dimensions. We pulled this one out of the literature because it was shown to be predictive of performance outcomes in other contexts. And so we wanted to try to apply it to technology as well. And it does work.

Here's what we find. This five-factor model includes things like vision. Do you understand and can you communicate the organizational direction and the team direction? And do you know the five-year horizon for your team?

Now, intellectual stimulation. Can you also challenge what's happening, the team status quo? Can you help your team ask new questions, and can you challenge the team on basic assumptions?

Inspirational communication. Can you inspire pride on being a part of the team and say positive things about your team? Inspire passion and motivation.

The next one is supportive leadership. Are you supportive of your team and consider their personal feelings? And are you thoughtful of their needs and interests?

The last one is personal recognition. Do leaders commend the team for better-than-average work and acknowledge improvements in work? Which I think is particularly important when we're talking about continuous improvement.

Now, this was the 2017 research model, which I've included because the giant diagram is a little bit overwhelming. What I want to point out here, I'm not going to walk through the entire model, but look at transformational leadership. It's on the far left here. That doesn't mean it's most important, but what it does mean is that it underlies the rest of the model, and it underlies everything else that we find.

Now, everywhere you see an arrow, you can read that as predicts or drives or impacts. So what we see is that transformational leaders impact all of the technical work that we see there, which then flow through to continuous delivery. They also impact lean product management work, and their influence, their impact is seen in IT performance, that ability to develop and deliver software.

It's also seen in the work deployment pain, which I'll talk about in a minute. And it flows through into organizational performance, those measures of profitability and productivity and effectiveness and efficiency.

So an effective and strong and great leader, their influence is felt throughout all of the work that we do. They help amplify and support the work. Similarly, a poor leader, their influence will be felt negatively throughout the whole system. So strong leaders really help support all of this work.

Now, one of my favorite findings is that we can make work better. This was important to me because I remember having to do seven-day forced marches. And I included this here because it's one of my favorite memes. And again, I think this helps me hit DevOps bingo, because you have to have a great DevOps meme here.

But again, I had to do seven-day forced marches for a long time when I started in tech. And we joke about how it works fine in dev, it's ops' problem now. But in theory, our intuition and our stories tell us that doing the DevOps makes everyone's work better. But what does the data tell us? Does it actually work?

And here's what we see. Smart investments in tech and practices, by the way also in culture, because this does feed through, makes the work better. And what I mean when I say work is a few things.

First is it does decrease deployment pain, which you saw in that 2017 study. Earlier studies show that it also helps decrease burnout. It also helps improve Net Promoter Score. So when employees recommend your workplace as a good place to work.

I was delivering some early results on this, and Diego Almeida came up to me and he said, "Oh, my word, I just talked about this actually here in London." And he said, "Microsoft found something very, very similar. We made investments in improving our technical practices around continuous delivery." And in just one year, their work-life scores shot up. They went from 38% to 75% because people could focus on their work. They weren't always stressed. They weren't always anxious. And then they could leave their work at work, and they could go home.

Now, for the NPS metric, we find that employees in high-performing organizations are 2.2 times more likely to recommend their organization as a great place to work. That's really telling because it says that I really love what I do, and I want to tell everyone else about it, and maybe I even want them to come work for me.

Is anyone here hiring, or is your organization hiring? Lots of hands. This also helps you with recruiting.

So some of my other favorite data findings, which I'll hurry through real quick. Change advisory boards are useless. Yes. Yay. Okay, not quite. There's an asterisk there. I'm not going to say they're useless. They're pretty good for notifying other groups. They're really bad at everything else, though.

They're actually negatively correlated with stability and software delivery performance. They have no effect on certain aspects of software delivery performance. What we do find that works is having a very, very lightweight change approval process or no change approval process. Because if we can speed that up and get fast feedback, that's what delivers superior performance.

Also, we found that integration times and branch lifetimes lasting hours are statistically significantly better than days. I know there's a bunch of people who hate me for that. That's okay. Come find me later. Come yell at me later. I'll just point at my stats. It's fine.

Now, you can help. What's your role in this? You can be the transformational leaders. And I do want to point out that everyone here can be a leader. You don't have to have direct reports here, which we did see historically with DevOps being a grassroots movement.

Also, I'm a data wonk. So of course, I'm going to tell you to measure a few things. I like data. Focus on outcomes. What's the goal on your team? And then measure a few things that you think will impact that outcome. Try to measure a couple of things in tech and a couple of things that aren't tech.

And share your stories, because we get better by sharing stories. We get better by figuring out what those intuition pieces are.

What else can we do? Evangelize this internally. Start small. Focus on the team that you influence. Grow from there. Maybe build a seed community. Drive some organizational change through communities and learning groups.

Also, supporting efforts across and external the organization can be really strong. Go to meetups, go to hack days. Utilize resources that you can get from each other or that IT Revolution has provided. There are plenty of white papers. And again, share your stories. I'll keep saying share your stories because that's how we end up getting so much better as a community.

And again, intuition is good, but we really need to confirm with data and science because sometimes we confirm and sometimes we find surprises.

So as a note, the 2018 Accelerate State of DevOps Report is coming soon. It'll be released the end of August. This year we're partnering with Google, and I'm really excited about some of the results. I've been cluing in Gene and Jez to some of the things we've been finding, and it looks very exciting. So stay tuned.

The TL;DR, in case we've all just been tweeting and FaceSnapping and everything else, is technology matters. Maturity models don't work. Architecture is more important than technology and technology stack. We can make work better. And you all can help.

So thank you so much, and I'll be around for questions later.