Nathen Harvey

Alright, welcome in, everyone. How's your morning going so far? Great. Are you ready to learn about solution and enterprise architecture and team topologies and bringing them all together? That is not the talk you're here for. Well, it might be the talk you're here for, but it's not the talk you're going to get right now. There were some travel mishaps — these things happen. I encouraged the organizers to just send me his slides and let me present them this morning. They didn't seem too into that. They said I should just give my own talk. So here we are. We're going to have a talk about a decade with DORA.

Anyone ever done Ignite Karaoke? That's kind of the same idea — you just get up and give a talk and have no idea what the slides are going to be. But I do have a little bit of an idea of what these slides are going to be. I've already wasted 40 seconds of your time with banter, so let's get started, because we have a whole decade to talk about. It's "A Decade with DORA."

Before DORA — 2009

If we're going to talk about DORA, we have to go back to before DORA was ever around. In fact, we're going to go all the way back to 2009. Was anyone at Velocity Conf in 2009? Me either. But there was this talk there given by John Allspaw and Paul Hammond, titled "10+ Deploys Per Day: Dev and Ops Cooperation at Flickr." Has anyone heard of that talk? Yeah, many of us in this room have heard of that talk.

In that talk, John and Paul reminded us that ops's job is to enable the business. Oh, and by the way, that's also dev's job too. We're all here to enable the business to create great experiences for our customers. And they said our job is to lower the risk of change. We lower the risk of change through tools and culture. So right there in 2009, this whole movement was really born. It was the time they put this seminal slide together — dev and ops together on a heart.

Later that year, Patrick Debois and a few others got together and held the very first ever DevOpsDays event, in Ghent, Belgium. In about two weeks, the 15-year anniversary of DevOpsDays is going to be held. Anyone planning to be there? Good. I'm giving this talk again there. So you probably don't want to show up there either. Although it's not in Ghent this year — they moved it to Antwerp. They're calling it "Ghent East," I think. I don't know my geography or my math very well, as we started in 2009 for a talk about a decade.

Why DORA exists

So, my name is Nathen Harvey. It's really an honor to be here. Before we go any further, I want to ask everyone here for just a brief moment — can we get a round of applause for Jeff and Elm? They're in the back there running the show. As a former system administrator and operator, that's invisible work that happens, and we only notice them when stuff goes wrong. So I like to celebrate the people behind the stage as well. Thank you for doing that with me.

That's me. I'm Nathen Harvey. I'm not wearing my avocado shirt today, dispelling all myths that I only have one. But if you come to the workshop I'm running with Laura tomorrow, it might just make an appearance.

Before we go back into the history, I just want to talk about what DORA is all about and why are we here. We need to understand: how can your organization optimize value delivery from investment in technology and technologists? I think it's important that we're investing in both — we're investing in the people and the tools. The tools and the culture, they all go together. In order to do this, you're going to need a way to assess, how are we doing today? You're going to need a way to prioritize the work that you're going to take on next. The truth is, the work that you want to do, the work that you need to do, is a much longer list than what you have capacity to do. So we need some prioritization framework. And then probably most importantly — well, second most importantly — we need a feedback cycle that'll tell us how are these changes going, so that we can understand as we're making changes, is it helping or not? The most important thing, though, that you're going to need is the organizational muscle and institutional culture to put this into practice.

What we saw on the plenary stage this morning — we saw organizations that are doing exactly this. They maybe didn't use these words, but we saw echoes of this in all of those talks. This is where I like to talk about this idea of get better at getting better. This is the fundamental question. How do we get better at getting better? That is continuous improvement in a nutshell. And this is where DORA comes in. So everybody say hello: hello, DORA.

The timeline — 2010 to 2017

Alright, back to our history. We're going to do a little timeline. I'm going to go as fast as I can through the decade. And again, apologies for the math — 2009 was more than 10 years ago, I'm pretty certain. But there were some weird years in between, so who knows.

2009. Hammond and Allspaw giving this dev-and-ops talk, "10 Deploys a Day at Flickr." If you remember that time, everyone thought they were crazy, completely mad. How can you do 10 deploys a day? It was a ridiculous and scary thought. As their talk showed us, they were doing that in order to decrease risk, and everyone knew that they were being cowboys and just increasing risk everywhere and everything was going to fall apart. It didn't.

2010. DevOpsDays — this DevOps thing starts gaining a little bit more momentum.

2011. A woman by the name of Alanna Brown at Puppet Labs said, "I'm in marketing — we should do some marketing around this whole DevOps thing and try to understand it a little bit better." So she ran a little survey, and from that survey she published the precursor to a State of DevOps Report. This report is just a presentation, just slides. But it was about the evolution of DevOps. Is it an emerging discipline or is it buzzword hype? Now the truth is, we could have asked that same question last year. Is this an emerging discipline or is it buzzword hype? And we've determined that it was just buzzword hype — that's why this is the ETLS now, no longer DOES. But the interesting thing — I picked a couple of favorite quotes — "DevOps is an emerging way, a new way to do business in IT." So it's always been about bringing the business and technology together. And in fact, they aren't different. The business is technology. Technology is the business.

2012. Alanna's like, "Hey, that was pretty cool. Maybe we should do an even broader survey." So at the end of 2012, they launched a survey worldwide.

2013. Two things happened. First, Gene Kim published The Phoenix Project, written together with Kevin Behr and George Spafford. How many of you have read that book? We all have stories about The Phoenix Project. Unfortunately, all of us have lived some of the stories of The Phoenix Project. Also in 2013, the very first ever State of DevOps Report was published by Puppet Labs. This really started things off. In this report was the first time that the four key measures of software delivery performance were introduced. Interestingly, they compared organizations based on "How long have you been doing the DevOps?" — and the people that had been doing DevOps longer had better results with those four key performance measures.

2014. Dr. Nicole Forsgren meets the team at Puppet Labs and decides, "Hey, I'm in academia. I want to come join this effort. I love what you're doing here. Also, there could be more science. So let's bring more science and rigor to what you're doing." So Nicole joined the effort. They published a 2014 State of DevOps Report. One of the coolest things that happened in the 2014 report — in the survey, towards the end, they said, "Hey, thanks for taking our survey. Just one final question — if you work at a public company, would you mind sharing your stock ticker?" Then they did an analysis of all those public stock tickers, and they saw that the teams with the highest performance saw a 50% higher market capitalization growth over three years. This was really the first definitive way for us to claim as an industry: software delivery performance matters. It matters to the business. It helps drive organizational performance.

2015. What did we learn? One of the things was that as teams were focusing on throughput and driving more changes, maybe there's some economic value to be gained by also focusing on stability. Maybe you've reached a throughput maximum and it's time to really focus on that stability side as well. Again, these ideas of speed and stability playing together.

2016. Another report, the 2016 State of DevOps Report. In addition to the report, also, this organization, this company, was founded — Jez Humble and Gene Kim joined up with Nicole Forsgren, who is the main founder of DORA, DevOps Research and Assessment. So the 2016 report is actually a report published by Puppet and DORA. This is the first time that DORA comes in. Now, if you're doing that math like I failed to do, 2016 is not 10 years ago either. So where is this "decade with DORA"? My personal take, just so you all know — let you inside my head for a minute — when Nicole Forsgren joined the effort, that's when I sort of consider it the beginning of DORA. That was 2014. We're now in 2024. So I call it a decade with DORA. This year we will publish our 10th DORA report. But if you do the math, that would actually be the 11th DORA report, wouldn't it? I'll explain that part in just a minute. You'll trust me, I'm sure.

In this report, they looked at: employees on high-performing teams were happier. This matters. We want to keep our employees around — people in our organization know how our organization works and what's going on.

2017. Another report. This report looked at transformational leadership. It looked so loudly at transformational leadership. Can you still hear me in the back? Okay — it felt like my mic just cut out there for a minute. Thanks, Elm. Again, a round of applause for the people behind the curtains.

Also in 2017, the book Accelerate gets published. This is a book that everyone in this room has read, right? If you've read this book, raise your hand. If you haven't, lie to me. There was a challenge with many people that read this book: some of you just lied because you got to page 17, where they talked about the four key metrics, and you're like, "Damn, I got it. This book can go back on the shelf. We're going to go do those four key metrics." What you failed to do was get all the way to Appendix A. This is the meat. This is the beauty of the book. This is the structural equation model — that's what those data analysts call this thing. And this shows the connection of all of those capabilities and how they all work together — it shows the predictive relationship. Transformational leadership predicts better test automation, predicts better continuous delivery, drives better organizational performance, et cetera, et cetera. This is the important piece. We all hyper-focus on those four metrics, and we tend to forget the capabilities that help you get there. You don't get faster at deploying. If you think about deployment frequency every morning when you wake up, you don't get faster — you get faster because you do these capabilities, and these capabilities grow over time.

2017 fork, 2018 Google Cloud acquisition

We hit a fork in the road in 2017. AI came out and generated an image for me. Isn't it beautiful? Isn't this a glorious fork in the road? It couldn't be a talk here without some AI in it. If you're wondering: the fork is that Puppet Labs and DORA had a happy split. DORA said, "Look, we want to continue our own research, and we're a company now, and we're going to keep moving forward." Puppet said, "That's awesome, you should do that. And we're going to publish a State of DevOps Report." And DORA said, "Great. We're going to publish a State of DevOps Report too." And today, if you look hard enough, you can probably find six or eight or ten States of DevOps Reports out in the world published by many different people. But DORA's the one that's near and dear to my heart. So obviously it's on my T-shirt. Very close to my heart.

2018. The Accelerate State of DevOps Report gets published. "Accelerate" is the word we use to help differentiate DORA's report from Puppet's report. It also leans back on the book. There's a beautiful thing that happened in 2018 — I just want you to look at this for a second. The sponsors for the 2018 report. DORA has always been platform and program agnostic. And as proof, in 2018, Google Cloud, Microsoft Azure, and AWS all sponsored the report. If you know the future that's coming — this is the last time that all three of those vendors sponsored the report. IT Revolution also sponsored — they'd been a partner with Puppet Labs and the research all along.

This year also was the introduction of the term "availability" — a term we've later changed to "reliability." We really started to, in addition to software delivery performance, look into operational performance and look at a team's ability to make and keep promises about the services it's running.

At the end of 2018, Google Cloud acquires DORA. This is where DORA lives today. It's where the research continues. I'm very, very happy and honored to be a part of that team, helping drive this research forward.

2019 to 2022

2019. Another State of DevOps Report. One of the big things in that was they looked into transformation strategies. By 2019, this had really been a decade of doing the DevOps. People were like, "Yeah, we see small pockets of success. How do we take those small pockets and get them across the organization?" The thing that really stood out in those high-performing teams was communities of practice. There were certainly teams doing things like dojos and sort of university styles — those take big investments and lots of commitment. Communities of practice take lots of commitment, but less upfront investment and less to keep them going. They're kind of self-sustaining. That's what we learned in 2019.

2020. Then 2020 happened, and it was a dark, dark year. So remember, it would be the 11th report this year, but 2020 happened, so there was no State of DevOps Report in 2020. In fact, it would be best if we just all forgot 2020. Wouldn't that be better? But it wasn't a completely dark year. In 2020, one of the things we did publish was an updated ROI of DevOps Transformation paper. Now, in 2016 they investigated this; in 2020 we published this paper that gives you formulas for figuring out: what is the cost of unnecessary rework avoided each year? What's the potential revenue from reinvestment? As we move forward, we're able to do more experiments. We should be learning faster. Frankly, this is probably the report that is most underutilized — how many of you are hearing about this report for the first time right now? You can lie to me, I'd just like to see your hands up. There were a significant number of people here that this is the first time you've heard of this report. You can get it for freeish — you just have to give us your email address and a couple other details.

2021. Dr. Dustin Smith joins the effort. In previous years, he had been helping Nicole. By this time, Dr. Nicole Forsgren has moved off of DORA. Dr. Dustin Smith is now our lead investigator, the lead researcher. We published the 2021 State of DevOps Report. This was the first year that we looked at documentation, and we've looked at documentation every year since. Part of the reason we continue to look at documentation, including in this year, is documentation has an extremely potent effect on a team's culture, their ability to drive software, and so forth. You can see the numbers here. I'm not going to read them to you, but these were the early numbers we got. Over the years, the numbers just get better and better, and the things we find about the importance of documentation, which you all know, it's really important.

2022. We get a new lead investigator, Dr. Derek DeBellis. I always want to give him an honorary doctorate. So if y'all see him, just say "Hey, Dr. Derek." He's not here. You won't see him this week. We published the 2022 State of DevOps Report. There's a heavy focus on security in that. And who knew — the number-one predictor of strong security practices: it's culture. It's the people. It's how we work together.

In 2022, we also did this amazing thing: we launched the DORA Community — myself and others on my team, Dave and Amanda. We met with lots of teams around the world and we were talking about things and we kept hearing lots of questions. While we have some answers, we don't have all the answers — we can all learn more together. Hopefully you're all part of the DORA Community, because you cannot improve alone. And even if you aren't a part of the DORA Community, you should join it right now — not right now, but when I'm done talking, or at the next break, dora.community, go become a member of that community. Come by the Google booth at 6:30 today, because we're going to take a DORA Community family photo. Sorry for the brief commercial.

2023 — DORA Core

Continuing on. 2023, another report comes out. But before we get to that, we launched this other thing in 2023, and this other thing has to do with that Appendix A — remember that SEM (structural equation model)? Well, in each year's report we would publish a new SEM, and each year the SEM would get a little bit different and a little bit different and a little bit different, because we're investigating different things each year. So how do you make sense of all of these? The best way to make sense of them all is just to squash them all together. Put them on — remember the overhead transparencies? We just made a big set of transparencies, we put them on the overhead projector and said, "Go do that." That's not helpful to anyone.

So we came up with this idea of what we call DORA Core. We launched DORA Core. DORA Core represents those capabilities and other things that we've investigated at least twice during the research program, and things that, through application in the field, we find that practitioners are getting value from. At its core, if I will: capabilities predict performance, which predicts outcomes. This is sort of the thesis of DORA, and how do we bring all of these things together. Would that we all lived in a world where it was that simple. We don't. But make it easy on diagrams — just go do that and everything will be better. That's not as easy as it looks, but yeah.

Oh, there was another thing we did that year, kind of fun. There was this new thing called AI that everyone was getting excited about. We launched this little prototype called Ask DORA Dev, because you've got 10 years of reports — you can go investigate. Why not use a bot to read them all for you and give you some answers? You can go check that out if you like.

2023 State of DevOps Report and AI

2023 State of DevOps Report comes out. This was our first look into AI. We found a lot of other things that were really interesting that year. One of the things I find most helpful from that year is we really put a focus on what we called user centricity — are you building for your users? Are they listening to their users? Do they care about your users? Do they know why they're building software? Those teams with higher user centricity had much better software delivery performance, much better organizational performance, and much better team performance.

We also looked at AI, trying to figure out what was the impact of AI. The TL;DR was: it makes everyone a little bit more productive and a little bit more engaged. Minor improvements to well-being and the rest was, "Yeah, we don't see much yet." So maybe this year — because we looked at it again this year — we'll see more.

2024 — DORA Core 2.0

Now here we are at 2024. We listened to the community, and we saw: "Hey, that DORA Core — it's really good, we love it, but also there are some things that we'd like to change, and you've investigated some new things." So we launched DORA Core version 2.0. So all of you who took pictures of the first one, sorry, that's a legacy photo now. But don't worry, you can modernize right here in the room together with all of your peers.

This boils it down kind of simply. This is the summary view — maybe the view you take into the boardroom. But then we dig in a little bit deeper on these areas of learning, flow, and feedback. We blow it up so you can get a little bit more of the capabilities. Of course all of this is available on dora.dev.

What we've learned over the decade

So that's kind of the history of DORA. That was meant to take about 10 minutes, and it took more than 10 minutes. But here we are — we're a decade in, more than a decade in, and we've got this DORA. What have we learned over the years?

First of course, everyone knows DORA for those four key metrics of software delivery performance: two for throughput, two for stability. We set out on this journey to figure out — is this sort of known belief that if you want to be fast, you have to be unsafe; and if you want to be stable, you have to go slow? That's how we started in 2009 when Hammond and Allspaw gave that talk. We thought they were ridiculous. What DORA has shown us over the decade-plus is that speed and stability feed each other. Throughput and stability drive one another. If you want to be fast, you have to be stable. If you want to get more stable, you have to get faster. That's just how it works. And we see this continually happening.

We also over the years — and by over the years, I mean last year — changed that fourth measure from "time to restore" to "failed deployment recovery time." You've got to read the appendix of last year's report to figure out why, or just come talk to me later. I have no time to tell you now.

When it comes to those measures, everyone's like, "Yeah, of course there are capabilities, but I've got to measure at least four things. How am I going to do that?" Well, the first way to measure those four things is to get together in the room with your colleagues and ask the questions out loud and have a conversation. Just talk about them. That will tell you how you're doing.

I know that's not enough for you. That doesn't scale well. So let me give you some tools. The first is the DORA Quick Check. If you haven't seen this, it's at dora.dev/quickcheck. We make it easy to find. You answer four simple questions. We'll tell you how you're doing relative to your peers. We'll give you some other insights as well.

The other thing is, people are like, "Well, you told me first to have a conversation, then you told me to fill out a survey. I want data. Nathen, give me the numbers." Alright, well, you can have the numbers, you can have a dashboard. Just use any one of these tools. Every one of these tools is in the software engineering insights or software engineering intelligence space. Every one of these tools helps visualize DORA metrics. They help collect them from systems.

My plea to you is: please don't go build a bespoke system as your first step. I have unfortunately seen an organization complete a DORA survey, come to some conclusions about what they want to change. And the executives say, "Yeah, but that was just a survey. That's just like your opinion, man. I want the data. And we have all the data — so this team, go build us a dashboard." That team did. They went off and built a dashboard, and two years later they came back and said, "Hey executives, we've got this dashboard, and it's showing us that we are exactly as we were two years ago when we talked about it." What they did over those two years was not change anything. They made no improvements over those two years. But man, at the end of those two years, they had these beautiful dashboards — that I know, because I saw the statistics, no one ever looked at.

So if you want to capture DORA metrics, please start with your mouth and a conversation, or the Quick Check, or any one of these tools — and then decide how you want to change it for your own organization.

Culture, constraints, and continuous improvement

We've also learned over the years that software delivery matters. Software delivery performance drives organizational performance. It helps our people have better well-being, less burnout, more job satisfaction, more productivity, et cetera.

Culture is the cornerstone of everything. It always starts with the people. And that's what really drives our organizations. The good thing about culture is, it's not static. It's not set in stone. It's always changing. And the bad thing about culture is, it's not static. It's not set in stone. It's always changing. You have to be very careful. It is easy to set an incentive or a goal incorrectly and destroy your culture. So we have to think very carefully and test and iterate and learn as we create this culture.

Of course it wouldn't be ETLS — well, today it actually would be, because it's the first time, but it wouldn't be DOES if we didn't talk about Westrum somewhere. I'm sure you're all familiar with Westrum's topology of organizational culture. If you're not — you see yourself and your team on this chart right now. I know that, because that's everyone's lived experience. We want to move to that generative side, the performance-oriented side of the equation. Obviously psychological safety matters. We've used Westrum many times within DORA to do this research. We lean on research of others, like Project Aristotle from Google, who found that psychological safety was the number-one contributor to a good team.

The way that you get better is by alleviating constraints. That's how you get better. You don't get better at change lead time by waking up every morning and saying, "I'm going to make my change lead time faster," or "I'm going to make my deployment frequency better by bashing that button more frequently." You get better because you find out what your constraint is, what's holding you back, and you make that better. You make an investment there, and you use those four keys as your guardrails. Everything that we're doing is an experiment. We are just trying things out and we need a way, in an experiment, to know — did this experiment work or did it fail? The truth is, that's a successful experiment regardless of what the answer is. So use those four key measures to help you understand: is this change helping my software delivery performance?

As you improve, you have to improve iteratively. Try the improvement kata. Continuous improvement — do the thing. There are benefits beyond software delivery, specifically the well-being of your people. But I think I've hit that pretty hard.

Looking forward — the next decade with DORA

Alright, in the two minutes I have left, I want to break out my AI-generated crystal ball. I don't have an actual crystal ball, but I do have some thoughts. This talk was called "A Decade with DORA." We started in 2009. I want to think about what does the next decade together with DORA look like?

First and foremost, I want to thank you for contributing to, participating with, and listening to DORA. DORA the program, the project, has been along this journey with you, and we continue to learn from you every day. I think we want to continue to do that into the future.

As I look at what's happening right now in the AI space, I see echoes of things that we saw in the past. A great example: one of the things we've investigated over time is how do teams use the cloud? And it wasn't long before we saw that it wasn't the fact that you were using the cloud that mattered, but how you use the cloud. And this is going to be the same with AI — it's not the fact that you're using AI, but how are you using AI?

Frankly, there's a lot of focus right now on generating code with AI. And that's great, and I use it every day, and I love it, and I feel more engaged with the code that I'm doing. I feel like I can do more things. But if I want to improve the constraint of my system — how many of you feel like the constraint in your system is generating code? That's not the constraint. That doesn't mean we shouldn't use AI there, but I think we have to look beyond the generation of code. I want to use AI to help me get faster feedback, to help my developers stay in the flow, to help me ship software better, to create a climate for learning. I think AI has an enormous potential for all of these things, but we have to think about how we're doing it.

And every day I get asked, "How do we measure the improvement? How do I measure my return on investment in AI?" And unfortunately, most people that ask have no baseline measures that they're going to compare against. So we have to have something that we're measuring from, decide how we're going to experiment, and then use those measures to validate or invalidate our experiment.

I have two seconds left. Here are the questions that I have, and I'd love to talk to you about them over the next two days: What's the impact of AI? Which DORA insights are going to help us collectively most over the next community? How do we scale these lessons that we're learning? And how do we prioritize improvement over measurement? We need them both, but it's the improvement work that's more important than the measurement work.

That's my time. My name is Nathen Harvey. I'll be around and I will answer any questions you have. Thank you.