Host Intro (Gene Kim)

Anyone who has been involved in the DevOps community in the last 10 years will recognize Nathen Harvey's name, as his contributions were so visible, especially in the Velocity community, partially because that was his remit as the vice president of community development at Chef.

In more recent years, he's been leading the DORA team at Google Cloud, which has become the new home of the State of DevOps Report, which Google Cloud acquired in 2018.

Many of you have heard that working on the State of DevOps research from 2013 to 2018 with Nicole Forsgren and Jez Humble was one of the things I am most professionally proud of, which became the basis of the Accelerate book. I've so much appreciated what Nathen and team have done validating previous findings and researching new frontiers. He's currently the product manager at Google Cloud, and he's presented many of the DORA findings in years past, like last year.

They did a landmark research report measuring the effects of AI on software delivery performance. They found something that was startling and unexpected, suggesting that the more AI you use, the worse stability and throughput got, which is terrible. But this didn't quite sit well with me. Yes, Steve Yegge and I have certainly observed that AI can lead to horrendous accidents, but there's got to be a way to show that AI can actually help us achieve all the goals that we care about.

I appreciate Nathen allowing me to be involved in this year's study, allowing me to invite people like Dr. Matt Beane, Daniel Rock, Dr. Eirini Kalliamvakou at GitHub, and others, and Steve Yegge, to participate in this year's study with the goal of seeing if we could resolve the DORA anomaly. Here to share the 2025 report is Nathen Harvey.

Nathen Harvey

Thanks, Gene. Good morning, everyone. How's it going? Awesome. I'm super stoked to be here and share some of our results. I want to talk about the State of AI-assisted Software Development.

This is the moment that you've been waiting for. Take your phones out and hold them up. I don't care what you do while you're holding them up, just hold them up. It helps me feel very comfortable up here, like you're paying attention to what I have to say. Maybe you're going to scan some of these QR codes. That's awesome.

This report just launched yesterday, and you might notice a change in the DORA report. It is now called the State of AI-assisted Software Development. We've moved past that other word that used to be associated with this conference as well. Interesting.

This year we had an amazing research team. So amazing, I can put some of them on the slide, but not all of them. Here are some of the researchers that we had: Dr. Derek DeBell, Dr. Kevin Storer, and myself are all Google employees. But we extended our reach out to a number of different partners, some of whom are here with us today and will join me on the stage in a bit.

We launched yesterday and we were covered on CNN. Nice. Thank you. That was a pause for applause. Thank you for picking up the cue. That's pretty awesome, that we were on CNN. We were also on a bunch of other things, including Business Insider. The thing I love most about Business Insider is they said it was a four-minute read for this article. To be very clear, it's a 140-page report. Good luck getting through it in four minutes. I'm sorry about that. I'm not sorry at all.

Even this morning, we heard from folks that were actually featured in the report. Exabeam, who was up here earlier, is featured in the report. We have a brief case study on them. It is brief. You can read that in less than four minutes.

Let's dig into some of the findings of this year's report. What did we understand? First and foremost, AI's primary role in software development is that of an amplifier. Whatever you're doing today, however your organization is working today, you should expect that AI is going to amplify that. That means if you're doing really well, AI's going to amplify that. If you have a lot of opportunity to improve, AI is going to point out some of those opportunities for you.

AI adoption this year is off the charts. Next year, we won't ask this question. I've been saying that it's basically like if we asked the 5,000 survey respondents that we got, "Hey, do you use a computer at work?" Ten percent of them lie to us, it looks like. So everyone is using AI.

But it's really interesting: of all these people using AI, using AI is one thing, but how about, do you rely on AI? Ninety-five percent of those people are relying on AI for a task that they regularly perform in their work. Adoption is high. Reliance is high.

Reliance is so high, in fact, we asked this other question: think back to a recent workday. About how long did you spend interacting with AI? A couple of things to point out on this slide. First, the people all the way over there on the right-hand side, 12 hours in their recent workday interacting with AI, that's Steve Yegge right there. He answered the survey. We know that it's true.

The other thing that's really interesting is the median time is two hours. Two hours in a working day. If you look at other research outside of ours, how much time does an average software engineer spend actually writing code? A lot of research says that's about 30%. Well, two hours is almost 30% of a workday. So if you're a software engineer and you're writing code, there's a high probability that you're doing it with some AI assistance.

In a similar finding to what we saw last year, though, trust is not a given. Actually, I think this is super healthy. You should not trust the output of AI 100%, nor should you trust it 0%. What we're seeing again this year is that about 30% of people have a little or no trust in the output of AI. This is down from about 40% last year, but we are seeing huge reports of productivity gains individually. I feel much more productive, and yet I don't trust this tool. I think this is a healthy trust, and it's a sign that we need to validate and change how we're working with AI.

Of course, with DORA, we want to know more than just statistics. We want to understand: what's the downstream impact of AI? I think you saw a version of this slide in Daniel Rock's talk just a minute ago. It talks about, as you increase AI, how is that impacting outcomes that you care about?

Really quickly, I'll take you through this slide. Individual effectiveness is improving. That's a good thing. Software delivery instability is also going up. That's not necessarily a good thing. Organizational performance value, time spent doing valuable work, code quality, product performance, software delivery throughput, and team performance are all improving. Burnout and friction are staying about the same, which doesn't sound great. But honestly, if you are experiencing burnout or friction in your organization, I can't think of a tool that has ever been invented that you just throw at that problem and friction and burnout disappear. So we shouldn't expect that with AI.

There is an optimistic way to look at this. As you use more AI, you are also not increasing friction or burnout for the people in your organization. That's a good thing.

As we compare what we see this year with what we saw last year, we see some things that are consistently beneficial. They were good last year, they stay good this year: individual effectiveness, code quality, team performance, organizational performance. Some things we call stubborn effects. They were like this last year, and they're stubborn people things. They're still like that this year: no relationship with friction, with burnout, and that increase in software delivery instability.

But we see some changes for the better. The time that people reported spending doing valuable work has gone from negative to positive. Software delivery throughput has gone from negative to positive. Product performance has gone from neutral — as we use more AI, our product performance is about the same — to now positive. That's really cool.

Another thing that we did this year is, like I said, we see that AI is reflecting and amplifying those organizational capabilities. If you've read the DORA report before, you've seen us talk about different performance characteristics of teams. We usually use software delivery performance as the way to look at how well a team is doing. Maybe you've heard labels like low, medium, high, and elite performers.

We've always felt that that is not enough. Not a wide enough aperture, not enough understanding of how a team is working just to look at software delivery performance. This year we looked across a number of different aspects of a team: team performance, product performance, software delivery performance, et cetera, all of these that are on the screen.

We did a cluster analysis for our dataset, for all of our respondents in our dataset, to understand what the different teams actually look like. This gave us seven different archetypes, or seven different team profiles. We gave them names. You may or may not identify with some of these names. That's okay. We called them things like foundational challenges or legacy bottleneck, all the way through to harmonious high achievers.

What do these different teams look like? Now's the other time to get out your phone. Look at what these teams look like. Isn't that amazing? You all understand this? I don't have to tell you anything about it, right? No. We dig into more details in the report, but let's just take a look at two of them really quickly.

First, the legacy bottleneck team. Unfortunately, some of you might identify with this. These teams are in a constant state of reaction. Unstable systems, pressure, manual processes, poor documentation, unclear priorities, all of that creates a drag. AI is not going to magically fix that. It is going to amplify that reality and show you where those bottlenecks are.

Contrast that with harmonious high achievers. These teams have strong capabilities already. They have strong software delivery performance, strong team performance, strong product performance, and good learning practices. When AI comes into that environment, it amplifies the good things that are already there.

The point is not that one archetype is good and one is bad as a label. The point is that these profiles help teams understand their current system and pick the next experiment. You can use the capabilities model as a set of hypotheses for what to improve next.

One thing I want to show really quickly is the capabilities model. This is how you take the findings from the research and put them into practice with your team. For me, that's the most important part of this research: taking the findings from our research, using them as hypotheses for the next experiment that you're going to run, and putting this research into practice.

Now, to continue this discussion, I'd like to invite some of the researchers that are here with us today. Gene and Matt and Daniel, please come back out onto the stage to join me. Let's hear it for all of our researchers across the report. With that, I'm also going to ask Gene to take us through a couple of findings, and then we'll have a quick discussion.

Panel Discussion

01Gene Kim

Absolutely. Thank you, Nathen. I'll share with you maybe some of my hopes and aspirations, some of the hypotheses I had hoped to prove, and show what hit and what didn't.

One thing that I really wanted to understand is: to what extent, how important is it to have great modular architectures where you have independence of action? And how important is it to have great technical practices? We heard throughout the last two days that you need great practices.

There's something that I love to call the Nyquist stability criterion. It basically says, from control theory, whenever the controller must operate faster than what it is controlling. What worked at five miles an hour probably won't work at 50 miles an hour, as we increase code generation so much quickly. We were looking for markers of that.

One of them: we did find some hits. We know that version control commits happen more frequently. Testing feedback, I'm not sure if we really found, but we actually did find that it's important to decompose and keep tasks small. We were trying to figure out, do you want big commits or small commits? Not quite sure. But I think there's something there.

There was another one that I'm really excited about. You heard Daniel mention this: the notion that how much you trust is dependent on how much you use it. One strange definition of trust I love is very mechanistic, but I think it's useful: to what degree can I predict how another party will act and react? The more I trust the other party, I can make bigger requests. I need less communication. I need less feedback.

The MIT beer game is a great example. No matter who you put in there, they end up distrusting the other people, and they're convinced that they don't even understand the rules. That structure dictates low trust.

One of the hypotheses was the notion of fingerspitzengefühl. This came from Ed Gaiz at GitHub: the notion of how long have you been using it? How many hours or thousands of hours have you used AI? You need to get close to 10,000. We found this very interesting statistic that said, on the x-axis, how many months have you been using AI, and on the y-axis, how much do you trust AI? There's this line that explains the more you use it, the more you trust it. I think that's very interesting. It says that this is not something that you get right the first time. Instead, it is actually something you have to have deliberate practice around. So much more interesting research ahead.

02Nathen Harvey

That's great. Let's just have a little conversation. If you guys want to come over onto the stage proper.

I'd like to start with a question, Gene. We talked about version control, and we found strong version control practices really drive AI adoption. I know that over the past few years you've been vibe coding your way around everything, and you and I have had a lot of conversations around version control. How have your version control practices actually changed?

03Gene Kim

I have found that you have to check in so much more frequently. Often it is your only safety net, because AI will change your code, wreck your code, delete your code, and without version control, you're going to have no safety rails. That actually did correlate with performance.

04Nathen Harvey

A very big safety net that you want to have. Make sure you're using proper version control. Just as a reminder, if you're still using the .bak version control system, you're not doing version control. If you're not sure what that version control system is, come find me at a break.

Next up, for both Daniel and Matt. We talked about this mirror effect of AI amplifying what is happening in your organization. Maybe start with you, Daniel. What's your reflection on that from an economist point of view?

05Dr. Daniel Rock

There's that old line, you ship your org chart. Now we're doing that on steroids with AI, and maybe AI will be part of the org chart in the future. I think there's this idea that you really do have to be nimble in reconfiguring the work that you do. There are a lot of organizations that are fairly static in what they're doing right now. They're having a tough time with AI. But if you can be nimble in code and nimble in your organizational practices, then you get the most out of these tools.

06Dr. Matt Beane

That is a consistent and strong theme in the interview data from this qualitative study I mentioned before. There's an additional organizational factor here, which has to do with the lack of adherence to titles, functions, work processes, ways of thinking and talking, such that we end up spilling across typical divides to produce new stuff at new levels of quality.

Sometimes there are names in English for those new titles, and marketing folks are great at inventing them. I'm sure it'll all look perfect and clear in retrospect. But right now, on the ground, there are at least pockets in every single organization where folks are bending and breaking rules and expectations to invent a new way of doing stuff that nobody quite has a name for yet. You can count on it. You can find it.

07Nathen Harvey

That's super cool. Matt wrote a section in the report about the way that teams are thinking about, or organizations are thinking about, improving skills and how that's so important. Do you want to say a few words about that?

08Dr. Matt Beane

The short story there is, my talk was probably a bit doom and gloom. Experts can operate on their own, and juniors are sort of left out to dry, which I think is a very serious problem.

At the same time, it's also very obvious: there's a whole chapter in my book on what I call inverted apprenticeship, where senior folks learn about new tech from junior folks, and they get a good deal in the bargain. As a junior person, I get more exposure to the problem space than I would otherwise. Those are hard to execute well according to the research I've done, but you can count on those kinds of things happening willy-nilly. It's about naming it, quantifying it, and operationalizing the stuff that's actually working on that frontier.

I think the game in terms of what skill is and how it gets built is getting changed, and there's productive deviance in there that we just need to learn from.

09Nathen Harvey

Awesome. A final question, and this one will be for all three of you. When we look at the report and some of our findings here, one of the pieces that we encourage organizations to have is to really take a systemic view of their organization and their processes and so forth. But taking a big-picture view can be hard if that's not what you're doing today.

So maybe a word from each of you: first steps for how do I start systems thinking about my organization and how that relates to adoption and impact from AI? Who wants to go first? And we have to end on time.

10Dr. Matt Beane

Okay, fine. I'll say it very quickly. What we're doing at SkillBench is one attempt at this problem, which is we're going to decompose all your work down into these directed acyclic graphs that Daniel talked about, down to leaf nodes that don't normally have names in English: tiny packets of work. Then building back up from there is a way of thinking about what is the new meaningful way to put humans in a place where they're most valuable and getting the best development, and automating away the stuff that no one could stand before.

11Dr. Daniel Rock

I'll add to that. We do something similar at Workhelix, where you break down the activities that are happening inside of the organization and try to figure out where the hot pockets of good potential would be. That's a great way to align senior executives all the way down to the folks doing the work.

But then you actually have to run those experiments and see what works well and what doesn't. In the beginning, the AI governance can either enable you to run really quickly or it can keep you from ever doing something. So definitely partner with the people in charge of that.

12Nathen Harvey

Legal at T zero? Yes, legal at T zero. All right, Gene, 30 seconds. Go.

13Gene Kim

No, 10 seconds. They describe a task tree. The LLMs are going to get better and be able to do things higher and higher in the task tree. The more you are better at AI, that it earns your trust, the more you're able to delegate more to the AI. I think that's a great description, Nathen.

14Nathen Harvey

Thank you all so much for joining us. I look forward to hearing your feedback and how you're putting this research into practice. We'll be around the rest of the day to talk. Thank you.