Host Intro (Gene Kim)

This morning I described Brian Eno's concept of scenius, and how one of the characteristics of a productive and great scenius is a rapid dissemination of tools and techniques. And so, without a doubt, I think one of the tools and techniques that have been spread most rapidly throughout this community are the DORA metrics: deployment frequency, lead time for changes, change failure rate, and mean time to recover.

So these metrics, of course, come from the State of DevOps research that I had the privilege collaborating with Dr. Nicole Forsgren, who was the lead researcher for five years. We worked with Jez Humble, and again, without a doubt, this is the professional achievement that I'm most proud of.

She's the lead author of the Accelerate book, which I routinely cite. She's a co-author on the second edition of The DevOps Handbook. Dr. Forsgren was the CEO of DORA for six years, which was sold to Google Cloud in 2018. She was later the VP of Research for GitHub, and she's now a partner at Microsoft Research, where she continues her fantastic work, which includes the SPACE developer productivity framework and so much more.

And so here's a fun fact: that SPACE paper is now one of the most cited articles in ACM's history. So I'm so delighted that Nicole is back here at DevOps Enterprise sharing what she's been working on lately.

Here's Nicole.

Nicole Forsgren

Hi, everyone. Thank you so much for being here today.

Today I'll be talking a little bit about DevEx and developer experience. And I'm not sure if anyone has heard, but apparently DevEx is a little bit of hotness.

So for those of us who are kind of in the DevEx space, DevEx is short for developer experience, and it's kind of had this resurgence. So it's kind of some hotness. Is anyone in DevEx, doing some DevEx? Like three of us? Cool.

For those of us not in DevEx, I don't know if anyone's heard of AI or LLMs. I don't know if you've heard. That's kind of the hotness.

So here's the dirt: it can be the hotness, but you can't actually ship your generative-AI-enabled programs unless you can actually ship software. So we come back to where DevEx is the hotness.

Welcome to my talk.

So today we'll be chatting about DevEx, what it is and why we should care. I've got a little bit of research to drop, some brand-new stuff, and then I'll be talking about another reason we should care about DevEx: because you can actually use this as a lens to shift culture.

Because I don't know if anyone else has ever hit this, but sometimes it can be a little tough to shift culture, to do this whole transformation or get everyone aligned with change. And if not, then please come see me and probably a bunch of us. Let's create a new birds-of-a-feather session.

And then I'll wrap up chatting about DevEx and AI, and how we can really enable a bunch of teams and some of the promises that are promising new tech that's existing in this space.

So starting off with a little bit of 101, really: what is DevEx?

Developer experience, at its core, really is kind of the satisfaction with tools and processes and what it takes to deliver code, and what it takes to really even do our work as developers and technologists.

What it is not is tooling. This is not just about tooling. Sounds a lot like DevOps, right?

But what's a little bit different here is that we're really centering our users, right? Why should we even care? It's taking this really people- and customer-centered approach. And when we talk about DevEx, it's that developers are our customers.

So we can get some really valuable insights to the work that we're doing here and why it allows us to improve our systems in a way that can highlight the bottlenecks, it can highlight the blockers, it can really surface the things that allow us to improve and accelerate and amplify the work that needs to be done.

And many times this work is speed, sure, but it can also be reliability. It can be security. It can be compliance. And the thing that I love about this is that it's security and compliance in ways that are user-friendly.

Because then when we mean user-friendly, it's baked into the systems. So it's fast. So it's easy, maybe. So it's invisible, so that everyone really wants their systems and their software to be safe and secure and compliant instead of, "Oh, well, we'll just make them do it."

I don't know if anyone's ever heard that before and how successful that is. It's not, right?

And so when we do this, what I like to call the right way, we find ways to make it delightful to do the right thing.

Now, there are lots of ways to think about developer experience, but here is one way. And so when I talk about this, here are a few dimensions to think about contributing to and having a good developer experience.

One is flow state. We've all probably hit this in our work. So this isn't unique to developer experience, but it's when we have this work that we do where we can hit deep work, right? Where we can have this incredible ability to be fully immersed in our work, right? It's joyful, even when it's incredibly challenging.

Another one is having feedback loops that are both fast, or the right fast, right? And they're high quality. I don't want fast feedback that's wrong.

And then the last piece here is cognitive load. Now, this is how much mental processing is required for the task. We generally want this to be low, but I will mention that occasionally we want to introduce just a little bit of friction to help people think through things. But again, in general, we want to have low cognitive load.

Now, we used these dimensions to frame this latest research that I'll be talking about today.

Now I will mention this data is hot off the presses, so I just have a teaser. The fall paper will be dropping end of the year, but I did want to share some of what we were doing.

Now, first question is: why more research?

If people ask me this question, I will usually just stare at you blankly and say, "Because I love research." But it's worth asking. So I think there are a few things to think about here, because I will say it's a fair question.

First of all, I sometimes joke here that it's nice, right? Good vibes only. I love bringing this up in terms of DevOps, DevEx, because there is this misunderstanding that developer experience is just about making people happy.

That's not true. Now, is it worthwhile? Absolutely. I love that Chuck just talked about engagement and how absolutely worthwhile engagement is, but there's also more to it than that.

What else is there? There are impacts. And what do we mean when we talk about impacts, and how do we get these impacts, and for whom do these impacts fall?

And so these were some of the grounding questions, motivations to doing some of this research, right? And by the way, the impacts are for a variety of stakeholders. We're talking about developers. We're talking about teams. And yes, we're talking about the organization.

And I wanted to do some of this work because so often people are like, "Oh, well..." and I, by the way, preface: I hate this answer. But when they're like, "Oh, well, it just makes my devs happy."

First of all, you should care deeply about making your devs happy and making their work easier to do. And there's so much more here, right?

So when the question is, "But why should I invest?" now we have some really good and really compelling answers. And these answers fall into actionable insights that anyone can do, so that anyone can kind of deploy some of this research or quick surveys where you are.

At the end of the year, we will be sharing all of the survey questions as well.

So here's what we found really quickly. What is it that you actually get? What are these impacts?

For developers, we find that it boosts creativity, productivity, and the opportunity to learn at work. I don't know about you, but that sounds pretty dope.

For teams, we find better code quality and less technical debt.

Now, for an organization, we see that it drives a culture for innovation, it improves retention, and it improves their goals and their profit. I don't know about you, but I'm in.

Now, how do we get there? Again, we followed this three-dimension framework.

So for flow state, how did we define flow state? By the way, this is a quick infographic. I have all of it at the end. You can take a quick picture. For flow state, this was defined as fewer interruptions, deeper work, and engaging tasks.

Now, for deep work, folks were 50% more productive versus those without dedicated time. This is huge. This echoes some of our previous work that we found in Good Day Project.

For those that had more engaging work versus those that had kind of boring work, we see a 30% boost to productivity.

When we look at cognitive load, how did we define cognitive load? Intuitive processes, understandable code, and easier deployment.

Here we see a 40% increase in productivity for those with great code understandability. When we look at intuitive process, we see increased innovation to the tune of 50%.

Now finally, when we come to feedback loops, fast feedback loops are fast responses for developer questions and code reviews.

Here we see fast code review turnaround times see a 20% increase in innovation. And fast responses to developer questions see a 50% decrease in tech debt.

I think this is really interesting and really, really promising.

Quick, take a picture. Here's your infographic. I also have a Bitly link at the end of the talk.

Now, we will have a full paper coming out at the end of the year, but I at least wanted to share some of our early findings because I think it's really exciting, not just for what it means for our developers and our teams and our organizations, but what it means for the software and for the experiences that we want to put out.

So listen, if anyone's only on this generative AI hype train, here's how to get that hype out the door, okay?

Now, when we want to change the culture, the thing I also love about developer experience is that we can use a DevEx lens to help align people to this as well.

And I'm really excited, incredibly fortunate, that I've been able to work on some of this on a cross-Microsoft effort over the past year.

And this quote from Brian Chesky really sums it up well: "For one definition of culture is that it's simply a shared way of doing something with passion."

Now folks who know me will not be surprised that I am now going to start talking about metrics.

Metrics can be incredibly powerful because they can be a communication tool. Data gives us wonderful opportunities to clarify and define what it is that we're talking about, regardless of the organization that we're in. And by org, I kind of mean different business units, right? Regardless of the technology that we work on, it helps align and clarify.

It also creates shared language among teams. Sometimes we find that we can be talking about something totally different and use the same word, or talking about something that's exactly the same and use different words. This gives us an opportunity to align.

The other thing I love about this is it gives us an opportunity to move from intuition and just shooting from the hip to use data-informed insights.

I also love data-informed here, not data-driven, because we probably don't have something that just auto-executes when a certain number pops up. I want to surface insights and then let that guide decisions, because we still have experts where we work. But I want this to be informed.

So across Microsoft, we created something called Engineering Thrive. This is an actual example visual that we started sharing across the whole company.

This is a handful of data points that fall into broad categories of speed, ease, quality, and then Microsoft culture metrics that we use.

This has presented a wonderful opportunity to surface conversations, invite engineering leaders to get curious, think about what's happening across their organization, think about the trade-offs that are happening, think about the strategies they will take to improve.

Now, also look at what's happening here. It invites you to think about what's happening there. Now, yes, I have redacted a few specific labels, but speed, ease, and quality are durable.

Something I want you to notice as well: there is no one metric. There will never be just one metric. There is not just one metric. It is a suite of metrics across durable categories. It really helps you think about what trade-offs happen.

And when I say trade-offs, it helps engineering leaders put themselves in the seat of the developers that are there, right? Again, this is a very strong developer experience framework. What are the trade-offs that the developers in their organization are making between speed, ease, and quality, and culture? What does that look like? What do these data points represent?

By the way, some of these are objective metrics. They come from telemetry and instrumentation. Some of them are subjective metrics. They come from surveys and people. It's an explicit representation of this developer experience throughout the org.

Also, remember the visual that was there? They show these trade-offs and constraints.

We know that we work in a resource-constrained world. And by constrained, I mean the answer isn't, "Give me more headcount. Give me more..." I don't know about you. Does anyone else work in this environment? Do we just get to ask for the unlimited budget? I would love this world, right?

But when we think about making strategic investments, what will these investments be? Where is the most strategic place to invest first? What does this look like? Because the leaders know that this is the world in which we live.

Also, look and notice that there were categories: speed, ease, quality, and culture. These will remain durable.

This is also an explicit, and we've made this very explicit, acknowledgement that the specific data points within will likely iterate as we continue to improve in our data access, data fidelity, data quality, data instrumentation. But the categories of speed, ease, and quality will continue. These exist in many companies.

And then again, this creates a shared language and understanding for improvement, for change. There are no comparisons. The only comparison is against yourself, your own basement, your own baseline. Sorry.

Listen, if it's not great, embrace your own baseline.

And then what matters is improvement over time and what that graph looks like over time. So this spider chart, let's call it a spider chart, it's not quite, that is paired with raw values. And then one of those columns includes comparison to last period and last periods. So, like, you know, sparklines and stuff.

And so that's what really matters. And I love that that is the culture and the conversations that it brings.

So this is leveraging kind of this DevEx lens to embrace change, to motivate this understanding of how we improve.

Now, the other thing I love about this is it creates new opportunities to think about improving. Not just surfacing these strategic pieces of what, but then thinking about how, and thinking about how we can improve that, how faster and better, and allowing everyone to do that.

And so my team at MSR has been looking at how we could maybe use AI and large language models to allow everyone to do this.

So is any of this familiar? As we sort of talked about, efficient infrastructure is really important for any of us who want to be able to ship software.

Does anyone hit complexity that slows us down? No? Maybe? Okay, perfect.

Does anyone have situations where problem diagnosis is just really hard or really complex, and we need like 27 people in a room, and then we can't schedule it on the calendar, right?

Do we ever find that we have repeated solutions because two or three different teams didn't realize what we were all working on? The same problem? I'm seeing hands.

Does the complexity ever make it difficult to understand code and context? Oh, we all work together. We are all on the same team.

Okay, what if you had a dream team? What if you could all call on any one of us in this room, any one of us at this conference, just to come sit next to you for a minute or an hour to help figure out and solve this problem?

Oh, and by... so me, Courtney, Jim, Jason, Chuck. Would that be great?

Oh, and by the way, we happen to have already been working with you for a year, so we knew your code. I mean, sign me up. Are we on for that? Okay.

What if every single engineer and team in your company also had that dream team?

That was the question that my team had, because once we know what we need to solve, we still actually have to solve it, or figure out what our next steps are.

So we wanted to come up with our personal LLM-powered experts: consultant expert, guidance analysis, system analysis, tech lead. What if you could have a code-based tour? Stack trace insights? What if you need to figure out all of the things that are in your code that's not limited to a file, but it's the whole repo? Or a data scientist expert, software engineering expertise? I want Steve McGill to come help me all the time.

These are the types of questions you could ask.

For consulting: How can I improve onboarding? Who else has improved build times in similar contexts? What's happening in my systems that I should know about? What patterns are there?

For a tech lead: How does authorization work in this codebase? Walk me through the build process in this repo. I got a lot of build questions. Listen. Help me understand this stack trace.

For data scientists: Does distributed development affect code quality in my organization? How does build time affect developer satisfaction? Does branching strategy affect my PR time?

Do any of these questions come up for any of you? Do these sound familiar at all? I'm seeing some hands.

Super quick raise: we built some prototypes that actually help with this. And if you want to come find me later, I can walk you through more detailed prototypes of this.

But we have a consulting expert that will actually create an expert panel and come up with a ranked list of matches for you in a number of minutes versus weeks or months.

For the tech lead expert, you can ask it a question and it will select and summarize a mountain of context, and it will give you a summary as well as a diagram, and the right kind of diagram that links between the code.

And then finally, for the analysis tool, you can pick any paper that you want. It will surface the assumptions for you, it will create an analysis plan, and it will generate the code for you.

So what's next?

Let us know what problems do you have that LLM experts could maybe help. Co-innovate with us. We would love to create solutions with you. Watch for the latest research around end of the year, and let's co-create the future of developer experience.

Shout-out to everyone who was involved in all of this work, because it took a whole bunch of people to pull all of this off. So many, many thanks to all of them.

And thanks to everyone. Please feel free to reach out and access the research. Thanks, everyone.