Host Intro (Gene Kim)

All right, our next speaker is Tim Cochran. I met him at the Martin Fowler event that was in Utah a couple of months ago, a couple of weeks before the Ship Summit. And there were plenty of legends there. But for me, the person I remember most, and the person who had the most incredible things to say, was Tim Cochran.

So Tim Cochran spent 19 years at ThoughtWorks, and three years ago he went to Amazon to be a part of their Amazon Software Builder Experience. And this is a breathtaking program. The goal was to elevate AI adoption and engineering effectiveness across Amazon's 100,000 developers. And he said in an off-the-cuff way, of a longitudinal study that he was able to do, there were only four things that mattered for success. And when he said them, my reaction was like, "That's what I thought." And so I was so excited to talk to him after the summit, and here to share what he learned — Tim Cochran.

Tim Cochran

So, a little bit about me to start with. I grew up at ThoughtWorks. ThoughtWorks is an agile consultancy. Done a lot of TDD and pair programming. Never done a code review. So I was there for 19 years, and that's where I became passionate about developer experience. You can't really have a bad developer experience when you're pair programming, otherwise you're just sitting there watching the build.

And then a couple of years ago, I joined Amazon Software Builder Experience. They own the core developer experience at Amazon. We don't think of ourselves as a platform team — we're more like a set of capabilities that the various business lines in Amazon build their own developer experience out of, obviously with AWS as well.

So I was initially working on AI adoption across the entirety of Amazon — that's 2024. And then piloting AI native workflows more last year, and that's with organizations that wanted to step in, that wanted to embrace AI. So it was about 15 or so teams. And I've actually recently left Amazon. I'm co-founding a startup to drive AI effectiveness. I'm happy to talk about that later.

---

So, beginning of last year, we had this word — AI native. It's a buzzword that leadership were excited about. Nobody really knew what it meant. Do we know what it means now? I don't know.

But we had this phrase: okay, we're going to reinvent product development. Amazon's always about reinventing, so let's reinvent the product development workflow with AI at the center. Great. Still a bit buzzwordy.

In practice, what actually happened? So in practice, the AI agents are doing the heavy lifting of all the low-value coding, maintenance, and operational tasks while the developer directs and supervises. There's a lot there. The most important thing is the words maintenance and operational tasks. We're not really concerned about code generation — that's easy. And then also that the developer is directing and supervising. They're not writing any code themselves. That was the important part. So we're fully agentic.

But it didn't quite do it for me. There's also this other piece, which is agents help you create. So they help you ideate, design, and build. And it's also important that that isn't just the developer — that's the entire team, and we can think about everybody as a technologist.

---

Okay, so I'm going to go through the four lessons that Gene mentioned that I said in about 10 seconds at the conference.

01Lesson 1: Empowering Teams

Where Amazon Software Builder Experience sits, it was a little tricky because we're trying to help 300 different business lines, and really what we're actually doing is enabling the enablers. And so how do we do that?

So when we first did some research, we talked to a lot of developers, and we found that people weren't using AI. There was no good reason why they weren't using it. Sometimes it was a misunderstanding or a misgiving — a misunderstanding about cost, about security, about metrics, about how their role was going to change, about productivity. Or they didn't think they had permission.

The early adopters didn't care if they had permission or not, but some of the team — Amazon, everybody's busy. And what we realized is maybe the org had made some general statements about AI, but the managers hadn't really empowered the team.

So we trained most managers in Amazon — it was a lot — about some of these things. Some of it's the art of the possible, sort of teaching them what it can do for coding, but other aspects too. I don't know if you have this at the moment: suddenly, everybody cares about developer productivity metrics who didn't care about it before. The number of people that I sent the SPACE paper to, or told them to read the Accelerate book, so they'd stop asking me stupid questions.

And then I think what we saw is that the teams that were most productive were where they had an AI-first culture. So it isn't about individual developers. It's about the manager thinking about the outcomes of the team, the outcomes for the customer, and changing the process. In a retrospective or in a planning session, you really want to think about AI first — so it's creating that culture.

And then expectation of impact. Whether it was stated or not, a lot of folks were feeling pressure — pressure that we have to use AI, we have to show productivity. So there was a lot of reinforcing that it'll take a couple of months before a developer is going to be fluent, and then beyond that, we're going to see that productivity gain.

So what we also did was create these building blocks. This is fairly basic: we should expose data and create integrations. But what we wanted to do is create something that was seamless, so that when a developer starts up their developer experience, they get access to everything they need in the core developer experience.

But we'll also go beyond that and actually really understand what the small micro jobs-to-be-done are. So you can take an example of a build log: you can expose the data for a build, you can expose the build log. But it's not very efficient — it's not very efficient for humans either. They have to read through this long build log and find out where it broke. So maybe you might optimize it, maybe you create a summary, just show the error message. But then you can take it a step further. Maybe now with AI, I know what all the common build failures are. Maybe I can suggest a solution. And then you can take it a step further — maybe you can apply the solution with one click. So it's about actually looking at the outcomes of what developers are trying to achieve and optimizing through that. The MCP tooling, the skills we created, the templates — were all about those kinds of things.

02Lesson 2: Continual Practice

Okay, so it's a massive paradigm shift. It takes a lot of time and practice for developers to master AI skills. A lot of teams do demos and workshops — those are all great. But working with AI is really messy. You fail, you try again, you improve your prompting as you go along.

And what we actually wanted to do was give developers exposure to best practices in the context of their team. So we did a lot of pair programming and mob programming. If you're familiar with mob programming, it's where a team works through a solution together. We had some mob programming sessions, and I think one lasted about 30 seconds before somebody went, "Wait, what the hell did you just do? That's amazing." So I think when you write documentation or training, you forget to share the actually useful parts, which is often: I had this problem, how did I overcome it?

And then, like a lot of people, we've been leveraging early adopter champions. The one learning there is: just because someone puts their hand up and says they want to be a champion doesn't necessarily mean they're very good at training, doesn't necessarily mean they actually want to train. So we did a fair amount of training the champions on how to share. Often they just want to do cool stuff with AI.

And then — this is fairly well-known, but I want to plus-100 this — understanding the context is how you get the developers to be more productive and to create a better outcome. Particularly the feedback piece got a little bit lost. Often developers will start with unit testing or writing code, but they might not think about joining up all the pieces in their developer experience.

And so that led to this anti-pattern: if you don't have a harness, then the developer is hand-feeding the agent, smoothing the way. They're creating these small little morsels for the agent to feed itself — and for the agent to be able to cook and shop and farm, to keep going with that analogy.

03Lesson 3: Coding Isn't the Only Bottleneck

So the third lesson is that coding isn't the only bottleneck. When we took these teams, we had the task to make them productive, but especially at Amazon, the developers were wearing many, many hats.

So we wanted to make sure the teams knew how work gets done and what their friction was. So we created this AI litmus check, which is: we take something out of the backlog for the team — either something small and deterministic — and see what it actually takes to get that through to pre-production, and how many developer interventions had to happen. The keyword here is intervention. The team should now feel like if a developer has to do something, it's not a good thing — it's an intervention.

And so this is like a modified value stream map, and it would help teams understand where their friction is. Quite often, developers form habits around friction, and they don't realize that what they're actually experiencing is a bottleneck. It's something they've lived with for many years, and they consider it just how work gets done. So running a qualitative study like this litmus check allows the team to realize what friction is and how it can be improved.

And the other piece here is not just the friction — it's also agentic friction. So when an agent has to use a lot of tokens or makes a lot of calls to the LLM, that's a signal that there's some optimization we can do there. So at the end of this, you create a modified value stream map. The phrase production ready is also key here, because I think a lot of metrics measure when a PR is created. For a lot of environments, creating a PR or having the code reviewed does not mean that it is production ready. And so doing this sort of litmus check helps the team discover things.

And so from that litmus check, we can also extend it: we can debug a problem, do triaging. A really interesting study you can do is take a code base that nobody's ever worked on — what actions does it take for the team, for the agent to become productive, for it to know about all the tooling? So these are just some studies that you can do that are useful.

And so through these value stream exercises, and through conversations and research and hackathons and things like that, across Amazon the developers started creating little tools and little fixes using those kinds of building blocks that we talked about. And those organic things — maybe they were created for their team — eventually ended up becoming platform capabilities. And a lot of it isn't necessarily coding. At Amazon, we write a lot of documents. So being able to create a document, being able to pull the context for it, to be able to work with an agent to create it in the right format — to create a PR FAQ in the right format — that's a really useful kind of innovation for a developer.

04Lesson 4: AI Amplifies Your Existing Bottlenecks

Lesson four: AI amplifies your existing bottlenecks. You should apply high-throughput engineering best practices.

When I look at how a team has previously worked, often developers in the team know about the friction they have, but because proportionately or relatively it isn't a big deal — if a feature took three or five days, does it matter that the code review takes one day? It's probably not that great, but it doesn't really matter. One day compared to five days to write the code, maybe that doesn't matter so much. So perhaps that friction is being deprioritized.

But now maybe that same change takes half a day, or maybe it takes an hour. So now that friction — that process that we previously deprioritized — is now a much bigger problem than it was. And so now we have to tackle those things.

So a lot of the teams that were most productive tackled these processes. And it isn't just with AI — developers want to apply a tool to solve everything, of course. But the most productive teams that had the highly effective managers actually reinvented their processes. They accepted, "Hey, our code review process is broken. We wait too long. We have too many conversations." And they actually changed those processes. And of course, applied AI as well.

---

All right, so here's where we go back in time. We had a lot of presentations from teams in Amazon that were really doing great stuff — really had a high throughput. When you watch that presentation, especially some of the principal engineers, they almost roll their eyes and say, "All they're talking about is things that we've been doing for the last 20 years."

So the other recommendation is: apply those high-throughput practices that we've been trying to do for the last 20 years. Continuous deployment — if you're not doing that, obviously, you shouldn't really be touching AI if you're not doing continuous deployment, to be honest with you.

And then small atomic changes — a lot of teams adopted a practice of one prompt per task. One prompt per task per change. And that helps with code review. It helps with the context. It's like the "if it hurts, do it often" thing.

Modularized code bases — who would have thought it? If suddenly you have a lot of agents working on your code base, having a clean code base reduces the amount of merges and the amount of conflicts. Suddenly mono repo tools became much more popular at Amazon — we started seeing that.

A lot of teams were starting to check in all their documentation into the code base so that the agent can discover it. And then just making sure that agents can solve their own problems, kind of what we talked about earlier but maybe at a larger scale.

But the most important one was this: as we create more changes, one of the problems we have is we need to make sure that those changes are actually correct. On a lot of teams, maybe you have a lot of unit tests, maybe you have a lot of mock-based unit tests, maybe you have an end-to-end test. But how the team actually knows whether a change works or not is still done by the human — they're doing exploratory testing, they're deploying it, they're trying it out. A lot of teams work like this and they haven't really fully automated it.

So the teams that were most productive were the ones that actually managed to fix this. An agent, when it makes a change, has a production-like environment. It can run a series of component tests, a series of contract tests, and it can gain confidence that that change is actually going to work. And this is key, because if we don't do that, then what's going to happen is we're just going to get tons of rollbacks later on.

I wrote an article about feedback loops probably five years ago, and I keep looking at it and going, "Is it out of date?" And I feel like it's actually more important than it ever was. This notion of giving better feedback and shifting left — for this conference, I Googled when did "shift left" actually come out, and it was 2001. So maybe agents will finally force us to shift left and to fix that problem, so that we get high confidence before merging.

---

That's the end of my presentation. So the four lessons:

1. Empowering teams through trusted leaders, building blocks, and community. 2. Continual practice — it takes time to master those skills, and learning needs to happen in the context of the team. 3. Understand your value stream map. Apply AI to that friction. Use some of those qualitative studies to make sure that you don't have learned helplessness problems. 4. Apply those good engineering practices that allow high throughput.

Great. Thank you.