Host Intro (Gene Kim)

Our next speaker is Dustin Warner. He's Director of Software Engineering at NRC Health. I met him after I was hanging out with his boss, Tim Ottensberg, who's here, and their CEO, Trent Green, who I've known for decades. And if you've ever gotten a patient satisfaction survey after being in the hospital, it probably came from them.

The purpose of the meeting was to see if we could quickly test one of Trent's hypotheses, which was: could we establish a link between patient satisfaction scores and the increased use of medical services? And so the next thing I know, I'm talking with Dustin. He's pulling data from all across the company, bringing in researchers, buying government data. He has Visual Studio Code up. He's generating massive pages of SQL queries, writing analysis code. And the whole time I'm thinking, this guy's like a second-line manager, and he's having so much fun coding. It was such an interesting thing to see.

And so since I've met him, he's told me about so many of his new management practices, which I've literally never seen the likes of before — such as showing up in the Slack channel where all their Vietnamese developers hang out, and him responding directly to them at midnight his time, in Vietnamese. So here is a person who really understands why we need to rewire our organizations and the forms it can take. Here is Dustin.

---

Dustin Warner

First, shout out to Gene. Every call has been amazing. This guy's like a shot of caffeine every time I talk to him, and super grateful — so thank you, Gene. This is fantastic.

I'm going to be a little different. I'm not going to tell you how to do things or all of this technology. I'm going to tell you what we did. And a motto for me has been production over prototypes. We didn't spend months or years building out something that never made it anywhere. We spent 90 days to get a core part of our platform into production.

So, who are we? We are NRC Health. We power human understanding. We are in the healthcare analytics and research space. The company's been around for 45 years. We've partnered with over 10,000 medical organizations. We focus in three areas.

Patient experience — the human understanding. Who are you in the hospital? What's going good? What's going bad? How can we improve? This is both for the hospitals themselves, but also for Medicare and Medicaid, the government. They want to know how are they treating patients. We also do market leadership — that's going to be, I want to expand, I want to go to a new location, get into a new market; we can give you data, strategy, and analytics around that. And consumer experience — we want to hear from all of you.

NRC has been around for a long time. It went from paper surveys — go into the office, make the little binder crimps, send those out to these hospitals — to being the first one to send out surveys via text message. So as you go to the hospital and you get that text message, it's one of three companies probably sending those to you, and NRC was the first one to actually get that up and running for the digital age.

So what am I? I am a Director of Engineering there. I lead about 30 engineers and four managers. My methodology is following California: reduce, reuse, and recycle. I don't want a large team. When I get a new project, new problem, I want to solve it with what I have and keep those teams small. And in the age of AI, two years ago that was crazy — "Oh, you're a manager, you don't want more people? That's like your badge, right?" Now it's, "Oh, you're a manager, you don't want more people — great. What are you doing?" It's like, "We're using AI."

01The Problem

So what was our problem? We had a platform in the middle of our company on fire. This was a third-party dependency that we did not own, we had no visibility to, we had no roadmap. We had no control of how fast they would get back to us or our customers. It supported three of our product lines and made up about 30% to 40% of our revenue. So this isn't something that is customer-facing, but it's backing up this entire product for three of these lines.

It was a goal we had. I came in two years ago, inherited this platform, was like, "We've got to get rid of this. What are we doing?" And it sat, and my hopes and dreams started to fade away.

And then last year — end of season two for me — the cast got rewritten. We got a new CEO, Trent, who Gene had mentioned. And Trent came in and started going product line by product line, meeting with business leaders, technology leaders, asking: what are our problems, what are our opportunities? So when he started coming through the product areas I support, it was amazing. We sat down, we talked about technology — kind of, oh, what is this doing, what technology are we using — for about four minutes, and then we spent an hour talking about where we want to go with it. What can we do? What are the opportunities? And it was great. And then the last five minutes it's like, "All right, so what do we need to do?" I was like, "Well, first we kind of need to own our platform. We have to get this in-house because right now it's been around for 18 years and is falling apart at the seams." So it was, "Great. Can you get it done yesterday?" "No." "Can you get it done in a week?" Probably not. "How about by the end of the year?" Still probably not, but let's go. Let's see what we can do.

So to kind of paint the picture of what this platform was: this is a survey engine platform we brought on about 18 years ago, and the customizations around it — I was talking to my boss, we're trying to resurrect the emails from that era to find out how long did it take them to implement. We have stories, anecdotes: "Hey, we tried to customize this." They said no. So we did this, and we did this, and they started building the platform kind of around our use case.

We had amazing data teams. An inventive idea of using Excel as our ETL. We'll copy from CSV and tab-delimited — and I know those are different, because why not, we should do both. And that was the platform. We'll manage things here, we'll copy them here with CSV, we'll copy them here.

I think in the LaunchDarkly talk they were talking about a hundred thousand lines of code that they were rewriting. We did 10,000 lines of XML multiple times a month just to get a survey out the door. And to get there, it wasn't copy and paste. It was Excel to SQL, run a stored procedure that spit out XML that we would copy and paste into another one and then have it do a join across these spreadsheets. We had a Six Sigma guy come in with process stuff, and he found all this amazing and wrote a lot of VBA apps for us. And then, yeah, we're trying to get rid of those now too.

So it was a giant pillar in our company that nobody knew. After 18 years, you can imagine people coming into this, our operations team — "You mean I went to school, and I got my MBA, and I get to play in Excel and copy and paste and do this day after day after day?" So it was like a revolving door of people not wanting this pain. But the product was doing so well. The customer-facing side was fantastic. We had built portals and applications and researchers, so they loved it. All inside the festering heart of NRC was this dying platform.

02Assembling the Team

So what did we do? We took three months to plan it out. We did an analysis and had that ready. No, we didn't do any of that.

Myself and my principal engineer, who lives close to me — I called him up. "Hey, let's do a road trip. We're going to Lincoln, Nebraska." It was great. We sat down with the two operations members that have been running this platform. They've only been there for three years — they've been at the company longer, but they've only been in this role for three years — and it was kind of a countdown: are these guys going to stick around to do this Excel hell?

We sat down with them and it started as a typical meeting. "Here's our list of requirements. We need all of these things." And I kind of said, pause — just show me what you do. Let's walk through this application. Show me your day-to-day. So as we started going through that, it's like, okay, what do you do to get these platforms live? What do you do to collect the data? What do you bring it back with? And that's where my eyes were like, oh wow, this is way worse than I thought it was — because we had a whole week and we couldn't get through the whole process.

So we were going through step by step by step, and instead of waiting for a product person or a designer to come in and start building this out, we recorded the meetings. We took pictures every time they pulled up an application or a spreadsheet or a query. We had one shared context we kept throwing everything into. By the end of that week, we had a prototype. And I just said I don't like prototypes — I don't like them so much. We actually had five, and we threw them all away day after day after day once they said, "Oh, this didn't work." We spent a lot of tokens, but not a lot of time, and that was really what was critical at this point with an end-of-year goal.

So the next part was assembling the team. I'm talking to this engineer, and I'm like, "Oh, we got work." We went and got sushi on a Thursday night in Nebraska. Don't get sushi in Nebraska — there's no ocean. We're sitting there talking about, what do we do?

The engineer that was with me is my DevOps guy. I told him I was talking to Kent Beck and he's like, "Oh, I hate you." So he's all in that space, and he's really wanting to go. He's like, "Well, we have the way to get here, but we don't have the people to do it." So it's like, all right, how do I get an army of people when hiring takes three months and I have to have it done in three months?

So the idea was, let me pick the team. I did my Rolodex on Friday, started calling up members of the different teams, talking to their managers: "Hey, who's switched on with AI?" And it's interesting — this is another study, Gene — the grumpiest developers I've met have been the ones that have been the most switched on and the ones executing, because now they don't have to go through all the cruft of engineering that at their principal level they've had to apply on their job.

So I call up Pat. He's my stereotypical angry Java developer that's been there for a long time. And it was, "Hey, Pat, I know you love being in the spotlight, so do you want to join me? High visibility. I might get fired. You might get fired. It's going to be so much fun." I look at my principal next to me and I was like, "You don't get a choice. You're just with me. I brought you here. You're going to leave with me if this happens."

Two engineers. We were given direction to say, "Hey, we need to get a product owner on this. We need to get a designer assigned." And I was like, "Great. You go do that. We're going to build." So we didn't wait for the product owner to come on board — he actually came on board day 80 of the 90-day countdown.

We started running, grabbing people as we needed them, and then about 30 days in we were able to get a manager to come on board. Again, I'm a great recruiter. I called him up and was like, "Hey, this might not last. You might come here and we'll all be gone in 90 days if this doesn't go well." And I was like, "Greg, are you in?" He's like, "Yeah." I was like, "Okay, we're all on AWS." He's like, "I'm all on Azure." "Well, you're going to have to learn." "What do you know about AI?" "Nothing."

I got a text from Greg yesterday. He's messaging me: "Hey, I've got a harness. I have a team lead. Closed 10 stories yesterday." My manager — not my engineers, my manager — closed out 10 stories around the phase two of this application.

03The Mission

So I framed this with the team — the Dirty Dozen, the Suicide Squad, the Magnificent Seven, the Seven Samurai, whatever you want to call it. All the movie tropes of like, hey, we're all going to get together, Rogue One — we're probably going to fail. And I am very inspiring. I start out with those.

And we did it, and I was amazed for two reasons. One, these guys wanted to work with me again — they had history, and they wanted to do it again. And the second was everybody was engaged. What I'd found out talking with some of the members is, "Yeah, we've been on products just kind of toiling. We didn't have a mission. We were just kind of a feature factory. Things came in, we would go." But now it's like, oh no, we have a mission, and it's live or die almost — at least career-wise. Can we make this happen? And the idea of a mission with that team changed the world.

04How We Worked

So what did it look like? We didn't do a long planning cycle. We didn't go through procurement for every tool we wanted to use. We got a straight line through me, through my boss, to our CEO: what do you need? If it's somebody to get out of the way, we're in. If you need to buy something, we're in. We'll talk about it later. It's 90 days.

So we started out with Claude. We already had Copilot — we used it in VS Code. It was awful. So we started bringing in other tools, open code, a bunch of tools. Each engineer, instead of saying, "Hey, this is the weapon, everybody gets to use this" — the discussion up front was, "Everybody try what you want, get the learnings, feel the pain, bring it back." So we kind of spanned a spectrum of tools, and that was the important part.

We didn't focus on building tools. We had a mission. We had to get this done. It was either going to be done with us working eight hours a day or going to be done with us working 20. The difference is how well we use these tools.

And I will say with this team, it wasn't at 5:00 that code stopped flowing. I'm getting texts at 9:00, 10:00. I'm getting the shaky, maybe-had-a-couple-drinks screenshot: "Look what I just did." And it was amazing. This team was completely on.

The tools were great. AI is amazing. It lets me do all the things I've wanted to do, and more importantly, it lets me do all the things I've never wanted to do — like presenting, which next time I'll have it do this for me too. But it's how do we use these tools? This whole team went from IDE to chat to agentic to harness in about two weeks. We blew through all of those dungeons in Diablo so quick. We just kept grabbing tools and upgrading.

I feel the power of the team wasn't the tool — it was how did we build around them? What did we do?

05The Lore Repo

So I stole this term: the lore. Context repo, context graph, knowledge graph — whatever we want to call it, there are a lot of products around this. But what we did was: this application had been built and it just kind of been working. Nobody knew the story. So we got into this phase of like, okay, I want to know exactly the 20-year history of this — the 200-year history of this elf and this dwarf. I want to know the name, where they came from, what they did. Really start digging that out.

That was interviewing the business. That was just grabbing their documentation. We got a little more free rein than normal, so we went into their shared drive and just gobbled it all up to write this story. It felt more like we were writing a story or a play than it did feel like we were writing software those first couple of weeks, because we wanted to know what we were doing and why.

We took all of that, it went into GitHub. We created a repo just for this — it's a great big pile of Markdown. And then everybody started connecting into that. All of our agents had this full context loaded in. We threw in other tools that started running embeddings and getting us into a data model where we could search it easier with semantic search and some weighting. But the idea itself is where the power was. Now we have this history, and now we're writing the next chapter of this history. So instead of being like, okay, we've got to write a whole new character — now it's like, hey, we're writing a new character, but we have all this history already about them, where they're from, what they're doing. We took that as a team and let the agents lead the way in our planning.

So we're looking at this and like, hey, we need to have a type-assist option in this survey platform. We need to be able to have users start searching for hospitals. It's got to be performant because it's every hospital in the US. It's got to be intelligent — where did they come from, what questions were being passed in? So we started with that story and had the AI help us build out the plan, or the chapter in my analogy. Start building out the chapter that was introducing this character into our new system.

AI was great. It got us 70% of the way there, which is 100% more than where it would get most teams spending weeks on it. So it got us 70%. We took it, we put it in Slack. We beat each other up — "Oh, I think we should do this." "No, that's stupid." Language is important, and the freer you can be with your language, the clearer you can communicate, and the faster you can move. That's why agents never get their feelings hurt. So just go.

And we started going back, iterating on this. One person would be like, "I love this chapter so much, I'm taking it to finish." So they would take it, get us from 70% to 95% of the plan with the agents, and then they would run, get it out, get it PR'd, get it in our pipeline and into production. We didn't wait. We just ran.

And part of that was our pipeline. Do we have a clear path to production for this new application? We're using some new technologies, and we didn't. So guess what we did? We added another chapter. More like an appendix — we drew the map of Middle Earth in the back, and that was our pipeline. How do we get this to go? Because it wasn't "hey, wait for five minutes." It was, "Let's go now."

As we were doing that, we kept adding to this lore repo, and it was fantastic because now I can see the history of where were we, where are we going, what are we building here, and how do we move forward. This included everything from Markdown with just words. We used Mermaid to get architectural diagrams in there to start sending those out. We would put in our code design patterns so when teams came in — and the other great part was co-learning. "Hey, I wrote a skill to just go do this." Great. Drop it in this repo. We all pull it down and we're all using that same skill now.

Every plan we had, we fed back into this repository. We would have it in our local, running, doing the agents, letting them go. But once we got it done, we fed in the original plan and the modified plan — okay, we did all of this, now what does this look like? We would put that back in there. And it really took us to the next level, because every story, every feature we did, we got better, but our context got way better.

06Feedback as Velocity

My goal to my team is: if we're in the office, I'm walking behind you and going, "What are you doing?" And you're going to show me what you're doing. Call them over-the-shoulder. We got those inside of our Slack now. Every day we're getting three or four videos — what is the feature? That has cut out so much friction on feedback because we have our business partners in there, product, design. Anybody who wants to join, I'll send you a link. You can come in, we'll look at it. We just get that feedback. The team will record, post a two-minute video. We have hundreds of these in our Slack channel now, and we're getting threads of feedback that we can run with.

Old model: plan, staff, design — we wouldn't have started the project yet. Now: plan, build, feedback — and that is a continuous loop, and we're doing that every day. We're not waiting. If a developer's not posting a video of what they did, I'm probably going to call them up and be like, "Hey, what are you doing?" They're like, "Oh, I'm building a new harness." I'm like, "Great. What is it doing?" And then I see the flurry of feedback coming in the days later.

My big red flag: I had a product person that's like, "Oh no, they're busy. We can't talk to them for like two weeks." I was like, "No, email them right now, please." Guess what? They got back with us. We're meeting on Monday. The idea of we're going to just assume and start filtering all of this context — that is death in this. You can't keep going.

So that's my red flag: any person that I'm talking to says, "Hey, we need to get back with you — ten days for your design." Really, guys? It's not beautiful. It's an internal tool. Let's go.

07Results

Going quick now. What happened? Three months. We didn't replace the entire platform. We replaced everything that we used, which was about 75% to 80% of its abilities.

Eight hours — we're getting Slack feedback, we're picking that up, turning it around. Eight hours doesn't sound impressive, but I wrote this about a week ago. I'm taking this down to about two hours now. So by the time somebody says there's a problem in our Slack channel, we have a deployment in stage and we're just waiting for them to say, "It looks good. Push it to prod."

Three stakeholders independently validated. We had security come in, our operations team come in — the survey ops are the ones who were doing the Excel hell — come in and do this every day. And they're all part of our channel, giving us the fast feedback.

So what's the big deal? We just protected 30% to 40% of our revenue from acquisition. These platforms have been getting bought and sold day after day by private equity or even by our competitors. Now we own that core piece of our tech stack. We saved $500K in vendor spend — yay. I'd rather protect the money. And then what's even more important is that manager is now spanning other teams and taking that same model and pollinating across teams.

08Testimonials

Coming up to the end, so I want to share some quick testimonials.

It wasn't just faster execution — it was a different operating model. AI compressed planning and development to the point where stakeholder feedback, not engineering, became the constraint or bottleneck.

Our operations manager — this was the person managing the people living through that day-to-day: "Their speed and flexibility, collaborative approach, have fundamentally exceeded my expectations for what's possible on a project like this." NRC has never seen this. They plan, hire, push, push — and then in a year, they've got it.

The product owner who joined at day 80: "AI has flattened the usual hierarchy by enabling access across roles to contribute at high levels, which in turn must become table stakes." So it's funny, I talk to this guy almost daily: "What is my job now? You guys are doing product stuff." And I'm like, "Help us." Every conversation now that I have with you should be the most important, most valuable conversation I have. I'm not going to ask you about acceptance criteria. Our conversation should be elevated — you're here now, let's focus on that. Let's focus on having substance in these conversations, not busy work.

09What Actually Changed

What actually changed?

Proximity over process. We got these teams together, same time zone — not in the same office, we did that for a couple of days — but keeping that closeness really helped.

AI as a team sport, not an individual tool. Not just a tool that each individual developer ran on their machines and got cocky about — "Oh, look at my beautiful AI tool." No, everybody came in, worked together. We started building each other up. And how much can we send to this new team member? The new guy really hated this.

Feedback is velocity. The faster we could get out feedback, the faster we could get it back, the faster we could move on to the next thing.

Constraint clarity. It was no longer: oh, product or design is going to sit with the business and they're going to take their filtered context, and then product's going to talk with them, and they're going to filter the context, and by the time it gets to engineering it's been so processed nobody knows what's going on — or it's one person's vision now. We got rid of that. We got in a room. We cut down the barriers, tore away silos.

Leadership coverage. Big shout-out to my boss, our COO, and our CEO. They gave us the blanket. They were providing fire cover for us the entire time. "Oh, you can't have Pat. He's so important over here." "Yeah, he is. Come on." None of that.

Validation became the bottleneck. We're still struggling with this. In fact, the project was done in 90 days. It is taking more than 90 days for them to validate and come back to us. We've run survey after survey through it, publicly facing. Over a million hits to the platform. Watched it scale up during peak times, and they're still like, "We don't know." And what I love is they're finding some issues — there's never no issues. And what we found out was, actually, those are issues that you told us to make from the other platform. So as this is going, it's: how do we get faster at feedback and getting us into the game?

Not every engineer is on board. Not every manager is on board. Not every designer is on board. My goal is: find the ones that are, bring them closer to this team, and let them grow. And as people see that, they'll love it and they'll want to be on that side.

Communication became a constraint. I've got teams in Vietnam, teams in Latin America, teams here in the US, even different regions in the US. How fast they could communicate, how clearly they could communicate without having to put filters on — that really drove the ability for this team to perform and execute. We didn't get upset with each other when we said, "Oh, that's a bad idea," or, "That's dumb," or other colorful language. We just went with it.

We're really looking at very localized two-person teams now. That's my org model. I saw somebody with 100 two-person teams — I'm not quite there. Maybe 20 two-person teams, maybe 30. But finding out that language and culture when dealing with AI actually makes a bigger difference than you may think.

The other part is: the tech is changing us. We thought we were changing the tech. Now it's changing how we behave and what we do every day.

10Call to Action

My big call to action: don't wait for perfect. You may not get Claude approved. You may not get all of these tools approved. You may never get every tool approved in your org. Don't wait. Find out where you're at, what you're doing, and start building and start proving. And once you've built up that trust, those barriers start to go away.

The help I'm looking for: how do I get other people into this without making them extremely mad at me? I've got a tendency to do that. I don't want to get rid of product people — I just don't want to talk about acceptance criteria. How do I start cutting those roles, really allowing them to mesh, so that the perfect team is a small group of high expertise, those conversations are substantial, and all the kind of waste conversations are handled by the models now?

And how do we validate at speed? How do we get feedback and say yes or no quickly?

Time's up. Thank you all very much.