Host Intro (Gene Kim)

So the next speakers are Max Reele and Austen Bruhn. One of the most rewarding experiences I've had over the last decade was working with Max when he was a lieutenant colonel in the US Air Force. He was the deputy commander of the famous Kessel Run in the US Air Force. It was the first software factory in the defense community, and together we — I got to study 35 organizations in the defense ecosystem, and we learned that two things really matter. And it matters for all of us. One is unity of command. Do we have one boss, clarity of mission, or do we have four bosses and no clarity of mission? And second is how close can we get to our customers, right? Obviously, we want to understand the problems even better than they do. We have daily access to them, and what we don't want is to not be contractually allowed to even know who they are, right? Can't expect good outcomes under those conditions.

So last week, I had the pleasure of being at their — he's now a VP of Delivery at Rise8, and he had this incredible conference called the Ship Summit. And I was there with Kent Beck and John Cutler, and it was an amazing experience where the goal was to get everyone to ship in three days. And I asked Max if he would be willing to share what that incredible experience was all about, what they learned, where the goal was to have everyone ship. And then he'll be co-presenting with Austen Bruhn, who created the infrastructure to make this happen. Like, what do you do when everyone has to ship and not everyone's a developer? Most were, but one was Max's executive assistant, another one was the head of HR. And so it was just an incredible experience.

Please welcome Max and Austen.

Max Reele

Great. Thanks, everyone. Gene, thanks for the intro. I noticed that you're the only one who is allowed to have a subdued intro. It's nice that you do a really long intro for the rest of us.

Yeah, so Gene intro'd it really well. We put together an event specifically for not just our company to upskill our company, but we really wanted to enable our partners, and we wanted to enable our customers. That event was called Ship Summit. It took place last week. We didn't really know what to expect — it was the first time as a company we were diving into this sort of thing. But luckily, we had a fantastic team that was able to put it together, and it went off kind of without a hitch, but a couple of things that'll come up throughout the presentation.

Why would we do this? Enablement is paramount, right? We can't levy these new AI requirements on the entirety of what is a tech company's staff without giving them the tools that they need to be AI literate.

Maybe this is a really loose analogy that'll fall apart really quickly, but just because we all know how to drive cars doesn't mean we know how to fly a plane. Like, yeah, sure, you might be able to get some air off a speed bump, but you're not a pilot. And so we can't expect that just because we hand the tools to the people, they're all of a sudden going to become AI fluent. And what makes you a really good practitioner in whatever your service is doesn't necessarily mean that you're going to be great with the AI tools at being a better practitioner.

So we thought, okay, let's put on a workshop where we enable everybody all together. Let's give them the tools.

I called back to something that Angie presented yesterday. I thought she did a great job of framing it. It comes from Gastown, so credit to Steve there. But the AI maturity model and where she said that they were as a company was very similar to how we were behaving as a company as well. We had the tools, we had the literacy, we were probably conversational, at least at most at that point. Our company leadership had really set a standard, and then our company practitioners were really diving in.

But we developed this summit so we can actually make Ship happen with AI. Not just tinkering with it on the side, but we wanted this to become a primary part of our workflow and a primary skill of all the employees within the company.

Now, not everybody loved it all at once. Our CEO put out an AI invite about a year ago — not quite a mandate, it falls short of a mandate, but it's an AI invite to get everybody working with the tools and doing everything they can. And as with any new tech that's being mandated, and that there's a hype cycle around, and that it's being talked about in every corner of the tech industry, there was a bit of resistance.

Unfortunately, as Angie kind of articulated yesterday, we don't have time for late adopters. We don't. This is just moving way too fast. Even this conference, where we're talking about AI leadership — we've gone to smaller chunks, faster repetition. Gene holds it monthly, because things are moving so fast we need to be talking about them and growing our leadership chops for it at the pace with which it's moving.

So we decided to go whole hog into Ship Summit. This was a massive investment by our company. As I mentioned, it wasn't just for our employees — it was for partners and for potential customers and current customers, all that kind of stuff. So there were over 200 people there. It's a three-day event. It's in Park City, Utah, so it's got this really strong kind of skiing analogy — that's why it's called Ship Summit.

And we set up four cohorts, and people self-selected into these cohorts. So if you were a complete novice at AI — all you had been doing is using it for chat and asking it what to wear for the day based on the weather — that's you. You're in the green circle. But then there's progressive experience levels up from there. So if you're skiing on the blue square, a little more advanced; if you're black diamond; if you're double black diamond. And then these were cut up across the venue that we were in, where people had their own workshop areas. It was facilitated by ski instructors — so we had people within our company that had already built the programming and run the courses and knew exactly how to help people troubleshoot through the courses. And the ski instructors wound up being instrumental to the program being successful.

We also had to build out all of the infrastructure for this. So we had two primary, really strong engineers within our company — Matt Pacione and Mike Gayhart — who come from a software engineering background and platform engineering background, and they spent over 70 days building out all of the infrastructure, all the configurations, all the pipelines, and the laydown that it took to make this successful, with plenty of iteration and rebuilds throughout that time.

So it was a successful event, as Gene mentioned. We'll get into some of the case studies and anecdotal stories about it a little bit later. Over 3,000 pushes to production occurred over the two-and-a-half-day event. 102 million Claude tokens were consumed during that time, so it got expensive pretty quick. We'll talk about that a little bit later in the talk too.

But to help us with all of the infrastructure and to be able to get the 198 workspaces spun up simultaneously and have all of this work together, we really leveraged a strong partnership with Coder. And Coder did a fantastic job along that journey with us. So introducing Austen Bruhn to help talk about that a little bit at a technical level.

Austen Bruhn

Hey. Yeah. Thanks, Max, for the intro there.

Yeah, Coder had the great opportunity of being able to be part of Ship Summit. I got to meet Matt and Mike, and they were telling us about this event, and I was super excited about it because in my mind, that's really what's necessary. We've talked about it kind of throughout this week already — Netflix doing big workshops. I talk to a lot of large system integrators that are trying to do workshops, trying to figure out how to get people to use AI. And without the infrastructure, none of this really works, right?

So — yeah, when I talked with Matt and Mike, they had already started playing with our open source and started figuring out, hey, how can we do this at a larger scale?

But ultimately, what we really thought about is: how do we reduce the barrier to entry for AI? So people in that green circle, like Max was talking about, have never opened a terminal in their life. Yet they're going to be opening Claude Code and they're going to be shipping code in less than three days. How are we going to make that possible? And then the other piece is, how are we going to distribute this across all these different cohorts? How are we going to make sure that everybody has the exact same environment consistently, and make sure that our blast radius is nice and small so that they feel confident going in and doing this?

So what we did is we've got a control plane that allows you to spin up different workspaces — and if you have more questions about all of that, come see me. But ultimately, what we did was an architecture that was actually really simple. It's four VMs, and we were able to run the control plane in an HA format that allowed us to do some auto-scaling, and it allowed us to create 198 workspaces at the exact same time. And everything was working very smooth and seamlessly.

But we did hit some snags, right?

What this was really trying to solve, like I said, was removing the barrier to entry as well as making this consistent for everybody. Because what we noticed — and what I've noticed kind of across the board — is that the works-on-my-machine problem is very, very clear, and it gets even worse with AI, right? Raise your hand if you've used Claude Code, and raise your hand if Claude Code has installed any software for you, ever, right? Exactly. So when that starts to happen, you start getting these bespoke systems, these snowflake systems, where now I don't know what's even on my system anymore, but Claude does, and it's going to continue to iterate on that, and now I've got this kind of crazy blast radius problem as well.

So this is a picture of the architecture that we used — sorry it's a little bit dark. But like I said, very simple. We've got a control plane that's running in an HA format. It's allowing all those workspaces to run on any of those different nodes. And it allowed for the true scalability for this conference. Over there on the side, we've got GitLab CI pipelines. We've got our bridge that's managing all the API keys. And what this really did is it prevented people from having to sit down and look at a guide or get a new laptop or something else. They were able to click a button and be off to the races in 30 seconds, right? Thirty seconds and now they're opening Claude Code and starting to ship outcomes.

And that's really what happened. I got to go to all those different groups — the greens, the blues, the blacks, the double blacks. And it was funny — I got to go into the green and I saw people using Claude for the first time. And then in the afternoon, they were already in Claude Code and shipping real things. But in the double black rooms, these guys were thinking far outside the box, looking at multiple agents. They had nine different windows open, doing crazy stuff. And all of this was possible because we gave them the tools to do what they needed to do quickly and accessibly.

The snag that we did hit actually comes from the way that this server works. So if you notice here, nobody had to think about networking. Nobody had to think about what applications sit where. All of that is handled by the control plane. And if you notice here, there's a Tailnet coordinator and a DERP relay, as well as provisioners and some other things that sit inside of our control plane.

And what happened is — if you go back to the other picture — you can see that's an auto-scaling group. So network connections and provisioner jobs without the right hooks don't like auto-scaling groups. So what happened is, everybody was doing great on the blues and the blacks, and then by lunchtime the first day, all those green people needed to start spinning up their workspaces. And so that auto-scaling group bumped up to eight, and then it was like, oh cool, we're all done spinning up all these workspaces, everybody's going to go ahead and slow down a little bit, we're going to scale back down to four. And all those provisioner jobs got orphaned. We had DERP relay issues with sockets and connections. And what we saw looked like a widespread problem, and we were very, very concerned.

So I sat down with Matt. We started hacking away. And I got to say, using Claude Code to troubleshoot as a DevSecOps engineer and an infrastructure engineer is incredible. But we were able to come down to the problem by pulling our team in, their team in. Within about five to ten minutes, found a solution to it. And then we did a poll and actually found out that only four people out of 198 were actually affected. And it all fully resolved itself within 20 minutes.

So for this event, it was incredible to work with these guys. It was awesome to be able to be part of the event. And I'll pass it back to Max to talk more about their impact.

Max Reele

Thanks, Austin. I appreciate it. The level of support that we got from Coder, and particularly from Austen, especially in that incident response, was just incredible. So a huge kudos to them. If you want to hear more about them and hear more about how they can enable these types of things for your organizations, their booth's over there, and Austen is just wonderful to talk to about his experience as well.

So what's the result of all this? Are we truly all builders now?

The example that's on the screen here — we were partnered with the Utah Avalanche Center for this event, which they were phenomenal partners as well, a nonprofit. And so our teams at each level were building appropriate levels of complexity of applications as they ran their course from green circle to double black diamond. And so this is actually one of the intermediate teams, but they built within their application a way that it's pulling API data from snowpack data and weather data for the day, so that it can give an avalanche rating to anybody who's going to go out there and get adventurous and go off trail, to know what their relative risk is for the day. But they built in a really cool feature here where they could take a picture, upload that, and then there's a decomposition of the picture to be able to tell you exactly what the conditions are in the specific location that you're operating in, based on the snow that it sees and its decomposition of that.

So that's really powerful. We made a lot of people who weren't builders into builders. As Gene said in the intro, our HR executive came down and started working on his project after hours, past midnight, because people were just getting addicted to what they were building and really wanting to see it through. We had admins within the company building things out. Our recruiting team built an entire new product to use here. So we've made builders out of non-builders.

But that sounds like kind of a lot of outputs, and it sounds a bit risky. So — we are all builders, and this is no shade to anybody, including myself, who hasn't coded in years but is now all of a sudden trying to pick this up and become a coder again. We know we're not the best at it.

And so I harken back to something that Kent talked about at the conference last week — perhaps we'll hear more from him later today about it as well. But it's like you can't quite trust everything that your augmented development is building for you all the time. It was actually mentioned yesterday by Eric too. He was super animated about the way he was saying it, but he was saying the agents want to reach their goal, and they're going to do what it takes to kind of please you as the user. And so sometimes we have to absolutely check them. And just because we all became builders doesn't necessarily mean that we have the seasoned experience of real technical professionals that have been engineering for a long time — to know the right questions to ask, to know the right tests to implement within the codebase to make sure we're checking our agents from just lying to us or passing tests that they haven't actually fully accounted for, and allowing us to push forward. So there is a bit of a cautionary tale here.

Everybody's builders, and everybody's excited about it, and they also have empathy for what it takes now to become an actually really well-seasoned builder who's going to build something with resiliency for scale. And so I say all that just to suggest that there's going to be a lot of bad building out there, and that should be okay. This is what we're asking everybody to do. We're asking everybody to become fluent in how to work with the AI tools. And for this time period — you can see it here on the graph — things are going to be really noisy, especially for non-engineers. And we're going to have to have a tolerance for that, because as we get across the adoption curve, things will start to calm down. But for right now, nobody really knows what's going on. And I'm not saying that based on myself — listen to Steve and Gene and Kent, and they all say the same thing. We don't know entirely how everything's working in the back.

So we have to be willing to tolerate this level of noisiness that's going to happen. And the noisiness looks like non-productivity.

And so I have this kind of fear that we're measuring people on token usage. And as we start to measure token usage, we're going to think everybody's great if they're using a lot of tokens for some period of time. And that's probably true, because they're exposing themselves to the maximum amount of learning. But if we start weighing that in some sort of value equation against productivity, we're going to fool ourselves into thinking, "Hey, this isn't making us any better at all," because we're not applying the appropriate amount of value to the learning that happens through the experimentation. So let's just have an appreciation for this and a tolerance for this. More experienced engineers understand this, and they can do a better job of testing the AI output. So we don't expect things to be super noisy for the experienced engineering teams. But for everybody that you've asked to be a builder that wasn't a builder three weeks ago or six months ago, they're still very much in this noisy part of the curve.

Okay. So what did this do for us as an organization?

We've built some organizational optimism around the use of AI. It was said yesterday by Steve: "SaaS is lava," right? And we have people within our organization that are already trying to rebuild parts of what otherwise we would license in an ERP. We have a designer paired with a really experienced software engineer, and that pairing is building out what is presumably a thick slice of ERP functionality for our company.

When I asked them to step into this, we had set a one-month goal for them to get to MVP, so we had something that the leadership team could tinker with and see if it was really helping us better than the Excel spreadsheets that we had been using in the past. And within two weeks, they were already past that. They moved so fast that I didn't even get to allocate a PM to their project yet, because I was still waiting for that PM to come available off of a different project. And so it's just incredible how much people can build.

And notice what I said — this is an experienced pair. It's not a full team of eight, right? PM, designer, six-pack of engineers. It's an experienced pair: a really seasoned software engineer with a very experienced UI designer and service designer. So that team is just absolutely crushing it. It's showing that the economy of scale is here, and that we can use experienced pairs that know how to test the AI appropriately and know how to push it for the right amount of leverage.

So I don't know exactly what that cost savings is, but it's certainly replacing something that we'd be paying at least — I'm guessing — a 30K license for, as Steve mentioned yesterday.

The other thing that was discussed yesterday from Tom was generating organizational FOMO a little bit, right? We want people to feel inspired to dive into the AI tooling. We want them to feel capable to do it too. And so that's why we made such a vast investment to get everybody the enablement they need and get them some experience with the tools. Now that they have the experience with the tools, they're diving into it unafraid. It doesn't mean we're going to push everything that any user does into production, but it does mean that we know that they're fluent enough to develop job aids for themselves, develop job aids within their work section, that are going to be pretty viable — instead of having to keep going back to the well and buying a ton of different SaaS tools for these one-off little unique features or capabilities that we need in terms of services.

How do I know that this FOMO was created? You can see it — maybe not based on lighting — but there's a background image behind these words. And the background image is of the conference room, listening to one of the talks towards the end of Ship Summit. We had a lot of practitioner coding going on. It dominated the majority of the time. But that was also interspersed with a bunch of talks from, as Gene mentioned, Gene and Kent and others, that were a practical way to look at the leadership of what you were learning. So it was in and out, in and out. While we were at the final talk of the event, after the two and a half days, you can see from the back of the room, half the room had their laptops open and they were still vibe coding on their projects. The coding part of the event was over, but not for them. They wanted to keep going.

And we had a lot of success. From the early parts of the project, the green circle teams — their average time to push to production was about four hours. And that would be expected, right? Some of these folks have never pushed anything to production. So it took them a long time to kind of run the course, learn the material, and then get to the point where they were pushing to production, so they had a viable MVP that they were able to tinker with on the app. But if you looked at the double black diamond teams, they were on average pushing to production for the very first push of their MVP in under 30 minutes.

So it's important to note that we're talking a lot about the green circle teams and how exciting it was when they were demoing, because we're turning everybody into builders. But even for our much more experienced engineers that were there in the black diamond and double black diamond, they were picking up a lot out of this too, and learning how to leverage the tools more reliably and more safely, and certainly with greater speed.

So I have a bit of a call to action. Gene always asks when we do a talk at these things that we should ask for help at the end, and so I've got one.

I have this fear — from back when I was running platform and infrastructure teams — that we're measuring tokens now, and we're letting people just use as much as they think is necessary for their job. And so we're going to see a lot of token usage. Like we talked about even for this event: 102 million Claude tokens. This reminds me of back when cloud costs started ballooning, and if you didn't put appropriate guardrails around platform and infrastructure engineers, your cloud costs would go wild because they're not burning down environments. Everything's just up and running all the time.

And so if we start to look at it that way, we're going to quickly get scared and ask people to stop spending on their tokens. So us, as leaders within this community, need to think about how do we responsibly handle this so the costs don't get overwhelming, and then we have a knee-jerk reaction and we don't let people use the agents any longer. I don't want that to be the outcome.

So here's what my ask is. I have a hypothesis. I think, like we wound up doing with cloud costs that were ballooning, I think we have an ability to suggest that your more novice people — in the early part of that curve where things are really noisy — give them unlimited usage. Let them experiment as much as they possibly can, because the learning cycles that come with all that experimentation is the value that you're garnering out of that. It's not really a productivity gain yet, but it's the learning value.

But for your more experienced engineers that are wanting to burn the heck out of tokens the entire time because they're building really complicated systems, and they're wanting to pair with the agents to make sure those systems get out there as fast as possible — why wouldn't we cap that? Why wouldn't we force them to be more responsible with their token usage? And there's a really beautiful outcome that I'm hypothesizing here. If your junior people have unlimited usage of tokens and your really senior people have capped usage of tokens, your really senior people are going to have to go find a junior person to pair with to burn their tokens to run their projects.

And so I think your junior people are going to garner much more learning out of that, because they're going to get to be paired with one of the more senior anchors as they get that done. So here's my call for experimentation: if you have an organization that's set up where you can force that type of pairing through this kind of incentive structure on the token usage, try it out, and then hit me up. Let me know how it's going, and I'll do the same. And we can aggregate our data and see if we have anything special there.

All right. Thanks, everyone, for paying attention to our talk. I appreciate all the enthusiasm and the support that we've gotten from this community.