Host Intro (Gene Kim)

The next speaker is Charity. So I'm so delighted that she's here because she's most certainly one of the central figures in the observability community. And I met her in 2018, and I just loved the book that she wrote with Liz Fong-Jones, Observability Engineering. And I think she's certainly one of the people credited with injecting so much energy into the SRE and observability community.

So she's the founder and now CTO of Honeycomb, and like me, she's a huge fan of feedback loops, and the value of immutability — not just for developers and functional programming nerds, but also ops too. And so I was so excited when she reached out to me describing how she disappeared for three months trying to finish up the second edition of her book. God bless her. And she wakes up, reenters the world, and apparently coding is a solved problem now. She's like, "Holy cow, what did I miss?" And so I laughed, and I cried as she described her view of the world, and how vibe coding might achieve some of the hopes, dreams, and aspirations that we had in the DevOps community for over 10 years.

And so I'm so pleased that presenting next is Charity.

Charity Majors

This is my first time coming to Gene's IT Revolution. It's been a lot of fun. I can't believe I haven't made it here before.

All right. What Handcrafted Servers Taught Us About Handcrafted Code.

So nine months ago, last June I think it was, I wrote a blog post called "Disposable Code is Here to Stay, But Durable Code is What Runs the World." At nine months, it's already out of date. Half the things that I said are no longer true.

Excuse me, it's very warm — and also I feel like my T-shirt is an important prop. It does say, "Test in prod or live a lie."

In my blog post, I made some large statements. The post was about disposable code and durable code. The difference between disposable code and durable code is not whether AI generates it or not. The cost of software is defined by the cost of its maintenance, right? And the cost of disposable code is very low because you do not even try to maintain it. And there are so many use cases that have sprung up for disposable code since ChatGPT came into our lives in 2022.

And durable code — or what we used to just refer to as code.

I said, call me old-fashioned, but there is no test suite on Earth that could make me trust a new chunk of code more than I trust the code that's been running in production for two years. That's how trust works. It takes time. The only way to build confidence in your code is by exposing it to reality in a controlled setting, then iterating on what is known to be good and stable.

And this is a truism of durable code: trust lives in the code. The change must be small and incremental. All these things that have been drilled into us our entire careers. Don't change the code. I've said so many things over the past — however long — years about how the definition of a senior engineer is one who is paranoid about production. They know not to make big changes. They have it deep in their bones. Don't change it. You're going to hell if you do this.

All of this makes sense. Indeed, it was the only sensible approach in a world where producing code was very costly, and time-consuming, and difficult. And battle-hardened code was scarce and rare.

It's kind of like how the pioneers never threw away a scrap of cloth. Your dress gets holes in it? Well, that's fine — it makes perfectly good washcloths. Somebody grew the sheep, picked the wool, carded it, spun it, wove it. You don't just throw a dress away because it has holes or stains. Well, that's how we've been about our software, right? And this made sense.

It made a lot of sense for us to — our careers were built on caring about the beauty and elegance of our code. And this made sense because it was the best proxy we had for how maintainable that code was going to be in the future. Because we knew that our ability to read it and understand it would have to gate any change.

Code that graduates into production becomes so much more than just code. It is the fossilized record — the only record that exists — of developer intent, business needs, bugs we've fixed, bugs we've not fixed. User expectations, the contracts and expectations and dependencies of the business, the dependencies of other software. This is the sedimentary layers of accrued expectations and long-forgotten intent.

The cost of software has always been defined by its maintenance cost, as a direct consequence of all these things — that if no one understands them or knows what they are, we certainly can't test for them. It has been necessary for the same reason that the only way to safely change the code has been with a series of tiny incremental diffs, because most of the behaviors have been poorly understood, implicit, discovered mostly when we break them.

The code is not the system, right? The system is the unique, irreplicable intersection of code, infrastructure, user patterns, builders, data, and time. Artifacts accrue meaning as users develop expectations. Like, as soon as you ship something, it's an expectation. It's a contract you've explicitly made.

Anyone here ever do a complicated database upgrade or API rewrite? How messy was that?

This has always been true. But because writing the code was so hard, it sucked up all the oxygen in the room. The most important parts of our system have never really been written down. This is why it's so expensive.

To bring down the cost of software, we're going to have to bring down the cost of maintenance. And with all due respect to my dear friends Steve and Gene, who wrote a whole book on vibe coding — that works for some things. But what does work for the commitments that you've made to your users is targeted replacement behind stable boundaries. Stable interfaces. Replaceable components.

This is the same logic. Does this all sound familiar? It should.

Battlestar Galactica. It's all happened before, and it will all happen again.

I am just barely old enough that my first job title was system administrator. Anyone else? Yeah.

I was 17 when I went on-call for the first time. Fate of the world on my shoulders.

What are sysadmins known for now, in the golden haze of memory? Handcrafted server pets.

By the way, this is my definition of infrastructure: the code you have to run in order to get to the code that you want to run.

In 2013, Chad Fowler coined the term "trash your servers, burn your code" — immutable infra, disposable components, immutable infrastructure. It stuck. By the way, I am quoting a lot of Chad Fowler in this — I'm probably plagiarizing from him, so I just want to give all kinds of credit up front. It's one of those things where you hear someone say something and you're like, "Oh, yes. That makes sense." He's been writing about Phoenix architectures, sort of a continuation of the Phoenix server thing that Martin Fowler and all the — anyway.

The lesson that we learned from immutable infrastructure, which is the lesson that we learned from functional programming, is this: mutability is the enemy of understanding. Systems that get edited and changed in place fall subject to entropy and drift. They fail in ways that are hard to diagnose, hard to reproduce, hard to resolve, hard to handle in any sort of programmatic way. They run on tribal knowledge.

When you edit a system in place, you guarantee that you don't know what happened to it.

When failure was rare, it was really, really, really scary when it happened. Oh God, we haven't rebooted this in 200 days. And now it's hanging. Did we do any backups?

What did we learn from infrastructure? You cannot regenerate what you have not yet defined.

So in the early-to-mid 2010s, we stopped patching servers. We started regenerating them from templates. This is great. Turns out, the server was never the thing we cared about. What we cared about was intent, reproducibility, functionality. We made infrastructure easier to reason about by taking its history away.

At Honeycomb, something we do is we kill the oldest Kafka node from cron every Tuesday at noon. Shoot it in the head. Because it was flaky, and we kept just getting paged. And so now we kill it ourselves. Now we always know it works.

Cannot regenerate what you have not defined.

Developer intent, specification, evaluation. When you can regenerate your code, you understand it.

Here is a quote from Chad: "When you edit code in place, you're doing the software equivalent of SSH-ing into a production server and tweaking a config file."

I love this. Code becomes cache. If you understand the commitments that you've made, the functionality it's supposed to represent — if you can specify it and evaluate it — if you can regenerate the code while keeping all your commitments to users, maintainability becomes a property of the system, not the code.

Now, I want to be very clear that the tools to do this do not exist yet.

The ideas to build the tools to do this absolutely exist, and they come from two branches of engineering that have often been regarded as somewhat lesser than — a bit snobbish on behalf of software engineers, if you ask me. It comes from operations and QA. I know, not first-class citizens of software development. But turns out we've been doing this stuff all along, right?

In fact, there was a book written in 1994 by one Michael Weatherby that talks about characterization tests — tests not for what the code is supposed to do, but for what it actually does. Which seems useful. Seems relevant to our interests.

Ops is full of — feature flags, feature flags and high-cardinality granular observability with rich context, and progressive deployments. This is what you need to test in production safely. Testing in production is just as important as testing pre-production, and software engineers are about to learn this.

What we learned from handcrafted servers taught us this:

- Mutability is the enemy of understanding. - When failure is rare, it is terrifying. - When failure is ordinary, it is boring. - And the value lies in the system, not the implementation.

How do we get from "how do we write safer code" to "what must exist so that the code can be replaced safely?"

Chad has this great piece about how this discomfort that we feel with fresh code — it's understandable, but it's actually misplaced. What we actually fear is undetected behavior change, performance regressions, security regressions, and silent drift. Those failures are caused by unobserved change, not by newness.

A system with stable contracts, strong evaluations, continuous monitoring, and clear rollback paths can safely tolerate very fresh code. A system without these things is dangerous even if the code is 10 years old.

Throughout computing history, working code has been expensive and hard to produce, so we have changed it parsimoniously.

Be clear that software has a gradient of risk. There are many use cases for disposable code, and there are many places where you can go vibe code your heart out today and ship it straight to production.

Have you all heard of pace layers before? There are layers of a system that can safely change at different rates.

There are many places where you can just destroy and regenerate without having to worry about stable interfaces and bounded replacements. And sometimes I feel like there are two groups of us kind of shouting across a chasm at each other, where some of us who are used to gnarly changes are like, "What are you doing?" And people who are not — I'm just saying there are different perspectives.

Code in production accumulates dependencies, which is a good thing. It means people use it. People like what we build, right?

But when software meets reality — and by reality I mean humans and data, basically — the reason, are y'all familiar with the argument about whether software engineering is really engineering or not? Super annoying, very pedantic, much fun over drinks. My interpretation of why this is such an evergreen argument is that every other type of engineering is held honest, kept honest by reality in some way. Electrical engineers, they've got electrons and AC and DC and all the currents and stuff. Civil engineers, they've got traffic. And ours — the structure we get is the structure we give ourselves. We are magpies for metaphors and ideas. Like Kent Beck — design patterns, borrowed from architecture, the language and the concepts. Biology, emergent behaviors. "Production," we borrowed from manufacturing. We're just like, "Oh, I like that idea. Oh, I'll take that. Cool, I'll take that too." Right?

The only structure in software we get is the structure we give ourselves. But that does not mean that it is not real or does not matter. The farther you get away from bits, the more it's like, "This is a glorious metaphor, I shall use it." And the lower it is, it's logic, language, and metaphor. But eventually, it collides with the physical world.

And the two big places that our beautiful theoretical world of bits and logic meets reality are persistence problems and humans — both of which have a well-understood dislike for too much change.

The last thing I want is to open my phone, pull up Slack, and have the buttons move around every day. Please do not change my UI any faster than you absolutely have to.

And persistence problems notoriously have a tendency to be permanent.

Structure matters. Just because we gave it to ourselves doesn't mean it doesn't matter. It means that we have to enforce it ourselves, or the whole thing falls apart.

Value — SREs sometimes get this backwards, and they're like, "Well, I put five nines on it. That means it's valuable." Sadly, no. But value — I've been trying to think of an example where value is not backed by durability, and I've been struggling to. Might be my own biases. But value is usually backed by some kind of, "Oh, I like this. I want it to stably exist."

The transition from treating code like a sacred relic to code as cache is unavoidably going to be a little messy. We need to unbundle our expectations from the artifact and move towards explicit specification, evaluation, tracking of our commitments. We need software engineers to stop living in a beautiful little world of how things ought to happen, and come join us in the messy land of reality where things actually happen.

Nostalgia aside, no one really misses the bad old days of handcrafted servers. I do not miss getting a phone call in the middle of the night, having to call a taxi to go downtown and flip the power switch on my SQL primary. I do not miss that. I do love me a good command line, but that's why I have my Raspberry Pi and my little FreeBSD side piece.

And someday, no one is going to miss the bad old days of handcrafted code that you have to do capture-replay and strangler fig patterns on to migrate either. There will come a day when we will sit around drinking, reminiscing about the worst software rewrites we were ever part of, and all the kids will just be like, "What?"

In the end, the sysadmins — it was rough. It was emotionally... your identity is wrapped up in it, and it's something that's very lucrative. Salaries for everything in the post-industrial landscape have been going like this for everyone except us, right? We were special.

In the end, though, most of the sysadmins got on board. They built the systems that replaced them. Wasn't easy, but we got through it. The feelings are normal, and we should make space for them.

And this is where I'm going to switch gears just a little bit. I want to talk about the expanding gap that I see between two groups that I will characterize as management versus engineering sometimes, AI enthusiasts versus AI skeptics other times, or early adopters and, well, ops people.

By temperament, I am inclined towards the second bucket. But as someone who started a company 10 years ago, it's literally my job to be in the first bucket. So I have a lot of sympathy for both. I really do.

And I feel like I've seen this at so many companies I talk to where engineers feel like they aren't allowed to criticize AI. They aren't allowed to be skeptical. They aren't allowed to talk about the problems that they're seeing, even though they're downstream of all the enthusiasm, and it's getting harder every day.

Very few companies have a discipline around this, because we don't — as was said earlier — we don't know these things yet. And maybe it is the right strategy to just brazen it out, and the models will get there and fix everything. Maybe. But I see a lot of engineers in this industry who are not reflexively anti-AI, but they're almost getting pushed into a corner to hate AI because they aren't allowed to talk about what they're actually going through.

And the thing is, I also have a lot of empathy for the managers and execs in the room who are like, "You guys never want to upgrade. You guys never want to do anything new. We're missing the boat. Our board is on us. We are going to get left behind." I hear that. I see that. You're going to have to push. We had to issue a mandate. Wasn't happening on its own. It's frustrating.

When one of our staff engineers complained to me that he felt like he wasn't allowed to criticize AI in public, I was like, "Yeah, dude, your voice carries a lot of weight. It's really frustrating for you to be visibly seeming like you're not on board." We've got to turn the ship. We have to build the credibility for ourselves to critique it.

But in conclusion, my strong advice to management at all levels, and anyone who is an enthusiast who genuinely wants to do good and not just make a name for themselves on social media: if you're forcing them to do something that they don't want to do, you better be right.

Avoid the reality distortion field. Don't just listen to other people for whom it's working. I suggest you go hands-on, try it for yourself, really walk a mile in these folks' shoes. Don't assume reflexive — I've heard so many games of telephone that get surfaced after two or three hops and get translated as, "so-and-so hates AI and doesn't want to move fast." I guarantee you that is not actually — 99% of the time, I can guarantee you that is not actually what's happening.

So try to bring them along. The people who are honest skeptics, who are open — you bring them with you. The best way to bring them with you is to go walk a mile in their shoes, see what they see. Don't force it, because if you are right, yeah, that's the perks of the job. If you are wrong, they will resent you for the rest of their life, and they probably should.

All right, I'm done.