Host Intro (Gene Kim)

Okay. Up next is John Willis, another central figure in the DevOps movement. How to introduce him — over the last decade we've been on so many adventures together. We did the Beyond the Phoenix Project podcast together. He was a co-author on The DevOps Handbook. He's always reader number one for anything that I write. And he came out with a book on Deming called Profound — Deming's Journey to Profound Knowledge.

So he's going to be talking about an incredible working paper from an amazing group of technology leaders who are inventorying all the types of technical debt being built up right now, and the incredible frenzy of distributed innovation that's happening, which someone is eventually going to have to clean up. And so, as they put it, it's like shadow IT all over again — but much, much worse.

So here to talk about that is John.

John Willis

Hey, y'all. Before we get started — if anybody questions my commitment to Dr. Deming, these are my socks. My son bought me [them] for Christmas. (~1:00)

01Dear CIO

So, dear CIO. You know, we've had a lot of AI in today. Gene wanted me to say this — I sent him [a note]. I was at John Allspaw and Paul Hammond's '10 Deploys a Day at Flickr' [talk]. I was in the room for that, right? And I really felt like this morning and today is gonna go down as the '10 Deploys a Day at Flickr' for generative AI in the enterprise. It was such an amazing [day]. Maybe I'm all in — you might not be as far in as I am — but I'm gonna do this thing called Dear CIO. It's about technical debt. This is gonna be really corny, but it could be fun and it could be terrible. Every time I say 'Dear CIO,' you say 'What, John?' Okay. Think we can do it. Alright. (~1:50)

02The paper and the Phoenix project for AI

We wrote a paper — Gene talked about it. I've been an infrastructure person my whole life. Operations, mostly. What do we do in infrastructure and operations? We protect the fort. We're protectors. So every time there's a new technology, I think a lot about — what is this gonna do to the protectors?

Each year Gene invites a bunch of us. We work on these research papers. The last one I did was turned into a book called Investments Unlimited. So it's sort of an automated governance, Phoenix project for DevOps, ready to go. So that's what we did here. We created a Phoenix project for AI, right? Very short. Just a shout out — it was Tapabrata 'Topo' Pal, Ben Grinnell, John Rauser, Damon Edwards, and Joseph Enochs, who's been my partner in crime for helping me with GenAI. (~3:00)

03Bio in passing

Gotta sell some books. I've written like 10 books, done like 10 startups. I'm not gonna give you my bio now 'cause it's a waste of time. That's my Deming book. All the stories I cut out of that book — that people told me they shouldn't have been in the book, but I didn't listen to 'em — created another book [Profound Stories]. And that's the book I'm working on now, which is the history of GenAI — Attention Is All You Need. Hopefully it's done by the end of this year. If you read my Deming book, it's very storytelling — and this one's really about neural networks and all that good stuff. (~3:20)

04_slide: "Dear CIO, Learn From the Past" — 1980s Automated Operations / 1990s Systems Automation / 2000s Infrastructure as Code / 2010s Composable Infrastructure / 2020s Agentic Infrastructure_

Alright. Dear CIO — John — can we learn from the past? Like, you know, shadow IT. I've been doing [this]. I'm old. I'm really old, right? I know I look younger than I am.

I started this stuff in 1980. In 1980 in a data center I worked at Exxon, there were like a gazillion consoles of just mainframes, and one poor schlump had to sit there and answer all the questions and monitor. So a few of us got together and we sort of [thought] — could we automate a lot of those stupid questions, just catch it, answer it, like things that you would do every time? (~3:40)

So we did that in the nineties. No disrespect to my great friends at IBM, but I have scars up and down my leg with Tivoli, a product that really took a couple years off my life. But it's system automation, distributed systems automation.

And then, infrastructure as code. This was awesome stuff, right? CFEngine, Puppet, Chef — I was early in it. Adam Jacob invited me into that [Chef] party. And in 2010, [I] did something really stupid — I sold the company to Docker. Disposable infrastructure.

And so now we're into another age, which I'm just calling agentic infrastructure. You could hear it called agents. There's all sorts of names for this, just agents in general. I'm toying around [with] 'agent DevOps,' but I know I'm sort of stupid that way. (~5:00)

05Ops people are like cicadas

But here — what's the [thing]? You know what ops people are like? Cicadas. Literally — we go to sleep for, I don't know, 10 years, 15, 17. I don't know what the real life of a cicada is. And then we wake up and what do we find? I'm just gonna try not to curse, and I won't this time — but I probably can't promise you I won't curse next time. We find a mess. And then we start cleaning it up.

Here's the thing. Every one of these phases that I've been through in my career — there's the promise, there's the delivery, and there's the gap. And what happens with the gap? Anybody want to answer that? They get shoved under the rug. We sort of move on — good enough. And some gaps are like tree rings — bigger, smaller — but it's all weird.

And so how come we don't learn? So I'm 65, I'm sounding the horn one more time. Hopefully I don't live to the next one. (~5:40)

Can't we learn that there is technical debt associated with every one of these promises? And they deliver stuff — by the way, without infrastructure as code, you don't have Uber, you don't have Airbnb. I know that directly — 'cause they ran Puppet and Chef to build that infrastructure at the scale that they did. (~6:00)

06_slide: "Dear CIO, Learn From the Past" — L Language Model Orchestration (LLM/SLM) / O Observability / R Retrieval Augmentation Generation / M Model Providers / A Autonomous Agent Management — the LORMA stack_

So one of the things we learned — oh, so dear CIO, can we learn from the past? Everybody remember the LAMP stack, right? The LAMP stack was really helpful, because all this stuff was blowing up on AWS and all that, and it gave us a context of how to understand — Linux, Apache, MySQL — if you're old like me, it was Perl. So I started thinking about — how, what is my role? I try to educate people. (~6:40)

What we're finding is a massive disconnect. And this paper is about this. In case I run out of time, I need to let you know — 'cause Gene will come up here and chase me off the stage — we need to train the people who protect the fort as fast as possible on what this stuff is.

So I'm not looking for a plaque — like 'he invented the LORMA stack' — but I think this is a good way to [frame it]. And Brian Scott had sort of some of this this morning, where he said — he didn't say 'stack,' but: L for language model orchestration — your LangChains, your LlamaIndex, your, you know, your Haystacks. (~7:20)

Observability. By the way, observability is different in GenAI land. It's not like your Dynatrace and Honeycomb — it is about evaluations. Patrick talked about that — hallucination management, correctness. You need to monitor that stuff, and Patrick did a great job explaining it.

You've heard enough about RAG, I hope. RAG is an important part of this picture. And then Patrick did a good job of [explaining] the models.

And then these agents. I've talked to some of these large consulting firms that are bragging about building 10,000 bots in your company. And the first thing I think about is RPA. Anybody who lived through the RPA technical debt monstrosity — like I [did] — you shouldn't be bragging about that. (~8:15)

07SRE for GenAI — the vector database question

So here's the thing — and I'll come back to this later. In case we're out of time: I think we need to have an SRE influence here. 'Cause I think the question should be — if your CEO or CIO comes to you and says, 'Hey, HR's bringing in this new AI tool,' I want you to be able to say — if you don't already — like, 'What vector databases are you using? 'Cause we're really good at these.'

And by the way, I do consulting for [Tapabrata] 'Topo' Pal at MongoDB, who has a vector database. So I'll just say — oh, we have MongoDB Atlas Vector Search, or we have, um, Chroma DB. But if SRE [is] managing it, those are the two you get. And if you don't want to be SRE-managed, you can pick any other one you want. (~9:00)

And so — like Brian had — we're [on] LangChain. Where it can't just be one. So we're two. You can have one. We're gonna have a certain set of models — probably not more than two, but not a thousand. And I don't think you want 10,000 bots running around in your company. (~9:30)

08_slide: "Dear CIO, Prepare for the Tsunami"_

So dear CIO — you need to prepare for the technical debt tsunami. And it's two orders of magnitude [worse]. Gene talked about shadow AI [vs.] shadow IT. There's a shadow AI coming, and it's gonna be way worse if unattended. Like — I've seen this movie, I've been doing it, I'm old, I've seen this many, many times. (~9:50)

09_slide: "Dear CIO, Don't be Another Air Canada"_

Don't — dear CIO, don't be another Air Canada, right? You saw the Air Canada thing? The poor guy went to a bot to get a discount, [and] when he went to collect, they said, 'Oh, that was the bot. That's not the policy.' In the end, it lasted about four years [in court]. I guarantee they weren't using any OpenAI GPTs — doesn't matter.

The lawsuit settled for $2,000. A $6 billion market-cap company got a terrible brand tarnish for $2,000, right? That's how simple [it is]. What do we do? We protect the brand, right? We don't want our bots giving [stuff] out. I mean, it could have been a lot worse, obviously. (~10:30)

10_slide: "Dear CIO, Here's the Numbers" (Microsoft/LinkedIn 2024 Work Trend Index — 75% of knowledge workers use GenAI at work; 78% BYOAI)_

Dear CIO — it's gonna get boring, but we'll keep doing it. Here's the numbers. So I was saying, before this report, I would say to people — shadow IT, if you had 100,000 employees in a bank, less than 5,000 were probably doing AWS CLI or API calls, right? So look at the work we had to do to clean up shadow IT for like 5,000 people in a 100,000-person organization. So I would say — I think it's gonna be 70% in these new [AI] organizations.

Well, right when my good friend David Edwards [Damon Edwards] comes up to me — I'm about to give a presentation — [says], 'You need to see this slide.' And Microsoft / LinkedIn says it's basically 78%. So 78,000 people in your 100,000 are gonna be doing GenAI. I'm not great at math, but that's more than two orders of magnitude more complex than shadow IT. (~11:25)

11_slide: "Dear CIO, This is an Infrastructure Opportunity" (Wiz Research / Hugging Face)_

Dear CIO — thanks, Steve [McGhee]. Unfortunately, I was in the back room, I didn't get to see it, but I know he's a brilliant guy. He talked about poisoning models. If you haven't seen what these Wiz guys are doing — it's amazing. The Wiz ones they did recently — I don't know if Steve talked about this — they poisoned a model because of the pickle code in a model. They put it out on Hugging Face, they escaped out of basically the inference model, and they found probably a zero-lockdown Kubernetes cluster. They broke out of that. And from what they said, they got onto a shared Amazon host — by just doing a chat GPT [prompt], with that particular model. (~12:10)

So here's the thing — this is not an AI problem. AI is network, compute, and storage. Sorry to disillusion you about everything you learned today. It's amazing stuff. The icing on that cake is incredible. But if we don't treat it like network, compute, and storage — dear CIO — you're gonna have a problem. (~12:30)

12_slide: "Dear CIO, Google Warned US in 2015" (Sculley et al., "Hidden Technical Debt in Machine Learning Systems")_

And so when we were writing this paper, I found this thing — dear CIO, Google warned us back in 2014–15. They have this brilliant paper of what our future's gonna look like. They literally talk about the hidden technical debt and it is just awesome. And we have some of the things listed in our paper. When you read it — oh yeah, that's gonna happen. Yep, that's gonna happen. And you don't have to be a scientist to understand what they're saying. (~13:00)

13_slide: "Dear CIO, You Run It, You Own It." (AWS Bedrock contact-center architecture diagram)_

Dear CIO — I have a love-hate affair with Bedrock, I gotta admit. Once you set it up, it's pretty cool. But take a look at that picture. How much on that picture is actually AI? Not a lot. This is an infrastructure problem. People are building scalable AI solutions — and you know what they look like: they got Kafka, they got Kubernetes, they got Redis. For some strange reason, they still run MySQL. And even though they have a vector database — but I, you know, who am I to judge?

It's an infrastructure problem, right? And here's the other part — again, I keep saying I'm gonna run out of time. So I want to get the core of this message in. (~13:50)

14The core message — CEOs vs CIOs vs Chief AI Officers

The core of this message is — dear CIO — the CEOs are hiring Chief AI Officers, and they're basically telling the CIOs (from my experience) not to slow them down. And so they're the ones [building this]. I'm pretty sure nobody who did sort of DevOps at Hugging Face has ever been to this conference — 'cause they probably wouldn't have put in a default Kubernetes cluster that could get escaped out of very simply.

So the problem is: we don't want to slow down the AI experts — but they don't know what we know. And they're gonna build infrastructure that is gonna be lethal to your organization. I hate to bring you down, you know, after all this great stuff all day, but — trust me. (~14:30)

15_slide: "Dear CIO, Threats of Not Implementing GenAI" (Competitive Disadvantage / Market Perception / Innovation Stagnation / Operational Inefficiencies / Inefficient Allocation of Human Resources)_

The last slide was the threats of not doing this. So this is really clear. OWASP has a really good paper on the threats of [doing and] not doing AI. So if you're still on the fence — like, I don't know if I can — after today, you shouldn't be on the fence, right? When [you heard] Adidas — the numbers from Adidas were like mind-blowing. (~14:50)

16_slide: "The CAIO Dilemma — The Case For" (Strategic AI Implementation / Reducing AI Fragmentation / Managing AI Risks / Driving Transformational Change)_

17_slide: "The CAIO Dilemma — The Case Against" (Potential for Conflict / Risk of Overemphasis on AI / Additional Costs / Potential for Shadow AI)_

There's a pros and cons of a Chief AI Officer — and that was sort of one of the cores of our paper, right? The dilemma — the why would you hire a Chief AI Officer? Well, there's a lot of expertise. They bring in a lot of knowledge. They understand the models. A lot of us don't have a master's or PhD in AI or GenML, MLOps, whatever. And these people do, right? So — yes.

In our story, the Phoenix-style version was: everybody's happy in the end. The CIO is concerned that the CEO hired a Chief AI Officer, but they get together, they have a great conversation, and they realize they're gonna work together — and hopefully they don't have default Kubernetes clusters and tons of vulnerabilities. (~15:50)

But the case against — which I think I've been making — is: if you put a Chief AI Officer out on an island and just tell 'em to go fast, and tell your CIO (even though your CIO instinctively knows this is a bad idea) [to stay out of the way], and you let them just build infrastructure — it's gonna get ugly. (~16:30)

18_slide: "Dear CIO, Here's Some Suggestions" — Legacy Migration Assessment / Security and Privacy Training for GenAI / Platform Engineering for GenAI / Service Reliability Engineering for GenAI / Secure Supply Chain for GenAI / Automated Governance for GenAI_

19_slide: "Dear CIO, I Need Help" — Agentic DevOps / SWE-Bench (Copilot Workspaces, OpenDevin, Aider) / Real Enterprise Code (Banking, Retail, …)_

Here's the deal. Patrick showed you this. Can I just tell my help? Yes. Alright. So this is important too — I am really fascinated about legacy migration in this new world. 'Cause there's a whole bunch of tools — like SWE-Bench, have you ever heard of it? Or OpenDevin, or you heard about Copilot Workspaces. A lot of these are getting benchmarked against academia — Python code — which is like, how do you solve an issue in GitHub, how well does it do? None of them are actually using the kind of code you all run in your companies, right? Like — and they'll say the dirty word: Java. (~17:00)

So if I'm asking for help — I found some great open-source tools that are sort of legacy Java code that I'm using to build my own benchmarks. But [if] anybody wants to have a conversation with me — and we can sign NDAs — I really wanna show: can this stuff work with — what is it, Capital One? — Topo Pal had 20,000 Java developers, right? There's a lot of Java code in the banks. (~17:30)

20Workshop and concept hackathon

Anyway — workshop. Oh yes, thank you Gene — that's the first Gene giving you extra time. 20 seconds. Yeah. At 2:45 we're doing a concept hackathon. We did this in New York. It was really cool. You literally get together and it's as if you're gonna build a GenAI product in the enterprise. You're basically just gonna sketch it out, and we're gonna have judges. Gene is a judge, Matt from MongoDB is here — we have a stellar judging panel. And then we will announce the winners at the Ignite Talks. So you get to be a winner of the first GenAI concept hackathon. So there you go. Thank you. (~18:00)