Gene Kim

Good morning. How was everyone's day yesterday? Awesome. Great, because we have another phenomenal day of programming for you today.

This morning I'm going to share with you two things. I'm going to talk a little bit more about the work of Dr. Carlotta Perez, talk about the five ideals, and frame so many of the stories that we've heard over the last two days within them.

But before that, I wanted to share something that Michael Winslow from Comcast shared with me a couple of weeks ago, but that he didn't share yesterday, but I want to tell you about because I do think it's important. So he shared the story of the Technically Speaking program that he started. And a few interesting things have happened to Michael, since he started that program one year ago.

His positive outlook and bias for action has made him visible at the highest levels of Comcast, including the CTO, the chief network officer, and many others. He's been able to obtain major funding and support for several initiatives, including the internal Comcast DevOps Days event and around the country, and he's being recognized nationally for the pioneering work that he's doing.

So it's great to see that not only is the work that he's doing benefiting society and Comcast, but it's also had great benefits for him as well. So I think it's so great that you can actually do both, and the reason why I wanted to share that with you is that there's so many amazing outcomes that happen when you drive important things. You get appreciated by important people. So that is why I wanted to share that. So

let's talk about Dr. Carlotta Perez. So yesterday I had shared Dr. Perez's work. Her novel contribution was showing how that any time a major mode of production, when the cost is rapidly diminished, certain things happen. And that's happened five times. The age of the Industrial Revolution, age of steam and railways, age of steel and heavy engineering, age of mass production, and I see Mik has cavalierly renamed her last one Age of Software and Digital. And we talked yesterday about how each one of them have also created a mode of management that transformed how large organizations work. And I had asserted that this empty box I genuinely believe is going to be created, called dynamic learning organization. And by the way, after the workplace engagement panel yesterday, I am at moral certainty that all the things that Dr. Andre Martin talked about on day one, what Dr. David Almeida from Kronos talked about on day two. They're all saying the same things, right? It is innovation happening at the edges, fully supported by core. It's clear to me that that is a common frame that embodies DevOps, the Toyota Production System, and everything that goes into dynamic learning organizations.

But there's another part of Dr. Perez's work that is incredibly interesting to me, and that is there's really two parts of these cycles. The first one is driven by financial capital. So when this new innovation happens, there's a boom driven by financial capital. So VCs, Wall Street. Mik noted that in Detroit, 100 years ago, there were 300 car startup companies. So after that huge boom, and often sometimes speculation, there's a huge crash, and then there's an intense period of re-regulation. And then there is, if all goes well,

a 50-year period that's often an economic golden age, that is driven not by financial capital, but is driven by production capital. And so in the last age, the real societal value and economic wealth was not created by the car manufacturers. It was the car manufacturers in combination with the interstate highway system and the re-engineering supply chains. That is what sets the stage for the 50-year economic expansion, the largest expansion that we've seen in modern civilization. And so the stories being told here are that type of story. These stories are being funded not by financial capital, but through production capital. And so the claim I'll make is that as much value that has been created by the Facebook, Amazon, Netflix, Microsoft, and Googles, that will be eclipsed by the economic value that will be created by this community. When every large brand adopts all of the methods pioneered by the tech giants, that will generate trillions of dollars of economic value per year. So hopefully that will set the stage. And when that happens, that's what actually corrects the wealth redistribution. That's when we see incredible gains and a more just society. So that's why I think this work is so important. So

the other thing I want to talk about is a little bit more detail about the five ideals. I had mentioned that the construct used in The Unicorn Project to embody the success stories and the heroic journey to the DevOps enterprise community are framed through these five ideals. The first ideal is about locality and simplicity. The second one is about focus, flow, and joy. The third is about improvement of daily work. Fourth is psychological safety, and the fifth is customer focus. So what I'd like to do in the next 15 minutes is just share with you what some of those elements of each of the five ideals are and point to specific examples of them that we've heard over the last two days.

So locality and simplicity. And by the way, I was very interested that Joe Aho, CFO of Compuware, actually mentioned how simple is not easy, right? Simple actually takes work. And so

in a measure of this, so in The Phoenix Project, the bus factor was quite prominent. How many people need to be hit by a bus for the service or project or company to be in grave jeopardy? At Parts Unlimited in The Phoenix Project, the bus factor was one. Brent. If something happened to Brent, no one could get work done, incidents couldn't get resolved, no major initiative could be completed. So the corresponding metric in The Unicorn Project

is the lunch factor. In other words, to get any kind of major work done, how many people do you need to take out to lunch to be able to convince them to help you? And we need to do this because everything's so tightly coupled together that everything requires 20, 30, 40 different teams. So the lunch factor is really how many people do we need to feed in order to get our things done? Is it two pizzas like the Amazon idealized two-pizza team, or do we need to feed everybody in the building? And in some cases, for major initiatives, you might have to schedule lunch with 43 different people, which will take months. Just to even get them to know who you are and what you want and why they should even care. So the ideal is that number should be small, two pizzas. In the not ideal, it might be two truckloads of pizzas.

Locality happens also in code. In the ideal, anyone to implement a neat feature can do it just by looking and changing one file, one module, one class. One container, whatever. In the not ideal, in order to make a change, you need to understand and change all the files. That could be millions of lines of code. It could, as Scott Prugh's story of, in order to get major features done, they would have to traverse across 10 different teams or more. Of which one has a workforce that is all retiring and maybe soon disappearing. So the ideal is enable teams to work independently, to be able to independently develop, test, and deploy value to customers.

Oh, another one. That we can actually, changes can be made and tested, isolated from other components. Not ideal. Our systems are coupled in such a way that in order to test the systems, we actually need the entire system. So the story we heard from Walmart yesterday from Scott Havens is that when in the bad old days, you would actually have to test in an integrated test environment to even see if our changes worked. By moving to event streaming, they were able to decouple those pieces from each other.

Another one. In the organization, locality. Ideal is every team has the expertise, capability, and authority to be able to satisfy customer needs. In the not ideal, everything needs to be escalated up two levels, over two, and down two. So visually, to depict this, this is called a square. Up, over two. Up, over two. And as a friend of mine once said, that's actually the ideal case for us. In the worst case, you actually have to do the reverse path in order for two engineers to actually work together. How am I doing? Is this interesting? Funny? All right, good.

So first ideal was about locality and simplicity. The second one is focus, flow, and joy.

So the ideal, all energy and time is being focused on solving the business problem, and you're having fun. Not ideal is all your time is spent solving problems you don't even want to solve. Googling, Stack Overflowing, writing YAML files, Makefiles, trying to figure out how to escape spaces and filenames inside of Makefiles. And for me, one of the side effects of learning Clojure and functional programming is that I've learned how much fun I have just working on my feature. The thing I want to solve. And here's all the things that I know are important, but I now detest. I've become one of the fussiest developers ever. I don't want to deal with anything outside of my application. I don't want to connect to databases. I don't want to connect--because it always takes me a week. I don't want to connect anything. I don't want to update dependencies, because everything potentially breaks. Secrets management. I am the idiot who puts stuff into the repos. All the keys, all the time. Patching. I don't know why my Kubernetes costs are so high.

So in my ideal, I'm just working on the feature I want, and I have friends in the infrastructure organization that can take care of all this for me. So in so many of the stories we've heard, like the team from Adidas, they are putting all the operations and security expertise into the platform so that any developer, just by virtue of using the platform, inherits the best known understanding of how to be secure, how to be operable, and how to scale, and they can focus very parochially on the application. Which I think is actually a good thing. And so that is why even though I've become a very fussy developer, I do make this claim that there's never been a better time to be in infrastructure and operations. The best time is not behind us. It is instead ahead of us.

Part of the second ideal was actually the reasoning, maybe the rationale for it, was made evident by Dr. Mik Kersten. He introduced me to the work of this notion that there's really two types of learning. The first is called procedural learning or declarative learning. So this is the type of things when we learn, we appreciate. Often, it's we're building upon decades of knowledge, and everything that we learn, we value because we know that will serve us for decades to come.

And then there's this other type of learning called one-shot learning. And so this is the things like Googling, how do I escape spaces in Makefiles? I don't really care. In fact, I have to say, when someone actually told me, someone texted me, "Oh, that's a double dollar sign in Make." I actually got angry because I knew that 30 years ago, but now I literally do not care. I resent the fact that I spent probably an hour Googling for things. So I actually save screenshots of certain things that actually make me angry because I resent the fact that... Yeah, this is actually a good one. The bad one was the double dollar sign Makefile. Anyway, there are certain things I just do not care about anymore, because it's a problem that I need to solve, but is one that has no long-term value for me.

Third, improvement of daily work. And so this, of course, was brought up in The Phoenix Project, and this is really based on the work of Dr. Steven Spear, where this is a statement that improvement of daily work is even more important than daily work itself. And

the best example for me of the not ideal was the General Motors Fremont plant. So this is the famous site of the amazing NUMMI joint venture in Fremont, California. But for decades, this was by far the worst performing automotive plant in North America. There are well-documented cases where because there were no effective procedures to detect problems during the assembly process, nor were there explicit procedures on what to do when problems were found, engines were put in backwards, cars were missing steering wheels or tires, cars had to be towed off the assembly line because they wouldn't start.

So that is not ideal. Ideal is the notion that we put as much feedback into our system sooner, faster, cheaper, with as much clarity between cause and effect. And the reason we do that is so that we can invalidate assumptions. Every assumption that we can invalidate is a learning opportunity, and we then spread them throughout the organization. And again, the notion of platforms I think is so important because it allows us to be able to share learnings. Local discoveries can then get broadcast globally and then inherited by every person who uses that platform, elevating the state of productivity across the entire organization. We all know about the Andon cord, and we all know how many times in a typical plant the Andon cord is pulled. Of course, in a typical Toyota plant, the Andon cord is pulled 3,500 times a day. And so we know that in this community, but what is probably not as well known, and I think what this community has difficulty convincing others, is that greatness isn't free. We need to pay down technical debt as a part of daily work.

And so I'm going to share the tool that Mik Kersten and I wrote to basically tell the story. And there should be an attribution there. We lost it in the copy and paste. Sorry, Mik. And so, again, the background of this is many years ago, someone very wise told me, "When dealing with executives, stick with small numbers and primary colors." And what I found in my journey is that when dealing with very senior people, that is not sufficient. You must stick with something even simpler, which is only up and down arrows.

So the story of how technical debt gets created is this. We've all been in periods where we need to get to market. So we push, push, push, to get features into markets so that customers will value it. Maybe it's to get first to market or maybe just to even enter the market because we're last. And when we cut corners, and this is what it takes, this is what drives up debts and risks. This is what drives down quality, and that is what drives up defects. And so, we all know that, but the story continues. When that happens, our ability to actually ship features goes down, and the amount of time spent fixing defects goes up to the point where we can be spending all our time, over 100%, just fixing defects. And so when that happens, this is when reliability tanks, customers leave, morale plunges, and developers leave because everything is so hard. And so John Cutler tweeted this at us, and I'm so happy that John Cutler is here today and presenting today.

He sent this to me, and I love it. He said in 2015, a certain reference feature would take 15 to 30 days. Three years later, because of technical debt, it takes 10 times longer. And so it really validated the notion that this happens. It's like dark matter. We finally know that it exists.

I shared with you earlier this week the story of Risto Siilasmaa, the chairman of Nokia, and for him to be able to identify the problem that Nokia had, that there was no way they could survive if they were to keep using the Symbian operating system. If the build times took 48 days, no developer could tell whether what they wrote worked or would have to be redone in two days. And that is what drove them to any other strategy, in this case, to Windows Mobile, which wasn't so great for them, but that was better than staying on Symbian OS.

What I didn't share was that every tech giant has gone through this. I had mentioned the Microsoft Security Stand Down, that basically every one of these tech giants have gone through a feature freeze to basically say if you had to choose between a feature or paying down technical debt. In fact, no, they said, "No, we're not going to work on features. We're going to pay down technical debt." And so here's that story using up and down arrows. We take features down to zero. This allows us to pay down technical debt, which allows us to increase quality, which allows us to take down defects. Maybe not to zero, but at least to something that's tenable, and that's what allows us to get back to a position where we can deliver features again. So, that is the choice that the tech giants have made. What is interesting--

So ideal. The tech giants spend between 3% to 5% of all developers dedicated to improving developer productivity. It has been documented that Google spends 1,500 developers just working on dev productivity. That's a billion-dollar spend per year. Microsoft has likely over 3,000 developers. Not ideal? In many of our organizations, we give it to the summer interns or people not good enough to be developers. Who's going to work on the CI systems? That's where we're going to put the developers who aren't good enough to work on features.

So I think in a really neat karmic continuation of the Microsoft journey, this quote came from Satya Nadella, CEO of Microsoft. He said, "If a developer has to choose between working on a feature or dev productivity, work on dev productivity." Because that will pay off way into the future. This is technical debt in the reverse. This is compounding interest working in our favor.

Last thing on technical innovation. There was a graph that Jeff Gallimore showed me when I was visiting him in DC, and it was this kind of Andon cord experiment where a team said, "We're going to put an Andon cord in development and see what happens." If a developer needs help or anyone on the team needs help, they type Andon in the Slack channel, and everyone will swarm the problem. And of course, this is preposterous. There's no way an Andon cord will work for knowledge work. What he showed was that in red, as the Andon cord pulls increase, flow time or lead time decreases. In other words, the more they pull the cord, the faster they get features into the hands of customers. And I thought this was a marvelous story. And if you're interested in this, this team will be presenting later today. I think astonishing and just a genuinely novel contribution to the industry.

Ideal number four, psychological safety. Of course, we all know the Westrum model. Dr. Flores talked about that yesterday. And I was so delighted that in the latest State of DevOps Report, they were actually able to integrate the work of Project Aristotle, this incredible program done years ago to actually understand what made great teams work. This was actually, again, referenced by Dr. David Almeida from Kronos, as a benchmark that they were using. And again, always showing up at the top as driving performance was psychological safety. To what extent do teams feel comfortable saying what they think without the risk of feeling embarrassed, or castigated, or made to feel bad? So that again showed up in so many of the presentations that you heard today.

And then ideal number five is relentless focus on the customer. That really should be customer focus. And again, I think one of the best examples was the empty data center at Compuware. Is that really being mercenary and deliberate about what creates durable, lasting advantage, that's core, and what is context? And what we need to do to make sure that context doesn't starve core. And Dr. Geoffrey Moore actually said, "The killing ground of companies is when context starves core." So,

that's the five ideals in a nutshell.

And with that, I'm going to bring out Jeff Gallimore to help us open the day.

Jeff Gallimore

The sun and the moon. All the best things come in twos. What would I do without a friend like you?

Who had a great time at the lightning talks last night? Yeah. What a great time. So much respect for those presenters. That is such a hard format to deliver and land well. And thanks to Sonatype last night for sponsoring that.

How's everybody feeling today?

Okay. All right. Well, let me tell you what I think I just heard in that response. Some of you are feeling super pumped about all the things that you've learned in the last two days here at the summit. Yeah? Yeah. Some of you are thinking, "I finally found my group of fellow travelers, like-minded individuals." You're feeling very connected. Some of you are very thoughtful about all of the presenters and the talks and the things that you've learned and wondering how those might apply back in your day jobs. Some of you might be realizing you've got a lot of work ahead of you when you do go back to your day jobs. Some of you have gotten some resolve and some hope because you've seen the success stories of the experience reports and the people who have been talking about the accomplishments that they've had applying these DevOps patterns, practices, and principles. And some of you might be just totally gassed at this point. Or all of the above. You might be feeling all of these all at the same time.

Well, we're going to try and fill the tank back up again. We've got a great day of programming ahead of us today.

We want to hear the stories. So before you leave Vegas, we'd love to hear the stories, the things you've learned, the people that you've met, and the ideas and the actions that you're going to try out back at your job. We've got a channel in the Slack instance. It's called Summit Stories. Please post that before you leave.

And speaking of sharing and stories and feedback, please get your session evaluations in. Sharing is caring. Want to thank our founding sponsor, IT Revolution. Our Platinum Plus sponsors, our Platinum Classic sponsors, our Gold sponsors, our Silver sponsors. Go talk to them in the expo hall. This is the last day to be able to do that and see how they can help us on our DevOps journeys.

And because it's the last day, it's also the last day to play the unicorn game and get those 23 unicorn selfies, which are keepsakes in and of themselves, I think. And then also the last day for the sponsor passport. In fact, you need to have that completed today by 1:10 PM and dropped off in the IT Revolution booth. Make sure that you write your name on it. You don't want to be that person whose card is drawn without a name, so we can't find the winner.

Remember, sponsors add sparkle to our journeys.

It's the final day. Buckle up. Back to you, Gene.

Gene Kim

Thank you, Jeff.