John Willis

So about a year ago or so, I talked to Gene about this idea of what we call Beyond the Phoenix Project. And so we started working on this project. It's an audio series only. So it's a book, but it's audio-only, and we've compiled about 10 hours of this, and it's something we're hoping will be out sometime next year.

But the idea was that, for those of you, most of you should know this, The Phoenix Project was really a purposefully written book based on the model of Dr. Eliyahu Goldratt, a management scientist, and Gene's going to get into some of this, who wrote this book called The Goal. And it had a format to it, and Gene modeled that format and did some deep research into it.

Twenty years later, after Eliyahu Goldratt wrote this book, he did this audio-only series called Beyond the Goal. Gene actually told me about this. I've actually offered people a free dinner if they bought it and didn't like it over the years. That doesn't apply to all you all. I couldn't afford this. But it's an amazing work that Gene and I have just, we both listen to it over and over, and we're big fans of it. And I thought, what if we could do this kind of similar thing for The Phoenix Project, not have to wait 20 years, right?

So anyway, Gene...

Oh, yeah, one last thing, and there's a lot of notes we have. We did this project together. We've been working on it, and we cover a lot of things like Deming and Goldratt, and actually, then we got into resilience engineers, Dekker and Woods and Cook. We got into lean, Peter Senge.

But one of the things I think was fascinating about this was, I knew Gene was smart, but I learned the depth of knowledge he had about Eliyahu Goldratt and how deep he went. And I learned so much in different areas of Gene's knowledge that I wasn't aware of. So that was pretty fascinating.

Gene Kim

And by the way, the same absolutely applies to me in terms of that feeling. A theme that comes up over and over, especially at this conference, is that we're lifetime learners. And I cannot tell you how much I learned working with John Willis, Jez Humble, Patrick Debois, working on The DevOps Handbook. And I feel very much the same about this project in terms of how much John taught me in terms of this whole other surface area of knowledge. And so I can't overstate how much I've learned in the last four years.

And by the way, make no mistake, this project is really John's. "Gene, I really want to do this 10-hour audio called Beyond the Phoenix Project." So thanks so much for making this happen. And we're giving you a sneak peek of it right here.

So John, actually, maybe should we start by talking a little bit about why we love Beyond the Goal so much?

John Willis

Yeah. No, I think that was the thing, the idea of, I wanted to hear your thoughts.

Gene Kim

Yeah.

John Willis

And we did this on the audio series. Gene, why did you love Beyond the Goal so much?

Gene Kim

So The Phoenix Project, you've mentioned my name, but I have to mention my other fellow co-authors. It was George Spafford and Kevin Behr, who I think is at this conference.

And so one of the things that we listened to over and over again was Beyond the Goal. So The Goal is this famous novel that was written in 1984, and it's a novel about a plant manager who has to fix his cost and due date issues in 90 days, otherwise they shut the plant down. And so we read that book over and over again. In fact, I listened to the audiobook of The Goal over 20 times. In fact, we listened to it every time that we would drive out on a vacation together to the beach, and my wife hated it so much that it's been banned from ever being played again.

But Beyond the Goal, I probably also listened to about 20 times during the writing process of The Phoenix Project. I would listen to it over and over again on walks just to try to channel our inner Goldratt.

So what was amazing about Beyond the Goal was that he recorded this eight-hour lecture series 22 years after he wrote The Goal, and it actually framed so much of the thinking that actually went into The Phoenix Project. And so Goldratt is considered one of the three people who really set the stage for the manufacturing revolution in the 1980s. The other two were Dr. Edward Deming, and the third are the group of people who really codified the Toyota Production System that we know as lean. And Dr. Steven Spear, who spoke at last year's conference, is certainly one of them.

And so one of the goals of The Phoenix Project was to create this isomorphic mapping between the plant floor and technology work, right? To show that there's a one-to-one mapping between how manufacturing works and how certain parts of the DevOps value stream work. And so if we can do that, then we could leverage 100 years of the body of knowledge that's been created around lean, theory of constraints, the Toyota Production System, total quality management, so forth.

And so just to show you the extent to which this mapping exists, we studied The Goal for over a decade, getting ready to write that book. We know that The Goal has about 170 pages of problems before they're described, before they even talk about the solution. So The Phoenix Project has about 170 pages of problems. The Goal had five breakthroughs of how they solve certain problems and moving the constraints. So The Phoenix Project has five breakthroughs. The word count of The Goal was about 110,000 words. So The Phoenix Project also has about 110,000 words. So yeah, to say that we closely modeled The Phoenix Project after The Goal is an understatement.

The other thing that I wanted to share, that was what Goldratt is known for. He's known for many things. And by the way, where did we learn all this? It was primarily through Beyond the Goal. And then it was a set of three graduate courses that George Spafford and I took at Washington State University.

So let me just share with you what some of those key learnings are. So can we go turn on the slide deck, please?

So Dr. Goldratt is famous for many things, but I think one of the things that he did better than anybody else is verbalize what he called the core chronic conflict. Are we show-- Okay, perfect.

And so what he did was say there's this tension that exists between silos that happens in almost every part of any complex organization. So in production, the goal is to manage production effectively. But to do that, we need to protect sales. We have to have inventory on the shelves so that customers can buy it. And so to protect sales, we want to increase the amount of inventory.

However, we also have to control costs. And to control costs, we need to reduce the amount of inventory. So two valid business goals result in two diametrically opposed actions. And this definitely causes decades of conflict.

A similar conflict exists in distribution. In order to have effective material control, you need to put inventory where we have the shortest lead times to get it to customers, and that means you put it very close to customers, near the stores. But you also want to keep inventory where you have the most amount of predictability and reliability in the sales forecast, which means you have to keep inventory close to the suppliers, right? Again, another core chronic conflict.

And by the way, Richard Jackson and Rosalind Radcliffe will be talking exactly about how they solved this problem in the 1990s at Walmart, revolutionizing how suppliers, retailers work together to win in the marketplace.

So in technology, one of the things that George Spafford and I did in this Washington State University graduate course was really work on seeing if we can verbalize the core chronic conflict in technology. And I think this will resonate with many of you.

In order to enable the achievement of business goals, we need to respond quickly to urgent business needs, like what the team from American Airlines shared, and that means push out changes quickly. But as any ops person will say, it's not just about delivery speed. We also have to make sure that we have reliable, stable, secure service. And that means make no changes ever. "Over my dead body" has been the last words uttered by many operations executives. Another core chronic conflict.

And so this is a classic dev and ops conflict. And I love that because this tool you can use to, one, show the core chronic conflict, and then you can build these amazing current reality trees to actually describe how every one of these problems can be traced to this core chronic conflict.

And so, as part of the work of researching and getting ready to write The Phoenix Project, we laid out every problem and then put them in a very specific way so that we could just traverse this current reality tree. And that's really the first 170 pages of The Phoenix Project.

John Willis

It's brilliant. If you haven't bought Beyond the Goal, you really should. It's a fascinating work. And then to the point of this project we've been on, it's an hour of Gene getting very deep into this subject. He covered the quick surface here. And so, anyway...

Gene Kim

The other area that just riveted me, and you can turn off the slides now. Oh, you did. Is the notion of shoulders of giants. One of the things that John talks a lot about is, where did Deming come from? What made Deming Deming?

There's a famous biographer named David McCullough, who's famous for his biographies of John Adams, George Washington, Jefferson. And what I learned in this project was that John Willis is probably the David McCullough of technology. In his head is an amazing biography of Deming, just describing how his early years probably informed everything that he did in the rest of his career. So can you tell us a little bit about that?

John Willis

Yeah. So what we tried to do for this thing, we tried to take one subset of a chapter that we did, and one of them was on Deming. Basically, this is some of the stuff we talk about in the Deming section, which is based on this shoulder of giants.

Gene Kim

And you did this one presentation called From Deming--

John Willis

Yeah.

Gene Kim

--to DevOps.

John Willis

Yeah. So putting on my history hat here. So at DevOpsDays in 2012, Ben Rockwood, who now works at Chef, he did this Open Space on theory of constraints, which is another Goldratt thing. And we all went there, and we entered Open Spaces, and we were talking about how awesome Goldratt is and all the cool things. And in the middle of it, Mr. Rockwood said, "It all comes from Deming."

And I was like, "Okay, explain that." And he couldn't explain. He said, "You need to do your own research." And so I spent a lot of time basically trying to answer that question.

So I decided to do a presentation at Pubcon 2012 called Deming to DevOps. I was trying to see if there was this correlation, and was he right? And I kind of instinctively knew I did enough research. So I went ahead and I did this presentation.

And what I had to do, though, is I started going back through history. And in Beyond the Goal, there's a point at which Eli Goldratt says, "Oh, and by the way, I was a physicist before I became a management consultant." And he says, "Oh, and by the way, Dr. Deming was as well."

And to me, that was the linchpin of something I wanted to research, which was, we had two physicists at the turn of the century, 1920, that were really just being immersed with quantum physics and nondeterminism. And then you could see that a lot of what we see downstream, I think, is kind of nondeterministic thinking.

And so in my research, I saw there was this really interesting thread of nondeterminism, starting with Darwin, Origin of Species, and the father of nondeterminism. But what made that interesting is--

Gene Kim

Hold on. Charles Darwin wrote The Origin of Species in 1859, 157 years ago.

John Willis

Yes. But the real hook was, there was a gentleman named Boltzmann, who was considered the father of statistical mechanics, who saw Darwin's work. And he was working with gases and thermodynamics, and he saw that nondeterminism, probability, statistics, and solved some really interesting problems.

Just to follow the thread, Max Planck, anybody who's been an engineer has heard of the Planck constant. Planck solved a Boltzmann problem, and Max Planck is considered the father of quantum physics. You lead into Einstein, and Einstein, basically, Max Planck's constant wins him a Nobel Prize. Now, here's the thing. That's all happening while Deming and Shewhart are both basically studying physics.

And so you watch them come out, and there's another gentleman named Dr. Shewhart, who I just mentioned, and Goldratt too, who actually created these foundational things called PDSA, Plan-Do-Study-Act, and something called common cause and special cause variation.

So it was clear to me, and I just followed that thread. I played those ideas off of my good physics friend, Mark Burgess, for those of you know who he is, and was able to really kind of--

Gene Kim

Inventor of CFEngine.

John Willis

Yeah, the inventor of CFEngine. Really, the inventor of the genre of infrastructure as code, if you will. Chef and Puppet, if you want.

So that was kind of interesting. And then I found this really other interesting story. Gene loves when I bring up these crazy little convergent, crazy stories. That a lot of people think that Dr. Deming met Shewhart at Bell Labs, but he actually met him when he was an intern at Western Electric in this place called the Hawthorne Factory.

Interesting enough, Hawthorne Factory is actually the birthplace of sociology. There were some of the most interesting studies there, not done by Deming or Shewhart, but it was like there's these places in time, Bell Labs was one of them, where you just find all this stuff that creates this history for us. So anyway, we get a lot deeper in the audio series.

Gene Kim

Actually, so John, yeah, you told me something that astonished me. So the famous Deming cycle, right? The Plan-Do-Check-Act, was not actually invented by Deming. He learned it from Shewhart. And you said the words, "Deming actually made Shewhart's work understandable and accessible."

John Willis

Yeah. They said that Deming explained Shewhart better than Shewhart. And I think that was the hook, that Deming always believed everything Shewhart said was--

Gene Kim

And I guess that's just one of the things that I find so interesting is that who actually makes the work accessible to everybody? Even Shewhart was brilliant, but it actually took someone like a Deming to actually make his work broadly applicable and understandable by everybody.

John Willis

Well, I'm going to go off script for a second. Sorry. But attribute to Gene, right? The Phoenix Project is why this room has 1,200 people in it, right? I mean, all honesty, right?

Gene Kim

And fellow co-authors, yeah.

John Willis

I know, but it takes a village, man.

Gene Kim

So John, why talk about Deming? We spent so much time actually talking about this person. Well, why is it important to this community, do you think?

John Willis

So when I did back my research, the first place you wind up is a book called Out of the Crisis. And Out of the Crisis describes something called the Deming 14 Points. Now, you don't have to go read Out of the Crisis if you want, but if you don't believe me about Deming being-- Like everything you're going to hear over the next three days, go Google Deming's 14 Points, and you'll hear those points just in almost every presentation that deals with culture or some type of system thinking.

And Deming really became the father of lean in Japan, which became lean. But then another book he did later before he died, which is called The New Economics, where he defined this thing called the system of profound knowledge. And this is the thing I told Gene is, I think when he did this book, it was pretty late. I think he died the year he published the book or very short after, and I don't think most people picked up on this idea of what he's done. It really is profound.

So the profound knowledge is based on an idea that complexity, in Deming's mind, requires four lenses to look at complexity. One, basically, theory of knowledge, which is really just PDCA, scientific method. Another one is theory of systems, which is systems thinking. A lot of what Goldratt does in global optimization versus local optimization, systems thinking, right? Those are two.

The third one, which was the theory of variation, and there's a deep body there of what's the difference between special cause and common cause variation, and understanding variation and statistical process control. I would say if you find a framework that has those three things in it, you've done really well, right? That's pretty cool. Just those three things alone, variation, systems thinking, and scientific method, which 90% of the world doesn't do.

The thing that made Deming brilliant, in my opinion, is the fourth lens, and it was called the theory of psychology. And what he said that is you-- Basically, what he was saying is you needed to understand the cognitive bias of the people involved with the complexity that you were dealing with. We talk about it today like it's the blameless postmortems. But he was talking about this 60 years ago, that you could have all the variation, all the statistics, all the knowledge about the system, the global optimization, all that, but if you were trying to change somebody's mind and their cognitive bias and you didn't understand that was the lens, or you didn't put that lens on to understand how to deal with that complexity, you were going to go nowhere.

And I was going to tell the doctor story, right?

Gene Kim

Yeah. No, tell it.

John Willis

So there's a great story about a woman who was a doctor. She was put in charge of a hospital, and this hospital happened to be doing a bunch of Deming stuff. So they sent her to a course when Deming was alive, it was called Four Days with Dr. Deming. It was a famous course. There's a book on it.

And she was like, "I'm a doctor. Why do I have to go to this stupid course on management?" So they send her to. The end of the first day, they're all having dinner. She's about five seats down from Dr. Deming. He's, of course, at the front of the table. And she's telling this story to the student next to her about she did some work in South Africa, where there was a bunch of young doctors would continually argue with the tribal elders.

And the argument would be that the tribal elders would say, "This is a curse. We need to deal with this curse and remove the curse." And the doctors would argue and say, "No, no, you're out of your mind. We need to give him medicine now."

So she comes in and she says, she grabs all the doctors and she says, "That's not the way you want to do this. You're not going to change their mind. What you're going to tell him is you totally agree with him that it's a curse. In fact, that guy has the worst curse over there. He looks really bad. And you're going to say, 'But can we give him medicine just in case, or as an alternative?'"

Deming overheard that story, immediately replaced the person sitting next to him, and moved her next to him. And then she spent the last year of his life traveling with him. And the reason why is he knew she understood and had a classic example of the theory of psychology.

Gene Kim

That's awesome. In fact, one of the things I loved about this project was that it makes Deming accessible. And just to show you how accessibility can hurt us, George Spafford, Kevin Behr, and I, about in 2001, 2002, we had read The Goal. We knew we wanted to learn as much as we can about the theory of constraints.

And there was some hints that there was something called becoming a certified Jonah. So Jonah is the Yoda character of which Erik in The Phoenix Project is modeled after. And the whole notion that you could get trained to speak in mythical ways that are difficult to understand, that's what we wanted.

I think there was some training that was offered, but you had to spend four weeks on-site. And finally, when we found this course at Washington State University, which actually was through distance learning, but it was just awesome.

And I think for people to take works like The Theory of Constraints, make that accessible through Beyond the Goal. One of the things that I think John does so well is make Deming accessible for everybody.

John Willis

It's my Twitter icon.

Gene Kim

Right. So actually, there's some things that-- or maybe we should talk a little bit about what Deming is most famous for. Tell us about what everybody probably should know about Deming.

John Willis

Yeah, I think, not everybody, but most people, or if you read about Deming, most of the books that you'll find will talk about after World War II, he was sent over to Japan to help them rebuild. And he used some of his management science and his ideas about quality and really system of profound knowledge, even though he didn't call it that back then.

And really, I say, and most people would probably agree, that he became the Shakespeare of quality and systems thinking in Japan and was probably the single most influencer of what became Toyota Production System, which in the U.S. coins as lean. That's where my theory is that there is deep correlation between what Deming did and what we do today, what we call DevOps. But that's the story most people know.

The story that most people don't know is Deming has this amazing history. Again, we go into a lot of it in the audio series. But one of the ones I think is very interesting people don't know is he had some foot problem or something, and he didn't fight in the war.

Gene Kim

Oh, and this is World War II.

John Willis

Well, I'm sorry. Yeah. World War II. But what happened was, I forget the details, we have it in the series, but he gave a course at Stanford on management training.

So the problem then is all the kind of men in the country are going off. The factories have just kind of older people and women, and now you've got to change the game, right? Throwing in Andrew Shafer, a Pareto inefficient Nash equilibrium. Look it up. The game has to change.

So what happened was they wanted Deming to use his ideas, and his ideas were very nondeterministic. He taught a class for two years, basically training factory managers. And some people credit that we possibly won the war because the quality of our tanks and our jeeps and all the things had to do with Deming's influence and training during that short period where factories had to be basically completely changed in how they delivered at speed and quality.

Gene Kim

So what I love about that is that he's probably most famous for his work in the Japanese economic recovery after World War II, but he had this incredible pivotal role in terms of creating the greatest increase in wartime production that the world has ever seen. So it's just another thing amazing that I learned.

John Willis

I thought we'd be out of time, but I want to tell another story. There's a famous story where Dr. Deming in 1950, it's called the Mount Hakone Summit, and they say that 80% of the wealth of Japan is in this room, the representative wealth is in this room. And he tells the audience that if they follow his principles, they will be a top-consuming nation in five years. And almost to the day, he's spot on.

Funny story. He's sitting as an octogenarian, and nobody in the U.S. even knows who he is. And NBC does this thing called If Japan Can, Why Can't We?

Gene Kim

What year was that? 19--

John Willis

It was like '83 or '84. And we're getting clobbered. Anybody saw Steven Spear's presentation last year about what Toyota did to us from 1975 to basically 2010, they decimated the U.S. car market.

So in the last seven minutes of this thing is a story about some guy that lives in Washington, D.C., that basically had these ideas that transformed Japan. And the CEO of Ford Motors watches this, calls him in to Ford Motors and says, "Dude, I need you to come here and do this." And you know what Deming tells him? "If you follow my instructions, in five years, you will overtake General Motors."

And basically, it was about three years, and I don't remember the exact years, but that was the first time Ford, since like the '30s, had actually had some metrics of market share than General Motors. I mean, the man is-- is the dude.

Gene Kim

So here's the other thing that I found very surprising is that he actually published books very late in his life. So, the famous book, Out of the Crisis, was written in 1986. He wrote The System of Profound Knowledge in 1992, and then he died probably within a year of that.

John Willis

Yeah.

Gene Kim

So you made this observation that it was actually probably him publishing so late in life that actually is maybe why we don't know this. Say more about that.

John Willis

Yeah, no. And what's interesting, too, The System of Profound Knowledge is actually in the book called The New Economics. But in Out of the Crisis, which is interesting, the first time somebody told it to me, I thought, "Oh, he's going to talk about how to get a corporation out of crisis." The truth was, he was so fed up with America not listening to what he said.

And he did this thing, and when everybody came back from war, what'd they do? They went back to the old Tayloristic Sloan model. "Hey, okay, that was good during the war. Get out of here." And his Out of the Crisis was, he was talking about what he thought was an institutional crisis in America, in education, in government. And he was writing a book to say, "These properties and these frameworks could work for education."

And really the intent, at that point, he was kind of done with the world in terms of like, I'm going to fix your quality or help you mass produce technology or produce parts. And really the intention in that book was really more about trying to say, "Hey, these are great ideas that work universally, and I think they would work for education and government."

Gene Kim

And again, you say everyone should know about Deming because we should know where we came from. Can you say something about that?

John Willis

Yeah, again, I think if you go back, if you start doing the research on Deming, you find that it's absolutely clear that he had a direct influence on the scientists that we know, like Taiichi Ohno and Shigeo Shingo, and all those people that did the Toyota Production System. And then we see the lean from guys, Mike Rother, Toyota Kata, Steven Spear, who was at it last year.

So we can see the threads of that. And then, again, I think I've made a strong argument in many places that DevOps, Deming could be considered the Shakespeare of DevOps ultimately. But again, I think that it's the nondeterministic thread that really excites me, is that a good friend, Mark Burgess, has a book called In Search of Certainty. The head fake is certainty doesn't exist. It's about uncertainty and nondeterministic thinking.

And at scale, as we grow, and as we grow with people, grow with technology, we have to give way to nondeterministic models.

Gene Kim

Awesome.

John Willis

Bang.

Gene Kim

And he called it 60 years ago.

John Willis

Yes, sir. Anyway.

Gene Kim

Awesome. So I had so much fun working with this. I just wanted to share--

John Willis

Yeah.

Gene Kim

--some of these learnings, and there'll be more coming soon.

John Willis

We're going deep on Peter Senge. We're going deep on Kata and Rother. And we're going deep on the resilience and human factors, Dr. Woods, Sidney Dekker. So each level, we're going to get deep on everyone. It's just been a blast.

Gene Kim

Very good. Hey, thank you so much.

John Willis

Gene, to you, it's always a pleasure.