Gene Kim

Good afternoon, everybody. Why don't we bring our panel out? Dr. Cook, Dr. Dekker, and Dr. Spear.

Now, while you're getting seated, let me share with you what we did today. We spent four hours together this afternoon to really explore safety culture, where they intersect, where there might be disagreements. I have to tell you, it was one of the most professionally rewarding things I've done in a very long time.

It was a magical, and if I may be honest, very stressful experience. And to be able to do this with John Willis as a co-pilot was just fantastic, and I learned so much. I think it lives up to the notion that I had said once, which was that I think it will be historic. And I certainly believe it will be important.

One of the things that I found so meaningful was that they actually gave me a lot of concrete advice in terms of how we can accelerate the DevOps movement, how we can take advantage of this opportunity that we have, and how we can increase the likelihood of this movement succeeding. And so we will make the video of it available to share with you and a much broader audience sometime in the future.

So I have prepared three questions, and hopefully you'll get a taste of why this was so rewarding to me.

Let me start with first Sidney Dekker, Richard Cook. Both of you have had time to spend in the DevOps community. What are the key places where safety practices can help us? Sidney, Richard?

For the last four hours, we couldn't keep them from talking. And now they're not saying anything.

Sidney Dekker

It's called stage fright. All these people. Oh my God.

I think a qualification is in place. Perhaps there are none. And I think Richard might speak to that. But if there would be, we've learned through lots of damage and dead bodies that a safety culture is a culture that allows the boss to hear bad news.

And so there's two problems with that. One is to get news to the boss to begin with, and for the boss to be open to that news, and for the boss herself or himself to be able to share bad news.

But then the question is: what is bad news, right? What constitutes interesting feedback that you really think you need to be concerned about? But it's that discussion that I would like to impart.

Other than that, and this is a discussion that we've certainly had, Gene, is the issue: don't see people as the problem to control. Don't think that your people need controlling through procedures and more training and more guidance. No, they are your problem solvers. They know what to do. Don't tell them what to do. They know what to do. Ask them what they need to be better at it.

And then don't sweat counting negatives. Understand how success is created.

Gene Kim

And you said something that I found very significant, which is it involves leaders and people on the front line. That there's a symmetry.

Sidney Dekker

That's good. Yeah, particularly in some other cultures it's not as problematic as in North America, actually. But it's easy to think that we just always need to legitimate the voice from below who doesn't have status, and, "Oh, make them speak and make them speak up and make them feel safe," which is incredibly difficult to do.

And by the way, I think none of us sits on a good example of an organization that does this perfectly at all, by the way.

But it is also the leader. She or he needs to be able to say, "I don't know. I need help. I actually am concerned about this. I need you people to help me do this and figure this out." And sometimes it's quite difficult for a leader to be seen to be losing face, to not know the answer.

And so that culture of psychological safety, if that's the term that you want to use, goes both ways.

Gene Kim

Awesome. Richard.

Richard Cook

I think one of the problems that we face, particularly in high-velocity production environments, is that we are better at forgetting than we are at learning. Or we're pretty good at learning, but we also forget very fast.

We have a forgetting problem that's probably more significant than our learning problem in many settings. And our forgetting is so quick because we're moving on to other things so fast that we can essentially lose sight of the lessons that we've learned.

And I think we need to really work on trying to sustain the lessons learned and the experience over time. And that means not simply dispensing with the problem and considering that it's done.

John Allspaw talked to this a little bit yesterday when he talked about the fact that people and their PMs were saying, "We all know what happened. Let's just fix it and move on." But I think that we really need to be able to remember these events and, if you understand a Heinlein phrase, grok them in fullness over time, because I think that's really crucial.

Gene Kim

Actually, before I ask you the question for the Lean community, you actually said something that was very memorable to me about the role of the leader, right? Something about stunning humility of leaders. Could you repeat that just so that everyone could benefit from that observation? The role.

Richard Cook

Have we said profound things? I don't recall this at all.

Gene Kim

Well, it's on video, so you can dig it out somewhere. Do you remember, Steve?

Steve Spear

Yeah.

Gene Kim

Thank you.

Steve Spear

So it picks up what Sidney was saying, which is the adversary for all human activity is ignorance. I mean, if we had perfect knowledge, nothing would ever go wrong. The reason things go wrong is because we didn't know what really we should've been doing or really how we should've been doing it.

So if we view ignorance as the adversary for all our undertakings, our well-intentioned undertakings, then what we have to do is make sure that we behave in such a way that we recognize we're ignorant sooner than later.

Gene Kim

Which requires humility.

Steve Spear

Right, which requires humility to say, "I'm in a situation, and there are probably some fairly important, profound things where I'm actually quite moronic about these things."

And to Sidney's point, I think he's right on. It's become a very convenient cliche to say, "We'll create an environment of psychological or emotional safety. We'll give voice to the person on the deck plate, the shop floor, the bedside. We'll empower that person." But then the senior leader still acts like a pompous jackass.

And so really, how well are they coaching and modeling exactly the behavior they're espousing?

Gene Kim

Good. Any comments on that? Otherwise, I'll ask Steve the second question.

So Steve, the corresponding question to you would be, what would you consider the important Lean practices that are important for DevOps? Is Lean good for us, and what are the right things to really learn from the whole Lean movement?

Steve Spear

Yeah. So we had a longer conversation about the source of the Lean movement, and it's important to understand what the headwaters were. That was Toyota.

And what was the start point for Toyota? It was incompetency in manufacturing cars that would be competitive in the world market. And this Toyota Production System orbited around the notion that what we do, we're really, really poor at. And if anyone catches on that we're really poor, we'll quickly become irrelevant.

And so you had Toyota leadership, from the very beginning, managing their systems with this idea that what we have to do is elevate and reveal what's wrong so we can come to a better understanding of what we're doing and why we're doing it, and how we're doing it.

So anyway, what happens with Lean is, and this is a longer story, but what happens with Lean is that Toyota's approach towards managing to identify ignorance and convert it into useful knowledge becomes a set of tools by which you can have stability over a system which would otherwise be chaotic. But of course, all stability is just temporary because the things you're doing are constantly changing in an environment which is constantly changing.

So a direct answer to your question of what's the most important thing we can extract from, and I'll say Toyota, not necessarily Lean, because that term has so much meaning. The most important principle is that of the Andon Cord.

If you think about what the Andon Cord represents, it's the coupling of standard work, which is the agreement you and I have that this is, in the moment, our best-known approach, not the best approach, but our best-known approach for accomplishing something.

But it also is coupled with this humility, this recognition that our best-known approach will be inadequate. We don't know when, otherwise we'd have a better-known approach, but it will be inadequate. And because it will be inadequate, we have to have a mechanism to calling out its inadequacy the moment that first inadequacy is recognized. So the Andon Cord.

Gene Kim

And you feel like that's an embodiment.

Actually, just to think out loud for a second, my first reaction is like, uh-oh, this is uncharted territory. We didn't actually quite get here. We thought this might be a trigger word in the safety community because Andon Cord means work stoppage and cease all operations, which could be considered an anti-pattern.

Steve Spear

Well, can I just elaborate on that?

I think that the literature which attempted to explain Toyota, or at least describe Toyota, said, "Oh, the Andon Cord, if someone has a problem, they pull this cord, and everything comes to a screeching halt." When in fact, that's logically undesirable and also not what happens.

What the Andon Cord does is it declares that there's a situation occurring which was neither expected nor desirable, and it has to be swarmed and investigated in the moment. Now, that doesn't mean that all these other situations which are actually working stop. That would actually not make sense. It's just that that one right there deserves attention.

And just a quick reference to the medical community. If you think about all the instrumentation that goes on a patient, particularly one who's in critical condition, the reason that's there is to indicate, "Oh, there's something going on in the moment which requires attention." And you don't shut down the hospital just because this person has an irregular heartbeat. You deal with that one and let everything else continue till you discover an adjacent abnormality.

Gene Kim

Sidney, Richard, can you react to that?

Sidney Dekker

Yeah. One of the things that I think has made other industries, and I don't want to be imperialist in any way about that because no one has got this figured out perfectly, but is to then tell stories about that irregular heartbeat, right? How could we not have foreseen that from happening?

And so if you're willing and honest and open to tell those stories and share those stories, not only, and this is certainly something that we talked about and I think is critical. Not only is it people love to tell stories, and we do this naturally, right? We're geared to remembering stories and telling stories.

We don't inhabit a universe of concepts and numbers. Well, he might because he's MIT.

Richard Cook

MIT.

Sidney Dekker

How many times did MIT come up in the four hours? I don't know, like 600.

But that makes him the deviant. But other people inhabit a world of stories, and this is what we remember, right?

And one of the things that Steve threw out there as a challenge was, how do we keep sensitizing the DevOps community to the risk of getting it wrong? Because the stuff you guys are playing with actually matters.

It doesn't seem to matter when you're sipping your Coke and you're going, "Oh, 23 hours, I'm still good. Let me bang some other code," if that's what you call it. Probably not.

But there are consequences to getting it wrong. Now, you cannot generically sensitize people to the consequences of getting it wrong, as in a little poster on the wall, "Oh, don't forget that this is risky stuff," or whatever. That doesn't work, right?

However, telling stories about how things went wrong or almost went wrong, that is remembered. We were having dinner last night. Richard was present as well. And so he and I have got plenty of war stories to share, right? Because something is either good or a good story.

But the war stories that were told came from the DevOps community. And it does something else. It not only builds this memory trace of, "Oh, hang on, I've seen this." This recognition primes action, where you go, "Oh, I need to do this because the other guy had this in that story. I remember that."

There is more than that. Out of it arises this ethic of what it means to do right, to do wrong, of what it means to be a good professional, to be a good practitioner. And it's that trace that doesn't exist yet, as Richard has beautifully identified, because you are such a young profession. Right?

Gene Kim

In fact, Richard, I had the opportunity to ask each one of these scholars for advice. What advice would he give us as a community to help us achieve our goals?

And one of them, you can either say Sidney covered it just fine or do you want to expound upon that advice? Because I found it very meaningful and I realized that I have actually made a mistake in terms of kind of how we--

Richard Cook

You'll learn.

Gene Kim

I'll share with you. Do you want to add on to that?

Richard Cook

I think the work that you do has lots of circumstances where you can't simply say, "I'm just going to close up the laptop and go home."

Gene Kim

Yeah.

Richard Cook

You have to solve the problem. You're confronting it. The situation is getting worse. You're in a situation of escalating consequences. You have undesirable options only.

Gene Kim

Huh.

Richard Cook

All of the choices--

Gene Kim

They're all bad.

Richard Cook

Are bad ones, and you're choosing between bad options.

There's an old surgical saying: "If there are more than two operations for any condition, none of them are any good." And I'm afraid that you very often are in that situation.

We respect the fact that you have to make decisions under those circumstances, and we would like you to be as empowered and knowledgeable and supported in those circumstances as possible so that you can make those decisions.

We understand that those decisions may not always turn out well, but you're the one very often who has that burden and that responsibility. And like the pilot in the aircraft or the surgeon in the operating room, we expect, whatever the long-term value of learning from the experience might be, we expect you to act at that moment in the most judicious, thoughtful, and prudent way that you can.

When we look at you, when we watch you and what you do, especially those of you who've got a few gray hairs or who have gotten a little bit of experience, we notice that you're very careful about trying to help other people who don't have that level of experience learn how to approach these problems thoughtfully and taking into account the ways that things can turn out.

And we think that's probably pretty much crucial to what DevOps is. DevOps is not simply the practice of fixing problems or generating velocity. DevOps is also the practice of building a community of people who do DevOps.

You are the only sources of information about how to do this stuff that are available to the people who are learning how. And so we recognize that you have a certain responsibility by virtue of your knowledge, your position, your status, to take on the care and feeding of those young people as they are learning how to do this sort of work.

Gene Kim

Yeah. And Steve--

Richard Cook

We think that would be a really important activity.

Gene Kim

Mentor, mentee. Yeah. Yeah.

In fact, I misattributed where the advice came from because it actually started out with Steve saying the opportunity may not be in the reward, but in the consequence.

And my reaction was like, wait a minute. Aren't you going to freeze everybody into paralysis because we scared the crap out of everybody because of what could go wrong? Could you expound upon that?

Steve Spear

Echoing what these guys have been saying, what you all do has consequence. Whether you're writing code for physical systems which are far removed from you and people depend on them for their well-being and their safety, or you're writing code for communication or financial systems where people are dependent on those to function well, it's worth thinking about what's the consequence of getting it wrong.

Because if you get it wrong, there's someone downrange who's going to suffer for it going wrong.

And I want to tie into Sidney's point about telling stories. We worked a number of years ago with healthcare providers in Pittsburgh, and this one fellow, Rick Shannon, who was head of critical care there, was trying to motivate people to worry about complications, central line infections, ventilator pneumonia, that kind of thing.

He kept showing them data, and everyone had an excuse for the data. "Our patients are sicker." They always are. These are sick people to begin with. "We're above average," et cetera. And he made no persuasive progress that way.

One day people came in and in the break room were posters of patients. Now, normally when you go into a hospital and you see posters, it's the patient you dealt with: the mother who gave birth to a healthy baby, the person who regained mobility.

And when people started asking who these people were, Rick started explaining, "This is the guy, because of a complication, he'll never have a catch with his son on a Sunday morning when the Steelers are playing in the afternoon. This is a woman whose daughter will never know her because she passed too early because of a complication."

So it was these stories of what the risks look like that proved to be, in the end, highly motivating.

Gene Kim

And my claim was, "God, Steve, all you're going to do is scare the crap out of people. You're going to paralyze people." And you said something somewhat startling.

Steve Spear

Well, look, fear doesn't necessarily have to lead to cowering. Cowering only occurs when your brain almost literally short-circuits and gets into a do loop of, "What do I do? What do I do? I don't know. I'll just sit on the floor behind a chair and hope the problem goes away."

But that's not what we're talking about. What we're talking about is with discipline, rigor, energy, enthusiasm, optimism, recognizing moments where we're not understanding what we should be doing and how we should be doing it. Say, "Yeah, but I know how to address that. I know how to address that because I'm trained and practiced and part of a profession of problem solvers."

So it need not lead necessarily towards paralysis.

Gene Kim

Just to reflect a little bit. When this distinguished panel was giving this advice to John Willis and myself, I realized I had made a mistake, right?

In the experience reports that are shared, I think we go out of our way to pick the very successful ones, right? These are the triumphant people who created a coalition. They mobilized against powerful incumbent systems. They triumphed, right?

And it made me realize that there's equal value in covering the other side. And if you can help me make that more concrete, what is the value of leaders that have captured these stories of, not necessarily failures, but of--

Richard Cook

Successes?

Gene Kim

Near misses?

Richard Cook

Near misses. Yeah.

It won't happen in here because of the social constraints on the way that we talk about things and this sort of thing. But after you go to the hotel and you have a couple of beers and you sit down and talk with other people, the stories are not about the triumphs, okay?

It's not people standing up and talking about, "We were able to cut our costs in this way," or, "We were able to do this and do the other thing." The stories are all about the catastrophes.

Sidney Dekker

Or near catastrophes.

Richard Cook

Or near catastrophes.

There's another old surgical saying, which is, "Good results come from experience, and experience comes from bad results." And there's a lot of truth in that, and you all know that.

Sharing those stories is actually very helpful. Being able to talk about what that experience is like and what the kind of dilemmas and problems you've been in is actually very helpful. It's what John Allspaw and a bunch of other people have been trying to look at in more detail, so that we can understand something about how those things evolve.

But we have to be honest about the extent to which we find ourselves in circumstances where we don't know what to do. That's what problem-solving is all about. Problem-solving is what you do when you don't know what to do, and that occurs much more frequently than most people are willing to admit.

And I think we should be raising that up because your skill is partly in being able to steer the system and direct it to where you want it to go, but your skill is also being able to look at a system that's not performing well or that's having some difficulty and figure out why and make those sorts of corrections.

We need to understand both of those kinds of activities, and we'll get to them by sharing both success stories and stories about the catastrophes or near catastrophes that have occurred.

Gene Kim

And it occurs to me that this also helps reinforce the notion of the humility of the leader, right? The leader is the only one who can influence the system as a whole, right? More than the person on the front line.

Richard Cook

Yes.

Gene Kim

Any other thoughts on that?

Steve Spear

Yeah, I want to tie back to Sidney's point about the leader and the modeling, the coaching, and the social rewards.

So many of you know the Navy had a series of mishaps in the Pacific this last year. Three collisions and a grounding. And you start thinking through all the reasons that happened. It led to a loss of life, injury, and tremendous material harm.

And let's think about the grounding one. So what happens there is a crew finds themselves in circumstances which is beyond their control. The tide has changed. There's wind that they hadn't anticipated. The weather has gone up. They're on a schedule which they hadn't exactly planned for.

Richard Cook

They're driving a vessel that is made to not be seen by others.

Steve Spear

There is that. That's right.

One destroyer captain described to me, said, "Just to understand the circumstance, imagine going to Walmart the day after Thanksgiving, in their parking lot when it's still dark out."

Richard Cook

In a black car.

Steve Spear

In a black car with your headlights off, right?

Richard Cook

With the headlights off.

Steve Spear

But you start thinking about the issue of stories and leadership. So let's back off from the ship that didn't actually run aground and encountered circumstances like that.

So what's the story that crew tells? Well, the story that crew tells is how they mastered the situation. And say, "Hey, Gene, man, I tell you, the waves were up to here, and the wind was going like this," and on and on and on. "But we conquered it."

Well, that drives in the direction that success is through heroism. Well, they may have conquered it only because they got dumb luck, because the wind just was slightly off from the crew that actually ran aground.

The question is, do they come in and tell the white-knuckle story, which is, "Oh, man, Gene, it was that damn close. And we did our best, but we were clearly not in control of the situation. And but for the grace of God go I, we might've been the one that ran aground."

So now, what drives the same story told, "I was a hero," versus, "I just escaped by luck"? Well, when you come and tell the story, how do people react to you? In particular, how do your leaders react to it?

They say, "Hey, great heroism over there. That was fantastic." Or, "You did what?"

Richard Cook

Or, "Oh, man, tell us more how you almost went over the cliff." Mixed metaphors.

Steve Spear

"Tell us how you almost did, so the rest of us could figure out better what to do in such circumstances."

Now, I think Allspaw has said it beautifully, right? That an incident is an unplanned investment. And if you don't see it that way as a leader, you are not getting a return on the investment that was already made on your behalf.

Gene Kim

By the way, if I can just pause for a moment. You see a very collegial interaction between three very distinguished scholars. That's all veneer. That's all veneer.

Three and a half hours ago, it was like World Wide Wrestling Federation. There were chairs in the air.

Sidney Dekker

That's not true. There were no chairs.

Gene Kim

Yelling, right? It is amazing to see this. When I say it was sort of white-knuckle at times and stressful, it was--

Sidney Dekker

Oh, that was only your problem, man. We were having fun.

Gene Kim

Well, yeah. John Willis and I were actually quite nervous.

Sidney Dekker

It was actually three on one.

Richard Cook

He's easily fooled, though.

Gene Kim

Any other advice that you shared with me that you want to share with everybody else, in terms of unsolicited advice that you would give to us as a community?

Richard Cook

Don't be so easily fooled.

Gene Kim

Don't be so easily fooled. Yeah. Thanks, man. Doctor.

Sidney Dekker

No, no, no, no. Dr. Cook's advice.

Gene Kim

His advice. Go ahead.

Richard Cook

I recognize that there's a lot of expertise in this audience, and I think you have a kind of moral responsibility to share that expertise with the people around you, to help them to become better prepared to deal with the kinds of problems that they will deal with, even though you aren't going to be there at their side.

I think that responsibility needs to become part of what DevOps is. I think DevOps needs to go beyond toolchains and beyond repositories and become a kind of practice that involves people, what John has called above the line.

To do this may require that we identify ourselves as people who practice DevOps rather than people who work for company X. That is, the practice of being a DevOps person has to be actually a kind of profession, a kind of skill and expertise that exists apart from the particular employment that you're engaged in right now.

You've chosen to go to work where essentially the rubber meets the road, at the sharp end of the stick, at the cutting edge of things. And because you're working there, you're going to have to encounter a lot of stressful situations, demanding circumstances, places where there's potential for real loss, and the kinds of things that will keep you up all night.

And I think you need to try and help the people around you who are learning about this understand what they are getting into and how to cope with that.

And I don't know exactly how to accomplish this except to say that the answer for this lies here in this room, not someplace else. It is not someone else's job to do this. It is not some other agency's job to do this. It is not your company's job to do this. It is your job to do this.

Steve Spear

Hear, hear.

Richard Cook

And I think unless you find a way to band together to find those common threads and the kinds of responsibility and moral activities that you engage in, that you run the risk of becoming regarded as just another group of technicians. And I think that would be a very great loss.

Steve Spear

You mean like anesthesiologists?

Gene Kim

Oh, I got to hear scholar jokes. Like, how do PhDs and doctors make fun of each other? I got to hear that.

Richard Cook

How do they?

Gene Kim

It was funny in the moment.

Sidney Dekker

We love you. We do. We do.

Gene Kim

Actually, why don't we end on that joke?

Sidney Dekker

Your foot is in your mouth already, so it's--

Richard Cook

We're very grateful to Gene for inviting the three of us here. He's taking some risk in doing that.

We're, as you might have recognized--

Gene Kim

You have no idea.

Richard Cook

--not all people who produce reliable performance under all circumstances. And Gene is taking some risk in bringing us together and building the kind of powder keg that exists with having three of us together.

But he's reaching for something, and I hope that you can identify what it is that he's trying to reach for, because I think that you have to embrace that and reach for it as well.

I think that, more than any other technical aspect of these kinds of conferences, is what this is all about. And I encourage you to think about that and to pursue that over the year to come.

Gene Kim

And then I will have the closing words, and then I have an ask for all of you.

I had made the claim to this distinguished group on this panel that this is such an interesting and powerful community because you have all self-selected to be here. The fact that you're here signals that you see something bigger that we're aspiring to, often at great personal risk, and that you're building that coalition to overcome very powerful incumbent systems.

So I guess here would be my ask to you. If you are willing to be able to share some of these stories of uncertainty and making decisions when you don't know all the answers, I would love to hear from you. Just email me, Jez, or Anne Perry. Right?

And this will be an obvious action item to create a section for that in DevOps Enterprise 2018.

So, with that, a round of applause for Dr. Richard Cook, the doctor. Steve Spear and Sidney Dekker made sure that everyone knew that he doesn't have a PhD. He's only a medical doctor.

Dr. Steven Spear, who spoke at DevOps Enterprise 2015. And he made sure that all of us knew that he's from MIT, not Harvard, not UPenn.

And then Sidney Dekker, who reminded us all the time that he is a pilot and has two PhDs, so--

Sidney Dekker

I have three now.

Gene Kim

Oh. Thank you so much.