Hey, everybody. Welcome to the DevOps Enterprise Summit, Amsterdam 2023. I'm Damon Edwards from PagerDuty, and here I'm sitting with, uh, John Willis. Yeah, John Willis and Costley.

Hey, John. How's it going? Good. Long time.

See? Yeah, I know, I know. So, uh, you know, um, how About that DevOps Cafe podcast? Someday we'll finish it.

I, I promise we have a long suffering podcast we've done together, but, so John, so you're here to talk about what, what is your, what is your talk about, Um, DevSecOps and operational operationalization? Can you give us a little preview of what this is all about? Yeah, it's, um, you know, I, obviously I'm, or maybe not obvious to everybody, but people do know me that I'm, um, fixated on this guy named Edwards Deming. I've got my book coming out an it revolution this, uh, August 8th.

You can order it on Amazon now, but, uh, but it's been a 10 year journey of trying to understand what were these principles that this, this person had, this guy had built, how did it influence, uh, America? How did it influence Japan, a Toyota? And, um, you know, the more, the deeper I get into it, the more I understand the, these sort of universal principles and one of the core principles that you don't get from his book straight up. In fact, you have to read his books multiple times.

Just can You say his books? You say, Oh, Edwards. I mean, like, what can you explain? I mean, just give the, gimme the, give people who dunno the super short brief as to why this person is so important in our lives.

Yeah. Probably the most, uh, you know, he was a management, um, sort of consultant, um, uh, theorist. Um, and he, um, probably the most famous thing is MacArthur sent over a team of people after World War II to Japan to, um, help rebuild all sorts of areas. Demming was one of those people.

Um, his ideas, um, which we call profound, um, really helped, um, changed everything about the way leadership thought about in Japan, and in some cases attributed to what they called the miracle in Japan, which, uh, you know, Dr. Spear talks about this often about this sort of decimation of what Toyota and some of the Japanese country companies did from, you know, starting probably in the late seventies, eighties, nineties, all the way up to two thousands, particularly Toyota. Um, so his principles, and so he's, he did a whole lot of other things. He also, when he came, um, you know, so he came back from Japan.

He was basically in obscurity until 1980. They, there was a documentary about what Japan was doing. Uh, if Japan can, why can't we? Deming was near the end of that, interviewed, and people in America saw, including Ford Motors, saw what this, uh, this, this guy's ideas.

And they realized it was, it was an American who taught the Japanese to actually come back and beat the American car makers. And so he just a fascinating life. Uh, but his principles are just built into systems thinking, into psychology, into, uh, all facets of variation, all all the things that make any type of flow. Not only sort of producing automobiles, but producing software.

Just so, uh, perfect. So anyway, so what one of the areas as sort of peel the onion of learning about this guy, um, is the, the how, how directly related to operations management, operations research, all the stuff that's used, I would say from building toasters to nuclear power plants, but not used in it, um, of this sort of operational definitions and operational definitions are basically the idea that you, you really shouldn't measure anything unless you have clarity of the sort of methods of what you're going to measure and how you're going to measure. So it's, you know, just measuring and saying the number is 42. Like I, I joke with Mark Burgess, the physicist, right?

There is no such thing as 42 right there, you know, um, it, it's 42.001, it's point, you know, um, negative 42, whatever. Um, but operational definitions allow you to sort of understand the why you're trying to get that data and what that data actually means. So, so can You explain a little deeper about what, what do you mean by operational definitions? Like what, gimme some examples or, Yeah, so I'll give you an answer.

So it goes back to a guy in the, uh, early 20th century, uh, Percy Bridgeman, who was fascinated, well, he actually, he was this, I'm gonna talk a little bit about this. He was basically, um, working on synthetic diamonds, and he kept breaking the gauges, and he was really sort of like, uh, perplexed, I guess based on how do you measure things that sort of are very difficult to measure. And when he saw, um, Einstein's theory of relativity, he realized that there was, um, that he, that Einstein, and I'm not a physicist so I'm gonna bumble this up, but that he was able to use different domains of measurement. And it opened up this idea about what, what does a measurement really mean?

Like, for example, a length in some senses, like a length in this room might be, you know, 30 yards, or we might do square footage, but if we go into things like, uh, different planets, they're light years, right? Like, so the, the method of, you know, what you're trying to do. So the example to go back to earth, um, is like, if you take the door metrics, I'm a big fan of door metrics. They help accelerate excuse upon our industry.

There Was the DevOps research, what is this? DevOps, DevOps, uh, research, something famous, The famous project to measure, to measure the performance and impact of We should this DevOps. DevOps, yes, there's A book, there's a book about it And the whole accelerate and the portion. But, um, but there, there's interesting metrics like lead time, for example.

Like, so, um, the question I have if do, you know, when we talk about lead time in the industry and in a company within sort of organizations within a company, um, different groups, if we say that our, so this is what we're doing for lead time, what are we defining lead time for? So you might be defining lead time. So I, and I, I'll ask simple questions like, um, is it on commit? Is it first committed is on the pull request?

Is it, it continues delivery continuous deployment, right? Is it, there's a big difference. Is it when The, is it when the plan started, started formulating, right? Yeah.

Three quarters ago. And I'll even give you the sort of, the accelerate version is from sort of commit to, to production. Yeah. But again, I can ask what kind of commit?

Is it a dark deploy? What is production? Is it a feature flag? Turn on, right?

So, and like, I don't care what your definition is. The question is if you're in an organization with different divisions, and and I, when I start bringing this up to groups, they come to me like, that's our biggest problem. We have no consistency on these metrics. You know, one team said, oh, well we hadn't really thought about it.

Another team, well, no, it's from commit. I said, what? Commit? Oh, well it's fish commit, right?

Or it's, it's, uh, the, the merge, right? And, and so you could do that, like, you know, and, and we can, like, we can poke, you know, a freight train through M T T R. I always joke that like anything that's any metric that starts with mean run, get out, leave. But, um, but, but even like, um, you know, um, any, any of the sort of the change requests, all those things, just to be clear about, again, the point here is, and you can see this in operations research, is having a clear criteria, a clear test.

It's a P D C, a way of implementing into, you know, plan, do, study, act, actually way of looking at the way you're looking at metrics. Do you have a criteria? Deming defined it specifically as you must have a criteria. You must have a test and you must have a decision.

And so if I just say, Hey, I'm gonna monitor, um, an ss l a, alright, so what is the criteria for the sla? What is the actual test that you're going to use? And then what is the decision that you're gonna make? And there, and there, their point is that unless you have that stuff, you really, there is no measurement's A Question.

Is this making sense? Yeah, it is. Do Do you think that, do you think also that people sort of back into it, they're, they're going the wrong way. Like they're looking for metrics and certainty, almost like KPIs before they're seeking to understand what is they're measuring?

I, I, I think that's the, that was Deming's point, right? And that was person, uh, Bridge's point too, which is, um, like, it sort of a joke, what might not have made sense earlier. Maybe it makes a little sense. We saw like, what is 42?

Right? You know, the, the answer to everything in the universe, right? Uh, Douglas Adams. But no, it, it's, um, that, yeah, I think that is the point.

We, we tend to sort of, I mean, we've seen this growing up in not just DevOps but infrastructure, right? You know, we've learned to sort of like, okay, we need to look at like what the percentage is of something. What does that really mean? And then we can get into the floor of averages.

But, but I'm, I'm even going beyond that. I'm saying that like, and Demming would say that, you know, there is, there is no measurement with an op, without an operational definition. 'cause it's interesting, I like you talked about the, you know, Toyota and, you know, and Deming's influence on what became, you know, Tai Chio and the, the, the great Toyota production system that kind of became the, the engine that, you know, powered a whole sort of economic, uh, uh, you know, renaissance and manufacturing dominance. Um, and it's interesting that, you know, I think is one that you told me that, you know, there's not in that system management doesn't come in and say, here are the KPIs.

Go make it happen. Right? Management comes in to understand, to develop management is there to develop the system. Yeah.

And then they expect the, the kpi, there's, you know, I don't need to call KPIs, the metrics, whatever They are, to Come from the bottom to come from the bottom, from the bottom up. It's people saying, we know what we're trying, we know how to, we're building a system. Yeah. And to make this system better, we should measure this thing, right?

Versus coming down saying, you know, thou shalt measure this, and I want you to, I want you, you p people down there to tell me how we're gonna make this number better. It comes from let's build a system. And, and the me measurement comes from the ground. Yeah.

Which is a very different, And not to get too meta, and this is why like, we should do more podcasts. 'cause like, you compliment me because I told you the story, I'll tell the story about KPIs in Japan. But, um, you know that we as humans, we're always looking for certainty, right? And it's our nature to sort of deterministically define a certainty.

Oh, I think the system is running good because it's at 42, right? And, and the truth of the matter is like, we live in incredibly complex worlds. We deal with complex systems and, um, and it's all more about probabilities. And, uh, and so like, again, another big part of operational definitions is you that whole, like likes P D s A plan, do, study, act is implementation of scientific method, scientific thinking, which is, there is clear uncertainty.

And so the idea is you approach measurement through the concept of uncertainty. And what, how do you do that? You plan, do, study, act, or what Deming would say was criteria, you know, test and decision. So back to the K P I story.

I was in Japan last week on a study group, and, um, I was only one other person was sort of DevOps, Glen Wilson, who wrote the DevSecOps book, he's shout out to that. He's an awesome guy. But it was me and him, and it was like 13 other lean practitioners from healthcare and all these different areas. And it was, um, interesting the difference between lean protectors and that don't do DevOps.

And DevOps and, and I won't say all DevOps practitioners are in on this, but the, the, these lean practitioners who weren't involved in DevOps kept asking everywhere we went, like tier one, tier two suppliers, what their K p I was K and, and you know, the, the, the Japanese like worker or leader or manager would just not understand the question and all week. And then finally at one point somebody said, you know, they, the translator tried to explain it. And, and then a person came back and said, oh, we don't do that. And he says like, would you ask your family for a K P I?

And so the point is that, um, like I, I like that you tied that together to this. 'cause I wasn't even thinking about that is that's why, because they have clear definitions about what they, they have, they sort of, they may not call 'em operational definitions, but they're operational divisions about what they do. And the answers are sort of systemically built into the process. So when you ask somebody in Japan, in most cases, like, what's your K P I for customer satisfaction, once it gets translated, they think you've just asked like, you know, you know, why do pigs fly, right?

They're like, what, what is that question? I don't understand it. Well, you know, it's interesting. Um, so I'm gonna tie back the DevOps, uh, movement, right?

That, uh, the metrics part of it, right? We talked about you need metrics, right? Part the CAMS thing was like culture. Like we, because we came up with this idea of cams, right?

Right. Long time ago was on a podcast was talking about sure, how do you know you're having a DevOps conversation versus some other kind of conversation, right? That was what we're trying to come up with. And we came up with this culture automation metrics and then sharing, right?

These are like the, the four things that when you're doing all those things together and you're talking about Devon ops, right? Then it's, hey, this is, this is a DevOps conversation, but interesting. There was no really metrics to start, right? Metrics are like, you should measure things.

That is pretty much it is the actual metrics, like the, like the door metrics, acceler, Kohl's book accelerate, all this stuff came much later. And I feel like the earliest part of the DevOps movement, all the DevOps days conferences, they were just talking about the problem, right? There wasn't a lot of solutionizing to say at that, at that point. There wasn't really any measuring or even knowing what it is.

It's just the fact of that, you know, Patrick's original idea of DevOps show up at the same conference, one of 'em in the wrong place, right? Yeah. Yeah. That was number one.

And then number two, why isr this flashpoint between these two groups, right? That things always go, The wall of confusion Always go wrong, right? And so, remember the longest time, the only question was like, well, it must be either puppet or chef, right? That was the, yeah.

In fact, the puppet and chef folks were like, code was, why do you have a DevOps conference? Why'd you come to puppet con or a chef conference? Right? Like, that's which We both went for the first, Yeah, it's over.

Right? He's used one of these tools. Then it sort of grew, expanded from there. But so much of the, of it was just, you know, it was years of just talking about the problems like mass group therapy, right?

Right. Versus saying like, solutions and measure and measurement. The solutions in measurement, I think came much later. So the whole movement was deep, deep conversation about the nature of these two kinds of work and the problems and why is it such conflict and, and, and then learning and, and later on came the, yeah.

And I, The metrics, again, maybe it's the natural evolution of any type of movement, but, but I mean, and again, like, let's be clear, like you need a measurement, right? Of metrics like you, it's like it, it's, it's necessary but not sufficient. Right? Um, b um, the Dora metrics, you know, again, excuse the pun, accelerated our industry.

The question now is like, so going from describing the problems and trying to understand the problems, like you said, to putting some deterministic envelope on it, which is sort of how we started monitoring to now let's expand the conversation to like, you know, what other trics use from building toasters to nuclear power plants. Like we, we, our evolution needs to grow. We can't just say, you know, you are, um, uh, uh, um, a high performer if your lead time is this, right? And like, and we don't have a clear operational definition of what a team, a silo, a division, an organization, an industry actually means by That.

Yeah. And that's what I, I was, I was using the example of the DevOps movement as a sort of a macro example of what should happen inside your company, which is talk about the prob, like build the operational definition. Sure. Seek to understand the problem first, build an operational definition, then you can actually measure, then actually measurement drives you forward, right?

'cause everybody has that common view of what it is we're, you know, they know we're trying to solve, they understand the process, you know, should or shouldn't be, and then we could measure it, right? So it's like that cycle, you know, of, uh, and I feel like we went through that as a industry almost, right? Where it's like, Yeah, in All first understand, understand the problem, then define what it is we're trying to Yeah. To The, the process we're trying to implement.

Then you measure it And we, it's like a rest stop on a highway. We got off at the necessary rest stop to deal with this sort of our sort of initial implementation of measuring. Now it's time to get back on the highway. And in case anybody's completely confused right now, I'm like, what in the heck are these guys start?

They press stop. Yeah, well, all right. At least they, that's true. If they stuck around.

I'm gonna give you one example I used in, in the presentation, which I think really, and it really comes from a couple of demming examples of why operational definitions are important. So I show a picture of, um, a restaurant and there's different people, people sitting, there's people standing. And I ask the first question, I point to a woman who seems to be looking into the restaurant. It might be in a mall.

And I say, um, oh, for the first question, sorry, I ask how count the amount of people that are in the restaurant. And then I ask, is this woman over on the side who's looking in, do we count her? And then there's somebody behind her who seems like she might be further back. Do we count her and then they point an arrow out to the waiters?

Do we count the waiters? Then there's an open kitchen in the back and I say, do we count the kitchen staff? And then I put a big old, why in the heck are we counting, right? Because if we're counting for, uh, food logistics, then it's the people eating.

If it's weight distribution and it happens to be on a cruise ship, then it's everybody. If it's just fire code, then it probably isn't the two women on the side, right? It may not be the kitchen staff. The point is the back to you saying like we, in the early days, DevOps, I think we were searching and we were starting to formulate what these definitions were that could solve the problems we were discussing.

We took the off ramp, we came up with a simplistic way to start measuring it helped the industry. We get back on the ramp and we start asking the questions about measurement. Like why exactly do we care about lead time and what is, what are we trying to get out of lead time? So what is the criteria of lead time?

What is the actual test? Right? Is the test should be very specific at this point? Oh, it's pull request to feature, flag, turn on.

Yeah. And then, you know, what is, do we have a perception of a decision that that's gonna enable us to do something in a continuous improvement? Well, it's fascinating too because then if you don't understand that just a slight change in the process you think might be for whatever reason, suddenly you get an improvement or a, you know, a degradation of a metric without understanding, oh, we didn't actually do something better or worse. We just, you know, well changed How it works.

And, and, and just the one other piece that I learned recently, again, it's the thing that's fascinating, Deming, and, and almost all of this we just keep learning, but I know probably wanna wrap up soon, but, uh, David Wheeler, who is the, I guess the canonical expert on statistical process control, written a ton of books, and I've started reading his stuff and he superimposes a definition operational definitions, and he says that like the criteria has to include consistency. So again, is it 42 or is it 42.1, 42.5 40, uh, 41.9, right? That again, the, you know, like Deming, Deming says is variation is a part of life. So, you know, so, so he includes a whole notion of it's even beyond that.

You need some sort of a statistical analytical, statistical model to be able to really know what the criteria is. Because it's, it's, it's the, you know, at its core, it's, um, measure twice cut once. The reason you measure twice cut once is you don't really believe the fir one measurement. Yeah.

One number 42 is the right number. So, um, just to wrap things up, kinda one last question for you. Sure. You know, so this is a, this is a pretty heady topic, right?

'cause it kind of strikes to the heart of, you know, business culture and how we operate things, right? And how, how we work as individuals. Do you have any advice for people who, you know, want to get sharper on this or who, you know, are looking to bring this thinking into, into their company? You know, how do they, is there a, you know, kind of a place to start or way to kinda get their, get their arms around this?

They can, you know, make it actionable? Yeah. I think, um, you know, I, I, um, you know, I I, I go back to like the people I've learned so much from, I've learned so much from you, Andrew Clay Shafer, Ben Rockwood, you know, Ben Rockwood has always been preaching. The, the notion of why we don't include operations, research and operations.

There's just an incredible wealth of information about how to build, you know, franchise and McDonald's and stores. And they're all about flow, flow and delivery to customers. It's like what we do. And like we, as far as I could tell, you know, um, apologies to anybody who sort of does do this, but in general, in our, our travels, I don't see a lot of operations management research.

So I think the answer is, and so operational definitions to me was the, the simplest way for me, you know, so the horse version of not a unicorn, like if I could get my hands on this, I've got sort of at least one principle of operations management operations research. So I take the, the short answer is operations definitions, you know, what I'm trying to build. Where some of the, the value read demings to sort of main books, um, out of the crisis. That's where he really talks about it the most.

But New economics, which was the last book. He covers a lot of examples. Um, but then beyond just operational definitions, start looking at how do we as experts and theorists and practitioners really start looking at some of the operations research. I, I've been taking some operations research classes on Coursera.

Um, I wish I could go back to university and get a degree in it. It's just fascinating. Good stuff. Yeah.

Alright, well those people are still listening to us That we say on our podcast too, by the way. Yeah. And everything people did, it was great. Um, so, uh, yes, thanks for, uh, joining us and, uh, stay tuned for more videos.

Thank you everybody.