Mark Michaelis

My name is Mark Michaelis, and I don't know if there's much else I can say about that. I just finished a book. That's my book, but it's about Dev stuff. So I don't want to tell you that, because then you'll think I'm biased.

I want to start by talking about some of the problems that would lead us to want to think about metrics. I'm sure none of you guys have these problems. These are just hypothetical things that I've put on the screen that you may know somebody, maybe somebody sitting next to you, has these kinds of issues. But you personally, of course, have none of these going on in your organization.

And I want to talk about this not by those specific things, but to start to think about a higher level of the whole build, measure, learn. The fact is, we hold this value from The Lean Startup, from Eric. We hold this value very tightly, but the whole measure piece is a pretty big hole for us. And when we read about this whole, "Oh, we've got to measure," we get the emphasis that we need to do it, but we don't really know how to do it.

When Eric talks about the fact that, hey, let's take a requirement and carry that requirement all the way through to production in a way that allows us to evaluate whether we're successful, not only in doing development, not only doing operations, but actually in evaluating the business side of this as well, the business value.

As developers, we tend to focus on: what does the product owner tell us to do? And as long as they are right, we're going to produce the correct thing. And as operations people, all we want to know is: how do we deploy this? We get the development team to tell us how to deploy and keep it up and running. We don't care about the what is running. That's not important to us.

In fact, we go through Sarbanes-Oxley kind of things, and they say to us, "Hey, make sure you have a record of exactly what you deploy." And so the developer hands us a script, because we're DevOps people. They hand us the script, and we execute the script. We have no idea what it does. That's not important, right? If it's going to steal money, that's not relevant. The only thing that's important is we follow the steps exactly as the developers give them to us.

At the end of the day, however, if we're starting to do build, measure, learn, we need to start transforming these metrics and start to talk about the sense of, hey, how do we take metrics and begin to measure the value proposition, not just within our team, but within the whole organization? And metrics, we want to go ahead and say they're not just specific to me, they're not just specific to the CIO or CXO, they're specific throughout the entire organization, and they're important.

The one message I want to tell you, however, is it is not the same metric for everybody. If you're here saying, "Hey, Mark, I want to know which ten metrics I'm going to use when I leave this room," you should leave now, because I'm not going to give you that answer.

Unfortunately, it is all a matter of perspective. It's a matter of what matters to you, what matters to the development, what matters to where you are in the life cycle of software development. All of those things will change your perspective in terms of what matters.

A couple things that we got as issues when we talk about metrics. Number one is, I think, complexity. And it's not complexity in terms of, oh, what are the units of measure? It's units of trying to figure out how do we actually measure this. For example, how do you measure wait time? What happens if it's overnight? Is that considered part of wait time? Well, maybe if you're down it is, but not if the requirement is entered into... if it's just part of your development cycle and you're running a Scrum. So just measuring certain things is hard. That's difficult.

Another problem we have is that metrics can be misleading. On occasion, we'll go say, well, the thing that's most important is total customer count. How many hits I'm getting. Hits is a very ambiguous term, but how many hits am I getting? How many customers do I have? Or you can say, how many new acquisitions am I getting? And then you talk to finance, and finance says, "Well, wait a second, it's costing us more to get these customers than we're earning from the customers. There's a problem." But originally, I was just looking at the fact that total customers was increasing, and that's what mattered.

Here are four things that I want to focus on in terms of why we should measure what's important. These are the kinds of things I think you can take to your management team and say, "This is critical, and this is what's critical about it."

First of all, they reduce subjectivity. The whole idea that a dev team says we are doing great because they feel good because they're delivering on what they said they were going to deliver in the sprint. Okay? The whole idea saying the whole point about Scrum is that we will go ahead and take the developer feels good, or their perception that they're doing well, and start to put metrics around it. Okay, developer, tell us how big is this story. Let's see at the end, what is your velocity? We're actually starting to put quantitative measures around our productivity.

When we look at that number, that metric, that velocity, one of the things that's key is that it's very hard to do at the beginning. In your first sprint, how many points are you going to get? Is it 25? Is it 30? Is it 3? We have no idea how many points we're going to get when we start. And we can't predict. For that, we can sort of make a forecast, but if it's accurate to 50%, we're lucky.

After three sprints, we're able to establish a pretty strong predictability. We're able to say, after three sprints, I know my velocity is going to be around 23, or 23 and a half. No, 23.

And when I make that prediction of 23, if I go into the next sprint and I've got 30, I should be very suspicious. I should probably ask the question, wait a second, I've consistently done around 23 or so for the last three sprints. What makes me think I'm going to be faster in this set?

And so the whole idea with metrics is they allow us to start predicting not only our productivity, but also in terms of what we think we're going to do in terms of uptime. How many of you can say before you go launch, "My uptime's going to be 99 point X"? You have no idea. Guess what? When you start creating the metrics, that's when you're able to start predicting.

The other thing that's interesting about metrics and choosing specifically which metrics are important is they force you to focus on what is strategic, what actually matters. Most of you, at least the people that I talked to beforehand, they wanted to know which metrics do I choose. And the question is not driven by the metrics then goes in and tells you what you're going to measure. The question should really be driven by the business. What do you want to know?

For those of you who are doing BI, this happens a lot. I get these big BI projects, and we start investigating, and the whole dream of the BI world is, give me a whole bunch of data and knowledge will just pop out. It's just magical. I go ahead and throw that stuff at machine learning, and I just start to know stuff.

I think when you start to approach metrics, it's not like that. You need to start approaching metrics with questions. What do you want to know? Because when you ask the question about what you want to know, you'll start to understand what you need to measure.

If you don't ask what you want to know, if you don't ask that first question, what do you want to know, you're going to choose metrics that are not relevant and they don't matter. So if there's any principle I want you to start with when it comes to choosing your metrics, it's figure out what your strategic direction is, and from that strategic direction, ask the questions of what you want to know. And then base your metrics, what you choose to measure, on what you want to know. You must strategically think about those metrics based on what the strategy of the organization is.

The other thing that's interesting about metrics is we have this concept of metrics being transparent. And transparency has a tendency to move us towards excellence. When you start to go ahead and expose and take those metrics and allow other people to know about them, that starts to create this desire to be better. And that's a natural tendency that we all have. Last time we did this, now we want to start doing this.

We want to take that metric and move it up or down, depending on the metric. We want to move that metric up or down and become something that we're challenged by.

Patrick Lencioni talks about job satisfaction, and he says that one of the keys to job dissatisfaction is immeasurability. And he's not talking about DevOps. He's talking about job satisfaction. He's talking about the HR quality of saying, how much do I like my job? And it has to be measurable.

When we talk about Daniel Pink's book, Drive, he says one of the things that creates job satisfaction is the ability to do mastery. And here again, mastery is key. If I'm able to measure, if I'm able to look over time that I've got a tendency towards mastery, that's going to go ahead and create the desire to be better, and it's going to move me towards excellence.

This slide doesn't show up very well. Let me see if I can do this slightly differently.

One of the things that became a problem when we looked in the DevOps Forum at metrics, we sat down and we did a brainstorming session, and we came up with a whole bunch of metrics. Any ideas as to how many metrics we came up with? I think it was about 30 minutes. Do you guys remember? About 30 minutes of brainstorming. How many metrics do you think we came up with?

Seventy. Higher or lower? What do you guys think?

Higher. Higher. Okay. What are we at?

Two hundred. Two hundred. Okay. We've just reached our capacity.

We came up with 150 unique metrics. Now, we didn't go through those metrics and say, "No, that one's the same." We didn't criticize anybody. It was a brainstorming session, means everything's right.

But with 150, the automatic desire of all of us was let's organize these. And this is kind of what we ended up with, a mess. Because it turns out that as we try to organize...

Oops, my slide doesn't work when I'm not looking at slides.

As we tried to organize, we discovered there were too many dimensions in which we could pivot. We could look at DevOps or the metrics for DevOps. We could say, "Hey, let's look it from the perspective of the CEO." What matters to them is presumably different than the nerd.

What about the HR person that's worried about who we hire? Their metrics are probably different, again, than the nerd or the CEO. We could pivot around organizational structure. We could say, "Hey, let's go look at it from a perspective of ops. Let's go look at the metrics that matter to developers. Let's go ahead and look at the metrics that matter to HR." And we came up with a different pivot of how these things work.

We could do them by category. We say, hey, there's operations, there's people, there's business, there's process. Is process in the middle? Process on the outside? We discovered metrics fitted into multiple categories. It became extremely difficult to sort of assign a taxonomy to the metrics that we had.

So we ended up with this big list, and we pivoted. I've got an Excel spreadsheet with all the metrics that I've gone ahead and created columns with. Okay, let's go ahead and look at it on this pivot, and then let's generate it on this pivot. And that produced for me that mind map.

Whoops, that's not where I'm going.

Here's just one of them. I went ahead and created a mind map of the metrics that we had. And I said, "Okay, let's look at a pivot that's based on this." Let's talk about some of them.

Is that big enough? Hey.

So one of the questions to start thinking about is from a business perspective. And remember the emphasis here when you look at the business perspective is that this doesn't necessarily matter to the developer. When I started looking at business, way more of them were financially based than they were the speed of my build, or the number of times my build failed, the frequency of failure in my builds.

I started seeing the business perspective allowed me to assign metrics that were important to people higher up in the organization, that were not necessarily in the world that many of us in DevOps live. But these metrics are truly critical, and these are the metrics that emerge when you start to do build, measure, learn. Because these are the metrics that you're actually going to measure your success by. They're not the metrics that you're sort of measuring on a day-to-day basis.

I'm going to stand behind here for a little bit so I can use my keyboard.

We started to look at the metrics that involve people, and we came with this definition of cultural metrics. The one thing that was interesting about the cultural metrics is that we suffered the same problem that HR departments suffer, is that there's a lot of subjectivity.

And you remember earlier that I said, "Hey, when you go ahead and start to have metrics, they remove your subjectivity." Well, they only remove it to the extent that you don't have subjective metrics. If you've still got subjective metrics, you're still going to have some semblance of subjectivity.

However, over time, that subjectivity tends to move towards or trend towards accuracy. So it's not valid for one person at one particular time, but if you can start to evaluate people over time and evaluate these metrics over a longer period of time with more people, then you start to trend towards accuracy.

A couple things that you'll notice we talked about. One of the things I had a hard time with when I defined the technology category here, what was difficult about that is it tended to be things that fitted into both dev and into ops. But they were mostly focused around things I could gather really quickly or really easily.

And that was the nice thing about the technology metrics, was it didn't require a lot of work for me to go ahead and get the information out of the metric. Whereas some of the other metrics, like lead time, are a lot more difficult.

So when we go ahead and start to choose, hey, what metrics we want to focus on, remember that things like technology metrics are metrics that you sort of get for free. You don't have to do a lot of work to get them. And so you might want to choose those as low-hanging fruit.

I'll talk about this later, but you don't want to spend a ton of time getting metrics that don't matter. But if you can get metrics for free, you want to get them.

Let me say that again. You don't want to spend a ton of time getting metrics that don't matter. If they require work, if there's a cost to go get those metrics, don't spend time on those. However, if there's metrics you can get for free, low-hanging fruit that emerges from your tools, keep that data, gather that data. Because maybe now you don't care, but in the future, you might.

So one of the principles is to get the data if you get it for free, but don't spend time gathering data that you don't have value for yet.

I noticed this especially, I work quite a bit with TFS, and teams come in with... There's a boatload of reports. And teams sort of start working, they're like, "Oh my goodness, look at all this data we got." And then they start, "Let's turn this on. Let's turn this report on. Let's make sure we're gathering this data, and let's assign this work item to this source code and make sure we're tracking all this."

And they spend tons of time, and they get mired in the idea of creating metrics that they haven't figured out the value proposition for. And they're now wasting time getting that data rather than doing their job, the focus of getting their work.

A couple more. I think these are well-known, so I'm not going to spend a lot of time talking about these. I think they're well-known. But there are some that we may not be tracking. Things like time to escalate. Time ones are hard to track.

Oh, thanks, Dominica.

By the way, I see a lot of you taking pictures. I'll be glad to give you these slides or whatever, so at the end, just remind me, and I'll just post them on my site. You can download them. I don't mind. If I'm beautiful, I understand you want to take pictures, but it's way easier if I just make this available to you. So I'll give you a download at the end. I'll put a URL that will be where I put the slides. They're not there right now, but I will put them there.

So a lot of these are automated, but a lot of it comes down to what tooling you're using. Are you actually tracking when downtime happens? How am I aware of that? And a lot of this comes down to how good your monitoring is.

So one of the things you'll find when you start to do DevOps, if you want to get accurate measures, a lot of it depends on how good your monitoring is, and can you turn your monitoring into automatic metrics? I don't just want to know, hey, there was an exception. I want to know when the exception occurred and then trace it through the lifetime of that exception to when it was fixed. So as much as possible, use your monitoring to generate your metrics.

Did I do all of them? I think this one I didn't do. These are your standard process metrics. I'm not going to say anything more about this because I think you guys can know this. I'm running out of time, so...

Process metrics. The two that I typically hear as the most important metrics. I typically hear, that doesn't mean it's right, I'm just saying I typically hear, lead time, process time, and wait time, and I've gone ahead and given some definitions to that.

A couple things to note. If I'm a developer on a team, and I'm the guy writing code and participating in Scrum, do I care about lead time? Well, it turns out we only care about things that are within our sphere of influence. So metrics only matter to us when they're in our sphere of influence. When metrics start to go outside of our sphere of influence, we care less. The tendency is to care less about metrics that are outside of influence.

So, as a developer, when I start to get metrics that come in, at the time that it becomes available to me to work inside the Scrum, inside the sprint, that's when I care. And now I care about estimating it and executing it. Before it enters the sprint, I really don't care.

So if lead time means the user reports it or the customer reports it and it enters the backlog, it's not important to me at that point. As a developer, I'm executing. My head's down, working on the stuff that's within my sprint.

And so we talk about these metrics like lead time as though everyone should care about it. But it turns out that we tend to only care about the stuff that matters to us. And so it's very important for a developer how much process time it takes to execute. What is the time that I'm actually working on this particular work item? How long is the sprint? What's the velocity of my sprint? That matters. That's something I can control.

But when you start to put metrics outside of my sphere of influence, they lose importance, they lose significance. This turns out to be true as you move up to stuff that matters to the CXO. As we start to work up to budget numbers, that tends to be true as well because the developer doesn't have control over certain aspects of the budget. Clearly, they have influence on others.

So what you need to do, again, going back to the build, measure, learn principles, say, what things can you control that do affect those? Can you control how much work you're accomplishing? Can you control how much time? Can you control the quality that you're producing? Because bad quality out of your sprint is going to be way more expensive than quality that you fix within your sprint.

Those kinds of metrics you can change. And so what you can do is have an influence on what's happening at the higher levels by changing the things within your lower levels.

I don't expect anyone to be able to read this slide. My point here is that metrics are also relative to the perspective of where you are in the cycle. So if you're looking at discovery and requirements, the metrics that are there are very different than what you're looking at in DevOps. Okay? So it matters to where you are in the cycle.

The other thing to note is that what you're looking at changes your perspective. What you're looking at changes your perspective.

When you start thinking about which metrics am I going to choose, you need to think about, again, the strategy of the organization and choose metrics that are based on those. Because what you look at will change your perspective. And I don't mean you're intentionally trying to game them. I mean that they will change what you focus on because the metrics tend to move you towards excellence in those areas, and if you don't measure certain things that are important, guess what? You're not going to improve those areas.

So perspective matters.

A few principles. If there's any slide that I think is most important, I think it's this one, because this one is the criteria you can use to choose the metrics you're going to have in your organization.

First of all, obviously, it has to be relevant. And I talked about not creating cost for metrics that aren't relevant.

Move towards automation as much as possible. Move towards automation as much as possible. This is key. In all the work that I've done in metrics, as soon as I have a lot of manual entry, metrics tend to get lost after a week or two. Right? It's too much work. Developers have a hard enough time putting timesheets in, never mind anything else. Hypothetically speaking. I understand your organization is different, but my organization.

Transparency. Clearly don't have metrics that you're not going to expose. And you should be willing to expose those completely, entirely publicly. Don't say, "Well, it's just exposed within my team." You should be willing to be transparent completely publicly. And that's the challenge: when are you willing to go ahead and expose your metrics even outside of your group, even outside more broadly within your organization or even outside of that, just to give us an idea of what's going on?

Don't bother to gather metrics, meaning putting work in metrics, for things you're not going to examine. If you're not going to pay attention, don't bother to put work in to gather the metric.

And probably one of the most important principles, lastly, is that metrics are comparative. I spend a lot of time looking at the metrics and going, oh, lead time is important. It turns out the metric of lead time is not important. It's lead time or average lead time for a release, because it always has to be comparative to something.

You can look at any metric, and if you can't compare it to that same metric at another time, it is irrelevant. For example, let's look at velocity. Velocity is, I talked about at the beginning, it doesn't matter whether my velocity is 23 and yours is 62. What matters is what was my velocity last week and is my velocity now. It has to be comparative.

If you're not taking your metrics and examining them in a way that's comparative to the past, they are not important. They have to be comparative. And when you choose the metric, choose the units and choose what you're going to compare them with, because that's key.

Here's the DevOps paper. For those of you who can't remember the URL or take a picture of it, if you go to the speakers page, DevOps.io/speakers, and go down to the very bottom, there's a forums picture. You click on that, and you'll see all the white papers. There's a quick link to them if you need it.

One of the things that we worked hard on in the white paper is that we worked hard to come up with scenarios that people had experienced, real-world scenarios. And out of those, we talked about the metrics that were useful out of those scenarios. So we talked about the scenario of being bogged down by too many alerts and what the cost of that was and how to transition from too many alerts to alerts that were meaningful.

There's wonderful authors. And by the way, we'd love to get your feedback. We would really like to hear if you've got anything to add, if you want to become a participant, we'd love to hear from you. That's a start. We consider that a V1 draft. So feel free to jump in, and we'll add your name to the list.

Some takeaways. Metrics create predictability. They focus your strategy. I'll let you read it. I hate reading slides.

Questions?

Q&A

Tomorrow at 1:00?

Yes. Sorry. Thank you. There is an unconference tomorrow at 1:00. Downstairs, Dominica? Where is it?

It's on the board, so. Right outside the ballroom. Outside the ballroom. Thank you, Sam. Appreciate that.

Any other questions? Yes, at the back or right there. Hi.

Q: Just regarding metrics bringing forward sort of bad behavior or changing how people behave. For example, capturing bugs or things like that, or technical debt. So when you speak about transparency metrics, should you make all of that stuff transparent, or should you not?

A: So the question was, is there stuff I shouldn't make transparent? For example, the number of bugs that I'm somewhat embarrassed about. And my answer is yes, you should, because that's reality. And you should go ahead and expose that reality, and that will still force you or cause you to move towards excellence, right? It's better to expose things the way they are than pretend that they're not there.

Q: Right. Doesn't change the reality. I think what I mean is, what it tends to do then is change how developers then do things. For example, they won't record technical debt, or they won't say, "This is a bug," because I know the acceptance criteria doesn't mention it. They will blame-shift to the BAs. So, does that make sense?

A: Yes, I understand the question. So the question is, hey, how much do we game this? Because we want to do it towards, hey, what we desire, what makes us look better.

My question is that, first of all, I actually think it behooves the developer to talk about reality because presumably the developer is there for a long time, and the more reality you talk about, the more predictable where they're going is going to be. So I strongly urge people towards reality and not to hide those.

The other thing is because they're comparative to what they were before, you may be able to game them for a short amount of time, but eventually, a trend is going to emerge. So even though you may game them, and you may be biased one way or the other, QA may be biased one way, a developer is biased the other way, that will still over time trend towards reality because you'll end up with more bugs, the bugs will show up more, or the bugs will show up later, and they're still going to appear.

So by doing it and exposing that reality, the trend over time will expose what's going on, even if you game in the beginning.

Okay, and with that...

Thank you, Dominica. Yeah. Sorry, we got the next talk happening in five minutes. So...