Damon Edwards

I'm Damon Edwards. The title of my talk is about the last mile problem. If you aren't familiar with it, it's a concept that came out of the telco industry, meaning the last mile to the customer. It's become an economic concept for, you've done all this work, you've built up all this value, how do you go that last step to realizing that value? And we're finding out more and more that solving the problems of operations, where I come from, is actually the last mile. That's the next thing we have to do: what happens after deployment.

So I'm Damon Edwards. Gene likes to refer to me, Gene Kim, our host, as an accidental analyst. I've gotten really lucky in my career. I get to see inside a lot of companies, high flyers, not so high flyers, large enterprises, small enterprises. And through my work, currently I'm one of the co-founders of Rundeck. We make operations tools, rundeck.com. You can learn all about those.

I was previously the managing director of DTO Solutions. We did a lot of DevOps and operations improvement work for large enterprises. Our trick was really focusing on helping translate what you see on the conference stage as to what works at scale. And then I was one of the first people to bring the DevOpsDays conferences to the U.S. I've been involved since the early days. I actually sent the first email to get Gene involved with the DevOps movement. We do a podcast called DevOps Cafe with the esteemed John Willis. Anybody a listener out there? Some people? A few. All right, fans. Great. We're on iTunes. You can watch that. We just started another batch of episodes.

I'm going to start by saying developers have had this unfair advantage, right? Which is, if you think about the lifecycle, devs had almost 18 years, 19 years now, of Agile seeping into their brains, right? Whether or not they were doing Agile, it's been in the tools, it's been in the books, it's been in the language. This idea of continuous flow, working in small batches, fast feedback loops, has really been seeping into the development side of the house.

And if you think about operations, what was the last major movement in operations? It was ITIL, right? So since 1989, that's been sinking into the operations side of the house. So dev has had this unfair advantage when it comes to this DevOps movement because, as I said, it's been leaking into their thoughts.

So now here we are in 2018, right? And operations, they're in this really tough position. On one side we're being told, "Go faster. Open things up. Go, go, go. Deploy, deploy, deploy," right? And it's all the DevOps, the digital transformations, and that's giving pressure from one side. And then you've got the other side, which is coming from often the same business folks: "Lock it down. Don't be the next hack. Don't be the next breach. The bad guys are out there. We have to lock down what we're doing. This is too important."

So operations are being squeezed between these two forces. And again, the last thing that they have on their mind is how they were all built and constructed during the ITIL phase, during the outsourcing craze, right? So operations has to catch up to this way of thinking. So let's see what that looks like. Hopefully some people will see parts of themselves in this, or hopefully not.

So it's a true story. And anybody who knows me knows I like telling stories.

It's about this company. We'll leave the names out to protect the not so innocent. But they did a lot of talks. They worked on a lot of transformation, a lot of activity. People were fired up, right? They talk about their digital work, their Agile work, their DevOps work, their new SRE program they've got. On the tool side, they're spinning up Docker, and it's all in the cloud, and they got Kubernetes and microservices, and everyone's like, "Wow, these guys are awesome. I want to work there," right? This is the coolest thing I've ever seen.

But nobody was talking about what happened after deployment, right? Before deployment, it's all cool, it's modern. Deployment was this mirage, right? And then when you finally burst through it, it's like 2002 all over again. So let's talk about that.

So all this really happened, right? It's just another Tuesday. If you can put your mind into your average Tuesday at work, you have the NOC team here. They're looking at their monitoring tools. They start to see some lights, and they're like, "Hey, there's been some intermittent problems with these services over the last week. Is this the same thing?" They're like, "I don't know. Could be. Maybe."

Next thing you know, phone rings, business manager, obviously, is running while he's talking on the phone, saying there's a customer issue, right? So the NOC springs into action, says, "Escalate," because that's what they're trained to do. So Bob becomes the incident manager here. And Bob says, "Okay, first thing I got to do here, open a ticket. I got to get the bridge call going. I got to get my app-specific SREs for my new SRE program together."

And they spring into action, and they get into their try-this-then-try-that loop. Of course, it turns out they don't have access to all the production machines because some have customer data on them. So they got to get somebody from the legacy systems administration team. That's Steve. He gets involved. They're working furiously. And of course, the business manager finds their way onto the bridge call. So they start asking questions. "Is it done yet? Is it done yet?"

And we move on, and finally we come to the point, a couple of hours later, "Hey, I think there's a problem with this Foo service." They ask the Foo SRE, "Hey, can you fix this?" They're like, "Well, we're moving fast. This is something new. I'm not sure what exactly it is." So okay, we've got to escalate up to the lead Foo developer.

And that's Karen. But of course, Karen works in a different building, got better offices. She's sitting there with her earphones on, drinking her tea on her mad dash to finish her work, totally ignoring her email, totally ignoring her chat. Ding, ding, things are going off. Then finally, someone knocks on her door, and it's like, "Hey, Karen, did you see that ticket?" She's like, "Okay, fine, I'll go and take a look."

So she logs in. She's like, "Well, I need some more logs. I don't have enough logs." So she updates the ticket, but she knows that's going to take forever to get a response. So luckily, Karen knows how to find her way into the chat that the SREs hang out in and says, "Hey, I need these logs. Can somebody help me?"

So of course, they come back with some logs. How many think it was the correct logs on the first time? Yeah, exactly. So they go around and around and around, going through this chat of, "No, not that one, not that one." Finally, Karen goes, "Ah, I got it. So who restarted all these services? Because whatever happened last time..." These are all containers, by the way. Supposed to be the new coolness. "What happened is the wrong environment variables, so this entire service pool needs to be restarted."

Well, the production container infrastructure is managed by the middleware team because it sort of seemed logical. So they update the ticket and say, "Hey, we need this quick restart. We've got to restart all the services." The middleware manager calls, says, "Are you crazy? This is the middle of the day. This could impact customers. I need business approval."

So okay, now it's 2:30 in the afternoon. And if you've noticed, we've got this little thing down here, these people. In the ticket system world, we call this the context wagon, right? Which is, even if you aren't actively working on that ticket, it's occupying a little bit of your brain. And as that ticket just keeps bouncing around, it just keeps filling up a little more of everybody's context. We call it the context wagon, right? Everybody's piling in.

So we escalate up to the SVP of the business, who tries very hard to stay in touch with what's going on, but it's Susan. Susan's talking to customers. She's really five, six degrees away from the keyboard and says, "Hey, what's going on here?" The chief of staff gets her VPs together, or on the phone, and says, "Hey, can we do this restart in the middle of the day?" And the VPs are four levels away from the keyboard, say, "Yeah, it's a restart, right? How bad can it be? Sure, go ahead and do it." Ding, restart approved.

So now it goes back to the middleware manager who says, "All right, well, who knows these services the best? Oh, that's Ellen. Well, where's Ellen? Oh, she just left for the airport. She's got to fly off to Europe to help with the new launch. Oh, crap. Okay, so who have we got? Scott. Scott started two months ago."

Scott's like, "I guess I could figure it out." So Scott goes dumpster diving, or I guess rubbish bin diving, as you would call it over here. Into the SharePoint server he goes, pulling up wikis and Word documents and trying to figure out what's going on. Says, "Okay, I think I got a handle on this. I can restart this stuff."

So he goes into the console, starts changing things. Looking good, looking good, looking good. "Huh, what's this Bar service?" Says, "Waiting for Acme service. Well, I'll just wait. Hopefully, that's going to take care of itself."

Ten minutes later, Acme startup failed. "Oh, God. What's going on here?" So now Scott's like, "Oh, jeez. Everybody's looking at me. This is getting bad, this is getting bad. My feet are on fire. What's happening?"

So then Scott goes, "Okay, Bar app startup timed out." Error says, "I can't talk to Acme." I don't know what Acme service is. If I look on the other end of the network, I can see it. What's going on here? Escalate it to the Bar SRE. That's Linda. And by now, everyone's looking. Our context wagon's getting bigger.

Says, "Yeah, it looks like the Bar connection's being blocked to Acme. It's a network problem." So we throw it off to the network team and also to the Bar lead dev. And the Bar lead dev goes, "Hey, great, I can comment out that check." Oh, forgot to say, the reason why it's failing is, following DevOps best practices, they put a pre-flight environment check into that service, and it needs to talk to another service. So it can't talk to that service, so the whole thing's failing.

The Bar lead dev says, "Hey, I can comment out that test, but my CD pipeline only goes to QA, so we're going to have to get the change management team involved." It's like, well, we can't go that route, so let's talk to the network team to fix that connection.

And so at this point, little side interlude here, the business managers are calling the middleware manager saying, "What's going on?" And of course, Melissa, the middleware manager, in an epic bout of finger-pointing goes, "It's the network. Your network's broken."

So the business managers then call the network VPs and are like, "What's going on here? Your network is broken." They say, "Whoa, it's okay. We're working on it." Unbeknownst to them, there's a different network incident that is going on. So the VPs are calling the network people saying, "You stay on that other incident because this is our top priority," thinking they're talking about the other incident, right? A lot of unintended consequences.

So luckily, Scott from the middleware team had some beers recently with Carlos because he was new. Scott was starting new at the company and said, "Hey, remember me? We had beers. Can you do me a favor? Get someone to look at this."

So we get a network SRE involved and says, "Hey, the firewall's blocking traffic. You got to take it up with the firewall team." So Freddy, the firewall engineer, pops up and says, "No way. We only change the firewall on Thursdays. It's Tuesday. It's somebody else's problem."

They go, "No, no, no, really, it's the firewall. You have to look at this." So now it's 8:00 at night, and he says, "Okay, yeah, there was a rule change last Thursday that would disable the connection between these two services because they don't need to talk to each other anymore, but no one told the people who put the pre-flight check in, so now that's blocked."

Says, "Well, hey, can we change that?" And of course, he says, "Sure. On next Thursday, we'll change it for you." And everyone's like, "Come on, this has got to go now." So they escalate it up to Nicole and the NetSec team, and she's like, "Whoa, this is production, right? So now we need three out of five people on the CAB to approve a production change."

And luckily, the chief of staff was on the call and says, "Hey, I'm going to call Susan." Next thing you know, ding, firewall rule emergency changed. Susan is the SVP.

So our context wagon is just massive at this point. It's 9:30 at night. This started at 9:30 in the morning. Finally, they get the three people together that can actually fix this. They go through it, they fix it all, and they say, "Well, we think we fixed it."

And they're like, "What do you mean you think?" It's like, well, we don't really trust people to check their own work here, so you have to get the customer engagement team to check the APIs to make sure things are working. Right. Well, how do we do that? Well, it's Varsha. Where's Varsha? It's 11:00 at night, it's her birthday, so she's not there.

So finally, we get her on the phone. She comes home, wah, wah, right? And checks, does the magic that we won't let anybody else do and says, "This looks good. Service is restarted." Okay, 11:30 at night, lights look green. I guess we're good.

Everybody excited, right? And then the kicker. Next morning, Susan, emergency meeting, right? Whose fault is this? What approvals and what extra approvals and additional steps can we put into place so this never happens again, right? Thinking that's going to be the problem.

So that sound familiar to anybody? Maybe? Just a little bit. Thank you. Thanks.

And then later, inevitably, what comes up is this meeting that says, "Hey, look, we've invested in cloud. We've invested in Agile. We've invested in DevOps, containers. Now it's time to have a serious conversation. Why do things take too long and still cost too much? What's going on?"

And some organizations are realizing that, hey, we largely have not touched ops. We made this DevOps thing a dev-towards-ops thing, and we forgot about the ops-towards-dev.

So what ends up happening in these situations is companies end up chasing the symptoms. So they follow the conventional wisdom, like, "Oh, we need better tools." Because remember, Ansible's going to solve it, and Kubernetes is going to solve it next year. And remember three years ago, it was Chef was going to solve it, and two years before that, Puppet was going to solve it. So it's always this chase of the tools are going to solve this problem. They never do.

And then the next one is, we need more people, which we all know in the room that's just a non-starter. We're not getting more people. We need to make our people more efficient, so we got to stop blaming that. Or worse, we'll just outsource, try to add more contractors to the mix, which isn't going to really solve the problem.

I love this one, because this is, we need more discipline and attention to detail. This is like telling developers, "Stop writing bugs." It's ridiculous on its face, but for some reason, looking towards operations, people love to be like, "Why are you so bad at this? Why can't you be more disciplined and attention to detail?"

And another favorite, we need more change reviews and approval. Because the people who are four degrees from the keyboard, they're going to know how to stop this problem.

So fundamentally, I think the companies that are successful at this, they're really challenging the conventional wisdom about operations work, saying, "Let's stop fighting these symptoms and fundamentally look at how we do things." So there's these four forces that undermine operations that I see consistently. I call them the four horsemen of the operations apocalypse: silos, queues, toil, and low trust. So I'll talk about each of those quickly.

So silos. People think of silos as, oh, it's in a silo because we're on a different team. That's not really the case. What makes something in a silo is, how are you working?

So think about you're on one team or you're solo. You've got your own backlog, your own context you're working in. You've got your own tools. You've got your own priorities that come down from a single management chain, and you can get things done. But the problem is, especially in the enterprise, you always need something from somebody else. Nothing lives in isolation, and that's when it becomes a silo, because there's somebody else, somewhere else, that either needs something from you or you need something from them, and they have their own backlog, their own context, their own tools, their own priorities, and that's when things start to really pull apart.

And you get these mismatches. So these context mismatches. We're thinking about things differently. Different process mismatches. Tooling mismatches. And also just capacity mismatches. We have 1,200 people in operations, but only 12 of them know anything about the network. It's a mismatch of capacity.

So these silos interfere with these feedback loops. I'm sure you've all seen this. This is the Three Ways from The Phoenix Project and The DevOps Handbook. But you could think of this as really anywhere in the organization has to cross those different boundaries where there's a producer of something, a consumer of something, and these silos and these disconnects are blocking the flow, they're blocking the feedback loops, and we know how that causes problems.

And then people start to turn inward. And what ends up happening is the silos become these functional, we call them functional labor pools. Where we've got the network team here, we've got the firewall team here, we've got the storage team, we've got the Windows system admins, we've got the Linux system admins, and they all start optimizing for themselves because they have their own context, their own backlog, their own priorities. And they turn inward, and they start to really become these groups of functional specialists that get a ticket, show up, do something, and then leave, make a lot of snowflakes. We have a lot of disconnects and all those problems between them.

And then the primary management idea around that becomes protecting capacity, because you're getting overrun by requests. So you start to become more and more insular and more and more walled off.

So silos are a problem. And so we have this brilliant conventional wisdom: how do we cover for these disconnects? How do we compensate for these disconnects between these silos? Well, we put this lovely thing called the ticket queue in place.

And we all know how well that goes. You fill something out, it's not quite right, it gets bounced back to you. You got to fill it out again. It comes back, it says it's the wrong thing you wanted, or you got to wait in line till next Thursday for the firewall change. It's a lot of thrash from these queues, and it turns out there's a lot of science behind this.

So I took this from Don Reinertsen, who wrote this famous book, The Principles of Product Development Flow. And it really goes through the science. It's real. We know in the manufacturing world and the product development world, even in the development world, that queues create all these economic costs: longer cycle times, increased risk, more variability, because people are always coming, things are one at a time, more overhead, you got to manage these queues, lower quality, and also less motivation. Just intrinsically, when you get slow feedback on things, people become disconnected and unmotivated in their work.

So we know that queues are expensive, but yet we inject them everywhere into our operations work.

When queues also really kind of... We've been spending all this time thinking about the value stream, thinking horizontally, understanding the system. Yet, when we break it all up in best practices into small batches, small tickets, we're really obliterating or disintegrating all that context and this value stream awareness that we've been trying to build in these organizations.

So tickets are also snowflake makers. The idea of these siloed labor pools, that individuals are constantly getting a ticket, showing up, doing something, getting the next ticket, showing up, doing something, and all these little one-offs create these little snowflakes. And snowflakes is a term that I love because it describes where each thing, it might be technically perfect, but it's unique, like a snowflake. And you'll never be able to reproduce it.

It adds little bits of variability all through the environment. So you're trying to automate things. There's nothing worse than automation that doesn't work, is automation that is slightly off. So snowflakes are a bad thing if you're trying to get your organization standardized and under control, yet that is what tickets and silos produce.

Third thing is toil. So the important thing about toil, if folks are familiar with the SRE movement, toil, does this sound familiar to anybody? So it's a really fascinating thing that they've come up with, which is something we've always known but have a hard time defining, which is what work is of value and what work is really not adding value to the organization.

And Vivek Rau from Google, it's a long definition, but I thought it's really good. They describe toil as the kind of work that's tied to running a production service that tends to be manual, repetitive, automatable, could be automated, tactical, devoid of enduring value. That's key, right? It's not adding enduring value to the company. And it scales linearly with the service. If we have more of these things, we'd be doing more of this work over and over again.

So you can think about toil as the busy work you've got to do delivering things, but also toil in terms of things like incidents, excessive meetings. There's all kinds of things you can say that are not really adding additional enduring value to our work.

And the opposite of toil is engineering work, right? Things that really actually take human creativity, things that actually add enduring value to our organization.

So there's this balance in life with toil. It's just going to happen. We have to do it. Operations is the only place where unplanned and planned work, by definition, have to coexist. So we have this balance, and the big problem is if we have too much toil, it's almost like we have engineering work bankruptcy. We aren't going to get enough engineering work time to actually do the projects to reduce the toil and do the projects to improve the business.

So we end up where the capacity just gets eaten up by this toil, and we're effectively in this downward spiral. So it's a very important concept to think about.

And toil also blows back on development as well. So my old company, DTO Solutions, was really run by Liatrio, who's a company that kind of took over our work, did a study across from 2016 to 2017. The Liatrio guys looked at 14 different large enterprises they had worked with, all different traditional sectors. The technology organization between 900 and about 7,000. And they looked at all the development teams, or took a sampling of them, and looked at how much of their time is eaten up by operational toil.

And it turns out that it was from 28% to 63% of a development team's total time was consumed by operational toil: waiting for environments, rework due to environment differences, network issues, escalations, broken lower environments, requests for information, handoffs. All that stuff was eating up the time of being productive. So this notion of operational toil is not just an operations issue. It blows back on the rest of the organization.

And the last piece here of the four horsemen of the operations apocalypse: low trust. So where decisions are made in a traditional organization, right? The people sort of on the keyboard, they have all the context. They're there in the firefight. But yet we're deciding that these people over here, right, that they are more, five or six, are the ones that can make the actual decisions for our work.

And John Allspaw at this conference in San Francisco last year did this great thing where he said, "Hey, is this dangerous?" Everyone's like, "Oof." Kind of makes you clench up a little bit, right? And it's like, well, what if I'm telling you that's just something we run on a content cache every once in a while? Not really that dangerous.

And he says, "Is this dangerous?" I just changed the case on some text. Is that really dangerous? And he says, "Well, what if I told you that that was a health check for a load balancer?" Now it's dangerous, right?

And his big point is, it always depends. The answer is it always depends. Why? Because all knowledge work is contextual. So if all work is contextual, these are the people that have the full context. Why are we trying to make decisions happen over here? And the decisions over here are generally based on folklore or ancient history or just kind of a shot in the dark.

And so you have this low trust plus these approvals. You basically get this illusion of control. And if you want to kind of prove it to yourself, go to your ticket system, add up the total number of approvals, subtract the ones that are information radiators: "Oh, I need to be in the loop," or the CYAs, proving I followed the process. Or subtract the ones that are too removed to be judgment, too removed from the process to actually have judgment, the mostly guessing ones. And how many are you actually going to have left?

How many were the right call, and how many got rejected? I mean, we had one company we worked with, insurance company. For a 1,200-person technology organization, they had something like 25,000 distinct approvals within six months. And so how many actually ever got rejected? It was like 1%.

So beware of that. So these are the forces. So what can we do about that? We only got a few minutes left. I have kind of a lean mindset, so I like to study the problem and talk less about the solutions.

But first one is obviously get rid of as many silos as possible, right? That seems like an obvious one. Let's try to put as many people together in these cross-functional teams. But the key thing is not so much to be on the same team as that there's a shared responsibility between people for the development and operations of the service.

And to kind of prove that, you look at, there's like the Netflix model people like to refer to, where they're all on the same team, right? And there's all these little cross-functional teams. There is no central operations organization. And you compare that to Google, where they have a clear development team and an SRE team. They both make the same model of having the shared responsibility. Clear handoff requirements from development, clear error budget requirements being pushed back towards development. So they arrive at the same high quality, high velocity results, but very different organizational models.

And then you have to think about, well, okay, we're doing this. What about the cross-cutting concerns? There's things we just can't divide up. There's things we don't have enough of. So now we're right back to where we were before. We're just ticket queues, ticket queues, and more ticket queues.

So the concept that we're seeing developing and we're trying to promote and push, take from our different customers and folks we work with and sort of document this, is this Operations as a Service idea. How do we build on-demand, pull-based services of the operational capabilities that people will need to get their work done? So that way, work doesn't have to cross out of these places.

So firewall changes, environment updates, running health checks, running restarts, you name it. Any of the operational things we constantly are filling out tickets for, how do we turn as many of those as possible into these Operations as a Service? And it really works in any operations model.

And next part here is only use tickets for what they're actually good for. So tickets are good for documenting true problems and issues, exceptions. They're great at that. Or routing for necessary approvals. We have to have the approval chain. The ticket systems actually do a pretty decent job of that, but they're not general-purpose work management systems. Remember the expense and the cost of those queues. Just think about, you can actually add up all of the cost that you're injecting into your organization by driving people's day-to-day work through ticket queues.

Tickets make everybody miserable. I'd be way happier if we had a lot less of them.

So the question always comes up, well, if we're going to give this self-service, what about security? What about compliance? And this actually becomes a place to build that in. We find that the auditors actually like this better because now they have a system that they can build in least privileged access. They can build in automatic evidence collection. They can build in the two key, like the nuclear missile key thing, to say, hey, only certain things you need two people to approve it.

All of that can be built into this Operations as a Service concept, which actually gets compliance on your side. So they freak out at the idea to say, oh, we're going to have non-privileged people touching production systems. But if you explain how you're going to do it, they see it as a way for them to get back in the game.

And then shift left the ability to take action. The guys at Ticketmaster have a great case study on how they went through this. They cut their MTTR from like 40-something minutes down to like four minutes. And they cut their escalations in half. They cut their support costs by like 60%, just by figuring out how can we empower the operators, people as close to the problem as possible, or close to an idea as possible, to take action and stop the escalations, push it all as much as they can that way.

And part of that is an organizational model that actually empowers those people. And a lot of that is tooling to get that done. And we see sort of this Operations as a Service pattern as a way for the experts to define the procedures that I can give to the NOC team, I can give to the SRE team, I can give to the front-level folks to take action instead of escalating it upstage.

Reducing toil. Google wrote a couple of great books about this, but the whole key is just tracking it. Just track it. Just try to figure out. I don't mean time tracking. It can be just kind of a survey after the fact. Just figure out how much of your team's time is spent on toil, how much of it is spent on engineering work. Set a limit for what you want to have. I think for Google, it's like 50% or something like that. Just as a benchmark to say, is this team struggling or not? If it is too much toil, we're going to have to swarm to this team and figure out how to help them out of this problem.

And sometimes toil, it just happens. We're going through a new launch. We're just going to have to put in the elbow grease to get it done. And then fund the efforts to reduce toil. I said the Google SRE books are great. There's another O'Reilly book coming out, Seeking SRE. I wrote a chapter in it about doing this in the enterprise.

So to recap, because I threw a lot at you, don't forget about ops. Really challenge the conventional wisdom. Don't just chase the symptoms. Understand those four horsemen of the operations apocalypse: silos, queues, toil, low trust. Focus on removing the silos and those handoffs as much as possible. Where you absolutely can't do it through organizational design and redesign of work, leverage the Operations as a Service design pattern as much as you can. Shift left control and decision-making. That's huge.

And then learn from the SRE movement. You can think of SRE as like an operation-specific sub-conversation of the larger DevOps movement. So a lot of cool things going on there. And really focus on getting the toil in balance.

And if you want to talk about it, I love talking about this stuff. You can find me on Twitter, Damon Edwards. You can email me. At Rundeck, we wrote this kind of generic or industry-agnostic book on the Operations as a Service idea. It's kind of our first step in documenting these different companies that are putting this pattern into place to help their operations get out from under that capacity crunch.

So that's it. I'll be around. I hope you're enjoying the show, and I hope you're enjoying this new operations focus. Thank you, Gene, for letting us add that to the agenda. And that's my talk. Thank you.