John Esser

Good afternoon. Just after noon, and I'm the only thing in between you and lunch, so I'm not sure what that does for this. But anyway, we'll get through this, and I hope you're enjoying the DevOps Summit so far.

I've been involved with this conference since the beginning. I was on the inaugural program committee and have really been in the DevOps space for many years, and this has just been transformative, not only for the companies I've been involved with, but in my own personal life. And so I really can attest to the fact that this can change lives and make the workers' and the employees' lives better.

So with that, I wanted to talk today about this interesting topic. I called it "Avoiding the Infamous DevOps Team."

First of all, I most recently was with a company called AdvancedMD. We did medical software, practice management, and electronic health records, and in that space. Most recently, I shifted. We had an exit acquisition and an exit event, and so I decided to shift into consulting.

What happened was, over the years, I would do small engagements here and there with my background and expertise, and just really wanted to be able to spread this DevOps love, if you will, around a little bit more and be able to help more companies. And so I shifted into consulting, and I'm with a company called Veracity Solutions, and we look at the whole software delivery spectrum.

It was DevOpsDays Salt Lake City 2017, so last year, and I was invited to be on a panel. There were four of us on this panel, and the general audience was listening and participating. Anyway, the first question that came out was...

Let's see here. Move forward. There.

"What was your biggest regret or mistake in your DevOps transformation?"

I think I was third in the list, and so I heard some of the other comments, and it got to me. And really, as I thought through that, the answer that I gave was, "I created a DevOps team."

At that moment, I thought, "That was just the honest answer."

And this is what happened. I was shocked. Basically, the entire crowd started clapping. Several people, not everyone, stood up, but several people did stand up.

Subsequent to that, once the panel was done answering these preconceived questions, they opened it for Q&A, and pretty much the rest of the Q&A session was dominated by this. Things like, "We have a DevOps team. How do we get rid of it?" "Why did you create one in the first place?" Just all of these kinds of things.

So I just want to, if you feel anywhere similar to this about a DevOps team, would you mind applauding?

Just by a raise of hand, how many have seen this anti-pattern, this dysfunction, occur in an organization?

Yeah. Lots of hands.

Anyway, it was this reaction that I'm like, "Whoa, I just tapped into something that I really wasn't aware of."

But I had just recently joined another company, and this was accentuated for me because there was the infamous DevOps team that was in that company, and I was dealing with that issue as well. So this was even more heightened.

I don't want to belabor it. I think we all understand, but there's a couple of slides in here I just would want you to, as a math teacher I had once said, "Just burn this into your eyelids."

DevOps is a set of principles and practices and culture. All right? And they're awesome, but at the end of the day, you can't take that and embed that into a single team or group, or even a person. Even the DevOps engineer, and I understand there's a power behind that as we're looking for talent, but even that is kind of pushing the edge. And so we really need to be careful about this.

So what I want to do in this talk is look at, one, why do they get created in the first place? What are the forces that are happening? This took me some time to think about this and understand: what is going on that these keep popping up? And as I've been also consulting and working with other companies, I'm seeing this again and again. As you all raised your hands, this is a very common thing that's occurring.

Signs of a DevOps team. Obviously, there's a team over there called DevOps. They're some centralized group, some whatever. They're called DevOps. That's obviously the most...

This was an interesting thing that I'd never encountered before. I came in as the head of operations, so that last mile that Damon talked about today. And as I sat there and talked about, "Hey, we want to embrace DevOps. We want to embrace those practices and principles," there were people in that group and on that team, on my team, that said, "John, we can't do DevOps."

And I'm like, "Why? What's the..."

"Well, we don't do that. They do."

And I was just really shocked about that mentality. It had been so ingrained into the culture that that group over there did it, and we can't participate. We don't even know how to participate.

Obviously, you see silos of responsibility. Even to some extent, kingdom building. The, "Well, that's under the DevOps purview, so that's our responsibility. That's what we do," et cetera.

I think Damon today really hit it on the head when you talk about self-service platforms. What happens is that, I think, I don't know if they intend to start as self-service platforms, but they certainly don't become self-service platforms. They become teams that actually embed themselves into the value stream itself. They become a part of that flow.

In other words, perhaps maybe they're doing release engineering. They're releasing code. They're participating in some way in what's being actually created, and so they become another part of that value stream. And most of the time that you see them, they end up becoming exactly what you wanted to eliminate in the first place, which was a bottleneck. And they themselves become a bottleneck.

A case in point: this particular team that I came in and eventually I had to actually dismantle this, completely dismantle it. Just like, "Okay, this is going away." It was a pretty big pull-the-Band-Aid-off event. But it had gotten so dysfunctional, and the DevOps team was involved in the release of code principally, that they were even merging developers' code into branches. Developers were not even merging their own code into branches. That was something the DevOps team did because it was part of the release process.

Now, that is really, really messed up. I hope you can see that.

This is the two organizations, dev and ops. And what happens is that somehow through the formation of a group or a team, we get this that is put between the two groups. And things that are flowing across dev and operations have to somehow end up going through this DevOps entity. That actual flow of value.

I want to just say this: the DevOps movement, there's two components, there's no question. There's dev and ops. And the whole movement is around, how do I bring those two together? How do I streamline the flow of value across that whole continuum?

But for whatever reason, we end up creating yet another entity that somehow gets involved in this.

So why does that happen?

DevOps, if it describes a cultural context with practices and tools, et cetera, why does it happen? Well, one, the easiest thing is, "Well, we're just going to rename a team." This team maybe was doing release engineering or some other function. Maybe there's some kind of a tools team. "Well, let's rebrand it into DevOps." But we really haven't changed anything.

That's probably fairly common or reasonably common.

I think one of the most common things, though, is this. As a leader, as a transformational leader, and not understanding what's really going on here, you really have this new set of work that you've got to make happen in the organization. You've got new toolings, new capabilities. You've got change to drive through the organization.

Well, the existing organization is already doing a whole bunch of stuff. In fact, everybody's all very busy just doing what they do day to day. How many of us really are able to just take on new stuff with no thought, with no direction, with no priority? No. By definition, we're all busy. We all got something to do.

So at the end of the day, to be able to drive transformation, we've either got to create some kind of capacity in the system. We've got to free up resources to create change. We either have to free up resources, free up capacity, or we need to get new capacity. We need to augment capacity. So either we need new people. Oftentimes this requires different skills. Requires different skill sets.

And so at the end of the day, when I look at this, I look at DevOps, in a lot of ways, as what I would call innovation work. It may not be new product innovation, but it's new technical innovation within an organization. We're trying to do something different today than we did yesterday. And typically, to do innovation work requires that we have new additional resources or that we make capacity available, different from the ongoing day-to-day work of the business.

So maybe you've heard of this book or these couple of books, The Other Side of Innovation. This was kind of the first seminal work of Vijay Govindarajan, a professor, I think out of Dartmouth. But he talked about the idea that, yeah, ideas are plentiful, but once you have an idea, how do you actually make it happen? That's what he called the other side of innovation.

And there was another kind of a leadership parable called How Stella Saved the Farm that kind of illustrated this in a fable setting. But for me, the follow-on work called Beyond the Idea is really where this became more practical and more implementable for me, because he really described a framework of how you actually execute innovation within an organization. And this is where the light bulb went on for me about what is going on here.

What he describes is that there are what he called the performance engine, or the core business. There's the ongoing operations. It's the business that we're in today, what brings in the money, what brings in the cash, and what actually is what funds the innovations that we're going to do.

In other words, without the current system, without the current business, we wouldn't even have the ability to fund new innovations. So it's very critical that we do maintain that ongoing operations and that we don't disrupt that.

But then you have this non-routine, uncertain, new thing that you're trying to do, and in this case, DevOps-related activities that are uncertain. And so you need to be able to do that and integrate that into your overall production and your overall system.

So innovation leaders need to think differently about how to do innovation. And this is the model that he describes.

He describes three different styles of innovation. I'm not going to go completely in depth, but for our purposes, the two that I think are the most applicable are what is called Model S type innovation, which is simple innovation.

Fundamentally, simple innovation is that I can actually do the innovation within the ongoing operation. I've got enough slack, or I've got enough ability within the existing operations to drive change into that operations, or into the existing operational context.

The other one, however, is a much more disruptive, he calls it a custom innovation. It's basically a high-risk, non-linear type disruption or event. And if you think about it, many of the DevOps practices that we attempt to incorporate are just that for many organizations. They are non-linear. They are disruptive. They are completely outside of what we do day to day. And so really, many of them fall into this Model C category.

And so it takes a different model about how to execute a Model C. You don't execute it the same way as you do a Model S.

But what happens is this, and this is what I tried to do. This is what many try to do. Normally Model S, or simple innovations, these things that can be absorbed normally by the business, can be absorbed in just kind of normal working, leaders making some space for those things to happen. But they're just done within the normal sequence of production.

But we try to take the Model C, and we try to shove it into the performance engine, and of course, it doesn't fit. So that doesn't work.

So what do we do? We create another team. Which, by the way, is actually a viable alternative. We need to somehow do something different, and so a viable way is to create another team.

But what happens then? I'll say this: when I did create the DevOps team initially, it was to solve this problem. I didn't understand this, but it was like, "I need some additional ability. I need to get some stuff done, so let me get another team. Let me start building the tools," et cetera.

So for a while it worked, but the longer it went, the worse it became.

So the secret is not necessarily that you can't create a team. It's how long do you keep it around? Why do you keep it around? What's the future of that?

The secret here is that when you're doing custom innovations, and you end up creating a team to be able to drive that innovation, so you've got that capacity, you actually do need a team of some sort to drive that innovation.

What he would say, or what Govindarajan recommends, and what I've learned subsequent, is that the link between the core engine, the performance engine, and the innovation team have to be very tight. In fact, you can see here there's a shared aspect. There's a shared staff.

If you want to drive innovation, you don't want to completely copy the personnel or the mindset of the core system, because you'll get just another copy of what you're doing. So you need to seed the new team with new ideas and new capabilities and skills, but you also need to embed in that team a very strong link back into the performance engine, because eventually that innovation team or that dedicated team must go away.

When you start and create the innovation team, you must start it with the express understanding that it will go away within a certain amount of time.

So in all, what I just generally would recommend: first, just don't name any team a DevOps team. Whether they're an innovation team or whatever, just don't name them DevOps team, period.

Now, I have a little less heartburn about hiring people called DevOps engineers. I understand kind of how that works. But at the end of the day, just don't call it DevOps, because it's going to create overall this dysfunction of, "Oh, they do it, we don't," et cetera.

I think there's a couple of exceptions for that. If the team you're going to create to help drive the innovation is a center of excellence or a dojo type organization where they're helping and enabling, then that may be okay. But again, at some point, you're going to have to figure out, okay, what is the ultimate end state of that group? And how does that translate into the overall organization?

If you do need to create teams around self-service, create it. Name the team, or give the team the purpose around what the service is they're creating. So if you need a continuous delivery platform, call it the continuous delivery tools or tools team, a service team. Very similar to a feature team, but just name it what it does, not the ideal of DevOps.

Think about selecting a long-term topology strategy, and I have a couple of examples of that here in a minute. A really good website to go to that talks about different team patterns and structures for implementing DevOps is found at web.devopstopologies.com. Really, really good information there. Describes the pros and cons of each model.

And as Damon, I hope... That talk of Damon Edwards today was just fabulous. But create these innovation teams, and if they are intended to be long-lived for services or platforms, those self-service platforms, then go ahead and do that.

If you need to drive DevOps innovation forward in another way other than with service platform teams, then again, plan on how are they going to get reabsorbed back into the core engine.

Lastly, I think this is probably the most key, is that cultural and behavioral changes, those things about how people are going to change their work, change their mindset. You can enable that with an external team like a dojo concept, but at the end of the day, figure out how to drive those directly into the performance engine through a simple type S execution.

Do not leave the cultural transformation to outside of the core engine. This might take more time. It's going to take more planning. But as you identify those different practices and cultural changes that need to be made, drive those directly into, using small changes and small increments, drive them directly into the performance engine, using an external team perhaps to help aid that process.

One model is, yeah, I create a DevOps, quote, "a DevOps-style team" to maybe enable this. But one pattern is that you actually set an expiration date. In other words, this thing is going to collapse and go away.

However, there is danger. Danger, Will Robinson. The problem is it may not go away. So you have to be very... This is what I call the ignorant. This is what gets created because of the forces, but people don't understand that they need to expire it.

This is what was talked about today. Again, self-service. Ops or DevOps as a platform or as a service. Actually, this chart shows it in dev, but it can live on either side. It can live in dev. It can be a part of ops, depending on what the service is. So a very effective model.

SRE, getting a lot of attention at this conference. Last year, there was some. This year, much more talk about SRE.

The fundamental idea here is that SRE really embodies a lot of the DevOps concepts. The key part of the SRE is it's more of an operational discipline. And as you've probably heard, the SRE model is, of course, around reducing toil.

But the one key aspect of the SRE model is that it promotes shared responsibility and ownership with the development team. So what you will find in a lot of organizations, especially your more traditional organization, is that developers really don't want to take on operational responsibility. They really have an aversion to that. And now, over time, you might be able to change that, but in the short term, what do you do?

You want them to take more ownership. In other words, you just don't want it to be thrown over the wall, but how do you make that happen as you evolve? And I think the SRE model is an effective model. It takes some time and work to put in place, but the SRE model pushes that responsibility back on development, saying, "Hey, we will be happy to operate this and free you up to do the things that you need to do. However, we expect that it's going to come into operations with a certain minimal standard of quality."

And so then we gain this shared responsibility and collaboration. So I think a very, very powerful model, especially in the transition from traditional to what I'd call modern operations.

And then I've already mentioned this, the evangelist or the dojo model. This group that really spans both groups, that enables and helps, in many cases, provides an immersive experience. So now I'm running the actual change through the core engine, but I'm actually supporting that through an enablement or an immersive experience for those teams.

And then ultimately, at the end of the day, we want that full collaboration between dev and operations to happen. Those are two fundamentally different entities. They have fundamentally different purposes, and that's why, again, DevOps is created to bring those two worlds together and to make them more effective.

With that, thank you.