Justin Arbuckle

Good morning. Good morning.

Can large enterprises do DevOps? Yes. Come on. Yes. Yeah. Better believe it.

As Gene said, for the last 25 years and change, let's not quibble, I've been working in large enterprises. Before joining Chef, the smallest company I worked for had 40,000 people. So being in a startup is different. For one thing, it's unusual in a large enterprise to refer to your CEO as "dude." It works pretty well in a startup.

What I'd like to do is share with you some of my experiences at GE Capital and also since with some other clients. I think what has been helpful for me is actually removing myself from that enterprise context and being able to reflect a little bit on it, and so hopefully you'll see that.

In order to do that, I'm going to tell a story, which hopefully you've all heard: Moby-Dick. Who's read Moby-Dick? It is like the greatest American novel. What it is, in a nutshell, is a story of a man seeking to capture a whale that took his leg, and that, of course, is Captain Ahab. He brings everyone else along with him in this crazed obsession that he has. In the end, it's a story of failed change, because it's a fantastic allegory of what we are all about.

In some cases, we are all thinking about DevOps as it's this thing that we have to go and capture. It's this thing that we have to go after. In other cases, where we are is we are so full of an understanding, and perhaps even a fear and bitterness, about the amount of legacy that's in our organizations, that we're chasing that white whale of DevOps with that legacy driving us and perhaps constraining us.

And so Moby-Dick is a great metaphor for how we think about change and perhaps how we caution ourselves not to do some things that Captain Ahab did.

What I'd like to do is just ask you: which are you in this picture? Are you the whale or the boat? Because if you're trying to drive DevOps in a large enterprise, in my experience you feel a lot like the boat.

This fantastic artwork, by the way, that you'll see through this presentation is used by permission from a guy called Matt Kish, and you can get his book on Amazon. It's called Moby-Dick in Pictures. He drew a picture for every single page of Moby-Dick.

So this is a wonderful quote from Moby-Dick, and this is my first cautionary tale for you. Anyone coming up here telling their stories, including me and everybody else, who says that what they have said will work for you is talking nonsense. You have to find that true way. You have to find what will work in your organization.

As Ishmael says about where Queequeg comes from, "True places are never on the map." But the key things that we're going to be talking about in the next few minutes are, fundamentally, what you have to be looking at is you have to be thinking about building a credible alliance of people that want to solve this problem with you and decide to go to sea.

So many times in GE, I think we were held back by wanting to get the perfect solution. There is no perfect solution. The number of issues and potential roadblocks that you will encounter are beyond your imagination until you're in it. You have to go to sea. You have to literally be in the whaling boat chasing the whale to fully get what it's like to be chasing the whale. And so over the next couple of slides, we're going to be talking about some of the specific issues.

That's really, I think, the first signal point. It's understanding that you have to go to sea. But what is that? What does that really mean?

In many organizations, that has different touch and feel. In our case, coming from a financial services-oriented organization, consistency was the root of that key idea that galvanized activity within the executive management, et cetera. Consistency was the root of stability. It was the root of security. It was the root of compliance. But at the end of the day, and you'll see this in a couple of slides, you don't want to have an argument that is just based on a single arm, on this idea of just consistency, and you'll see why in a second.

It's also really important within the organization to be clear about what is different. Very often I think what we try and do is we tend to go to extremes. Either it's completely different: "Guys, this is this new DevOps thing and it's a silver bullet and all of your problems are solved," versus, "We'll just do a little bit of tweaking here and there."

And we all know, having listened to the speakers today and yesterday, that in order to drive DevOps, it's culture that fundamentally keeps the organization moving, keeps that momentum going, and drives the productivity. And so it can't just be, "Oh, we're just playing around over here." But at the same time, you've got to be very careful about managing your expectations around it being able to solve everything.

This is the way I conceptualized thinking about the success of DevOps at GE and subsequently. If you think about what we used to do 15 years ago, we used to spend a lot of time talking about CRM systems that were really good at getting customer information. We used to spend a lot of time talking about product line systems that were really good at getting an efficient product line going. We used to spend a whole bunch of time thinking about how we get more efficient.

And the funny thing was, the efficiencies that we would get and benefits that we would get and build in from one system, one silo in our organization, didn't fit into the other.

If you want to know that you are doing well at DevOps, and I think that this is a great way of understanding what the whale is, it's to make a virtuous cycle of these three things. The basic idea is this: if you have a project that is driving consistency through the organization, and that's what we had at GE, a lot of consistency based on standard OS system builds, things like that, that consistency ultimately led to higher velocity. Higher velocity because developers could now start reusing and creating on-demand development and production environments without having to go through a lengthy process of creating tickets and that sort of thing.

And that then allowed us to get to the customer faster. Being able to get to the customer faster based on this common set of infrastructure defined according to a consistent set of patterns or building blocks or recipes then allowed us to make a better estimation of speed, the resources required on the infrastructure itself. And so you begin to see this virtuous cycle beginning to emerge, and that is an incredibly powerful thing in the organization when people start to see it's not just about this one thing of just compliance. It's not just about this one thing of just getting a developer to move a little bit faster in terms of velocity.

Your success in the organization, and this is really what killed it for us, your success is based on being able to get the organization to see that this is a virtuous cycle. That you're moving from consistency to speed to scale. That iteration lesson then gets baked into the next iteration and the next iteration, and that's what DevOps is. And using a definition that Gene's used a lot in the past, that's all facilitated, of course, by a lot of feedback, by driving a great deal of feedback. So people are super important.

Now, you will find this, for those who are reading along in the prescribed text, Moby-Dick. This is Father Mapple. Father Mapple in Moby-Dick does this big, very fire-and-brimstone preaching about Moby-Dick and Job and the terrors of the whale. And you will find many people in your organization who will do the same thing, who will say, "This is never going to work. Give up now. I've seen many careers, my boy or my girl, fallen by the wayside because of such crazy ideas."

But at the end of the day, what you've got to do is you have to build an alliance, an alliance of trusted people. And that alliance of trusted people should almost certainly be across business and across function. One of the things that we have to remember is that creating DevOps and enabling the velocity between Dev and Ops, closing that cycle, is really based on facilitating communication between functions that have historically been broken down.

And so what we did at GE is, because I was the sponsor, I was able to put 40 people into a room, and they were from compliance, development, infrastructure, operations, you name it. And we began to work together with a joint outcome in mind. That outcome slightly changed as we went, as we learned, but we did it together.

This idea of creating an alliance is absolutely critical to take away: that it can't just be creating a little DevOps team in the corner that just works for you, because you know what? If things fail, if things go wrong, and trust me, they will, then all of a sudden, it's the DevOps team's fault, and that'll hurt you.

So what you want is you want to create a distributed approach across the organization, try and get as many people. And you know what? It doesn't even have to be that formal. You would be amazed at the amount of people who want to be part of the solution to problems that they have every single day.

Find people with problems to solve. Find people with problems to solve, and they will give you half days, quarter days, three-quarter days. And you know what? At the beginning of your journey, that's okay. Very soon, if you're able to start showing some real benefits, and so what we did is we showed some pretty simple demos about how we could reprovision multiple nodes, 50 nodes, in a matter of seconds; how we could build three-tier architectures literally in a matter of minutes, whereas historically it had been an order of magnitude greater.

And pretty soon, people start saying, "Well, you know what? Maybe it's worthwhile making Bob be 100% on this." And that's how you begin to start changing the organization.

Again, it comes down to this idea of extremes. Is it the silver bullet that'll solve everything that requires you to have a conversation with the enterprise that basically makes it say, "My goodness, you are reinventing us," which typically creates a little bit of drag and a little bit of reluctance? Or on the other side, is it this idea of, "Oh, we're just going to tweak around here"?

Well, you know what? You want to minimize initially your exposure to the organizational antibodies, but you want to maximize the practices and the approach that you have in the team that you put together, and you want to ensure that that team shows people what they're doing as regularly as possible. In this way, you start to balance the difference between really taking on the organizational antibodies before you're really ready, but at the same time, you start driving change within the team that is beginning to produce this, and people see that change. People begin to see how people are operating, how they're behaving. They talk in the corridors, et cetera.

Now, in this audience, how many people have started on a DevOps journey? That's great. How many have got DevOps in more than 20% of your estate? However you want to...

So there's a couple of hands over there, couple of hands. That's awesome.

This slide is based on my experience, and that is in large organizations, particularly highly federated organizations, you hit this thing called the operating chasm. After the first 15, 20%, typical adoption curve stuff, after the first 15 to 20%, what happens? The organization suddenly realizes these guys that were screwing around doing a little bit of stuff, they're serious. They've now started to transform five, 10 applications. This is the real deal. They're beginning to threaten how we operate as an organization.

And people don't say that in some crazy political way. They truly believe, "Guys, we've got ITIL processes that are here to keep us safe. We have a particular approach to authentication that's here to keep us safe. We've got legacy APIs that we have to integrate to keep this business running."

But one of the critical things that you have to get over is when the organization suddenly starts throwing at you the full weight of the ecosystem, to start saying, "Well, hold on, guys. You've been solving these problems, but actually, this is the big problem that you have to solve."

You guys come across that? Where people say, "Well, it's interesting that you're able to solve this little problem, but can you solve these 100 other problems?" As if the existing environments that you're operating in can. Very often you're held to a level of judgment that far exceeds the judgment that is applied to what the rest of the organization does today.

But this operating chasm is an absolutely critical concept because if you don't plan for it in the initial team structures that you put together, if you're not thinking about this, if you're not beginning to have the conversation about ITIL, for instance, and how you think about service management, if you're not beginning to have the conversation about how you manage authentication across the organization, particularly in this fully automated world, if you're not beginning to think about how you manage and test APIs for these legacy infrastructure and legacy application environments, when all of a sudden people start coming to you with these roadblocks, you will be stuck. Couple of weeks thinking, planning, what are we going to do? And so what you have to do is you have to get that thinking in as early as possible.

One of the ways that we managed it was, one of the principles going forward was we created a full-stack application, and that went from essentially zero to deployment and iteration in six months, and that included commissioning a whole new data center. So that was in six months. Fully automated, full automation, combined pipeline of infrastructure and application as code.

And as part of that conversation, we started discussion about how we are thinking about ITIL, how we're thinking about the service management process. Where do tickets come in? So level-one tickets come in, and then everything else gets dealt with by the team that is responsible for the application. And then the resolution of that ticket then gets posted back onto, in our case, it was ServiceNow.

The whole idea, and whether that's right or wrong is irrelevant, the point is that that's a start. It's the start of a principled approach to understanding how you're going to deal with these three things: ServiceNow, authentication, and this massive ecosystem of APIs that you have to coexist with as you transform from the 15% that you've got now, most of you, where, let's face it, it's a science experiment. In a large organization that earns 100 million a year, couple of thousand nodes under management, what's that? It's a rounding error, right?

We know how transformative it can be. But the point is that what you really have to start doing, and this is a big objective for the DevOps movement, I think globally, and this is why this conference is such a fantastic thing to be happening, we have to start taking the argument to the pragmatists in the organizations. Honestly, it's not that difficult for us to convince ourselves that DevOps is a good idea because you wouldn't be here if you thought it was total nonsense.

How do we convince the pragmatists? And so that's part of what your job is. To do that, you have to discuss and interact with them on their own terms. Have these discussions early. Don't pretend to have the final answer at the beginning, but start the conversation as soon as you can.

What this all means is, what is the nature of the challenge? What Ahab does is, once he finally appears on the boat, he pins this gold sovereign to the mast and he tells anybody, "The first person to see Moby-Dick gets this gold sovereign." And of course, the whole crew is highly motivated by this, being avaricious pirates.

But what's the challenge? For us, from an organizational point of view, these points that we've made around making sure that you are beginning to identify opportunities to drive consistency, drive speed and velocity, and drive scale. Find ways of doing that. That's part of the challenge. Finding ways of working across businesses, both formally and informally, that's a challenge. Doesn't that freak you out, the whole idea of doing it informally? Don't you just want to have an organization chart just to hug, right?

But that's also a challenge, and the idea of exactly what to build. How do you manage the conversation with people so that you can start dealing with issues like this service management, et cetera, in some solution? It seems so big. But you have to be super clear about what that first go is.

And I think often what we do is, I think we often are so clear about what that first challenge is that we forget to share with our teams what the bigger picture is, because it's a long voyage. So as you are super specific and as you pin that gold sovereign up on the mast and you say, "Guys, this is what we want to do with this iteration," remember to remind them about where they're going and why they're doing it.

Something that's really, really important to bear in mind is compliance. So those of you in healthcare, those of you in financial services, and increasingly those of you in fast-moving consumable goods with the cold chain, have a whole variety of compliance-related issues.

One critical thing that we learned is separate out the idea of certification from testing. Very often what we do is we mix in the process of certifying that something is compliant according to some process. So financial services could be Dodd-Frank and healthcare could be HIPAA, whatever the case may be, but that is compliant with this process. We combine that with the day-to-day testing that we do.

Hopefully if you're doing DevOps and you're doing it in a continuous delivery kind of approach, you're doing testing really frequently and it's a continuous process. What you do not want to have is every single iteration, every single sprint demo held back by a conversation about, "Well, did you test for this component of compliance?"

What you want is you want certification of compliance at the end of the process. However, that may seem a hell of a lot similar to what we do today, where you deliver something and then the compliance guy comes in and says, "No, no, no, no, that wasn't what I was thinking."

So two things to remember. First of all, infrastructure as code is a fantastic way of getting people to specify declaratively exactly what the policy needs to be. How many of you have ever gone to a security organization within your company and said, "Can you specify exactly for me in a way that I can implement what the security policy is with respect to the Red Hat 6.2 image that we're supposed to be deploying? Which ports are shut, which are open, which agents are activated, which aren't, SELinux on and off," et cetera?

Often you get very vague responses back. Infrastructure as code, which is a core part of what we're talking about, is a fantastic way of saying to people, "You know what, guys, understand that first of all, if you can't automate it, it's not being done consistently anyway by definition." So if you can't automate it, it's not being done. Secondly, if you can automate it, well, or if you want to have it automated, you should be able to write it down in a way that a machine can understand.

And you can be using Chef, Puppet, whatever happens to be your tool of choice. Our approach, particularly when we were building the standard operating system build, what we called Secure OS, across the organization, we took a while to learn that lesson. And it's really, really important that you use the mechanism of infrastructure as code as that basis to get security people, get policy people to encapsulate very specifically exactly what they want to see.

This term whale sausage comes from a previous version of this talk that I gave, and the idea is this. The #whalesausage means if you're chasing the whale, you have to let people see how the sausage is made, how the whale sausage is made. Because once you get back to port and you've managed to catch Moby-Dick, which of course doesn't happen in the story because everyone dies, and hopefully you don't. Like I said: anti-pattern.

But if you're going to do something, one of the things that is so powerful with getting executives and compliance people involved is to have them involved in the process. And you know what? Understanding is nice, but not actually required. Have people on those weekly sprints. Have them call in. And for the first five, 10, they won't know what the hell is going on. But you know what? By 10, they begin to see the transparency that is inherent in how you build in a continuous delivery fashion, and that will really pay off when this happens.

There will be storms. There will be organizational problems, political problems. People will accuse you of all sorts of things, as you're stealing staff, changing process, goodness knows what. And having a diverse alliance of people who understand how the whale sausage is made is absolutely critical because it's so easy, if you're a bunch of guys that are sitting off in the corner, it's so easy for people to think of you as being a bunch of dilettantes that don't really understand and don't really think about what this organization should be doing.

So you want to have a broad alliance of people who have seen how things are done. And I'm sure any of you that have been part of a continuous delivery process, part of an infrastructure as code process, continuous testing, et cetera, will be aware that the amount of transparency that you had in that process, the amount of oversight through peer review, et cetera, that code goes through, makes it probably one of the most compliant and secure processes on the planet. People have to see that to believe it. They need to see it to understand.

When you do ultimately succeed, something you have to bear in mind is you need to be able to reward visibly in large organizations. Very often what happens is people don't want to reward the guys that have done this fantastic DevOps thing, and I'll tell you why. Because it's really high visibility by the time you start getting traction. Everyone wants to be part of the DevOps thing. Everyone thinks it's cool. All of a sudden, the senior executives start getting approached by a whole bunch of people who wanted to be part of the DevOps thing, but they can't be for whatever reason, and they feel bad. They feel bad that they're not on this team.

And the natural inclination from a senior executive is to say, "Well, I don't want to make the guys that are running the business feel that they're less valuable than the guys changing the business." Right? Which is an awesome idea, and that's great. But the fact of the matter is, you do have to reward. You do have to show the rest of the organization that these kinds of behaviors, the behaviors of collaboration, the behaviors of working across the business silos, the behaviors of finding solutions to these tricky problems, is the kind of behavior that you want going forward.

So if the organization is not willing to reward those, is not willing to give that serious airplay, it's going to be very, very tough once you get beyond that first 15, 20%. Because then you'll find their commitment begins to evaporate later on in the process. And I certainly think that, based on my experience, that has been an area that a great deal more effort in my space, effort on my part, needed to be put in.

This is something which I think kind of summarizes the situation. In many cases, we talk about doing infrastructure as code, and using Chef or Puppet or something like that, and you talk about automating your infrastructure. And that's fantastic that you can automate your VM images or bare metal or whatever in five, 10 minutes, but then you're still waiting, in some cases, three months for a test run, right, or for someone from the business to allow this thing to be deployed.

So how much have you really driven velocity in the organization? How much are you closing that cycle of consistency, velocity, scale? Well, you're not. But at the same time, there's a lot of you that are working on Agile, and you have continuous delivery processes, and that's fantastic. And through continuous delivery, you get a certain amount of agility, but there are limits on both.

And so full DevOps is really doing both. It's really having infrastructure and process and policy as code, and you have the benefits of all of them.

My recommendation to you is this: spread your bets. My recommendation is, don't go in and say, "This is the DevOps project. I'm going to automate this number of nodes," or, "I'm going to just do this little bit of..." Spread your bets. You have to start with one, but very quickly try and create apostles across the organization.

And what that will really do for you is those apostles across the organization, building on those little projects, building on these ideas, will essentially create the basis for the next wave, the next iteration, if you like. Sorry, I skipped a slide. The next iteration, if you like, of the project.

So Ishmael is picked up by the captains that they had met on their previous search for Moby-Dick, and you want to do that, too. You want to build a platform for the next project. Every single project needs to be building a platform for the next project. And that's a platform from an organizational point of view and from a technological point of view.

So I hope that this has been helpful. Thank you very much.