Log in to watch

Log in or create a free account to watch this video.

Log in
San Francisco 2016
Share

DevOps-as-a-Service: Towards Automating the Automation

It’s clear that DevOps is evolving – but to what end? Software maturity models show that while we’ve come far at automating infrastructure, we are only about half-way on the journey to automating DevOps itself, or what will be called software-defined DevOps.


Accenture has been operating a multi-tenant enterprise CI/CD pipeline for clients for more than 10 years in what has become DevOps-as-a-Service, and we will be share our patterns for success, as well as the pitfalls. Then we will look to the future, where history has shown that models and a higher-level of abstraction will thrive.


Whether you view evolution as a theory or fact, our corner (or curve) of the software lifecycle will be next to be automated – with profound implications for all of us.

Chapters

Full transcript

The complete talk, organized by section.

Keith Pleas

My name is Keith Pleas, and I've been with Accenture a relatively short period of time, but I think you can tell from my hair and all that I've been in the industry a while. And I'm going to mention a couple of connections that I've had in the past and how these things play with where we are today and, more particularly, where I think we're going to go in the future.

So this is not an intro to DevOps session. I'm talking about the high functioning, where we are today in our industry, and again, looking forward to where I think we're going to be in a few years when a level of automation comes into play and we can take another giant step forward. And I'm going to talk a lot about that.

This is an eye chart for anybody beyond the fifth row. And the thing I want to emphasize here is that I didn't even know until I joined the company how large it is: all of the large clients, the size and the scale of the projects that are there.

And so there's some numbers in the upper left that are kind of relevant. These are the number of people that have been trained on DevOps. We have our own internal learning system for this. And in fact, I'm going to talk about how some of those assets can be used for something equivalent in your world.

So the takeaway here is that we've been doing a number of large projects, large pipelines, CI/CD, for a number of years. In fact, that's also reflected in this timeline here.

Certainly all of the graphic elements on this, you should be familiar with. I ran into Gary last night in the lobby, and he said he had a new book, so I snapped a cover of that and put it up there. But certainly the people and the books and the events up here are relatively fresh in your mind.

This is what we think of as DevOps having kicked in relatively a few years ago. But a lot of what we're doing is basically carrying on work that did start in previous generations of software development methodologies and processes.

The one I want to mention is the far left end of this timeline, which is a little over 10 years ago, when inside of Accenture in the UK, they actually set up a shared service to instantiate and build and coach and deal with these continuous pipelines for client projects.

So that's the place where I'm going to harvest most of the lessons learned that I'm going to share throughout the next 55 minutes or so.

So I'm going to take another walk into the past. This was actually my first manager when I built a lot of things for Microsoft. I worked for Joel in 1991, and I built the Excel SDK. This is a long time ago.

Joel was a real pain in the ass.

But how many of you would consider yourselves to be developers or have a development background?

Good number. Okay.

Joel on Software, I don't know if you've run into it. He's got a blog. It's pretty interesting. He's got some very unique views on things like developers should have quiet, shared spaces. He likes private offices and so on. So he's a little bit of a maverick in that way.

He started Fog Creek, and of course, he's out there, the CEO of Stack Overflow. And I'll mention his blog in a minute.

And the reason it's important is that, again, some of these things come around, and having them allows us to examine this in context and maybe do a better job even in the next time around of applying them.

So the first thing I want to mention is his contribution to the world, something called the Joel Test. And this was the year 2000. And I was really struck by how the top three items on this list correspond to a lot of the practices that we talk about today in our DevOps world.

Now, again, this is before that word came into play, but these are things that he had devs doing. These are basically interview questions for him. This is nine years after I worked with him, but he basically boiled these down through some of his experience.

So anyway, there's a link at the bottom. I've got a half a dozen or more links, so I have a resources page at the end that has them all, so you don't have to try and scribble that down. You'll have that link.

So let's jump forward to DevOps. And this one just came to me a couple of weeks ago. My wife showed me this picture in a different context, and I just loved it because we have a number of people that say DevOps is tools, DevOps is process, DevOps is culture, and I thought this picture was wonderful.

I gave the little reveal there.

Now, I showed this to a guy last night, and he immediately saw it and said, "Well, those are patio umbrellas." And I said, "I thought they looked like druids." And then we went off on maybe a Star Wars thing.

But the idea there is that there's some mysticism involved.

All right. So just my nature from a young age is to try and build models of things and understand how they work. And I find that's very useful. If here's a model that I understand, I can put it in a new place and apply it and adjust it and so on.

So this is what I'm really doing with this talk: distilling some of my ideas around models with where we are in DevOps and where I think we're going to go.

So I'm more particularly interested at the end of this talk if you disagree with anything that I say, because that's how I'm going to learn.

I actually tell my wife this, that I listen by speaking. Now, that doesn't get me very far, but I'm saying something here. And I want to know. I want to know how it strikes you, how you might use it, how it might not work for you, and so on.

So anyway, I want to break it down into two lenses, because we do have a kind of a natural divide in our world between tools and process, and people and interactions.

In fact, those four phrases, those four words right there, does anybody recognize where that comes from? I did that deliberately. It's the preamble of the Agile Manifesto that says we value people and interactions more than tools and processes.

So you have to wear this lens of: am I caring about the people in all of this, the culture and the interactions, or am I really trying to get a cheap, easy-to-use tool set or whatever?

So I basically wanted two different lenses, and we'll look at it from a technology standpoint, and we'll look at it from a people standpoint.

In the technology, I think this theme is something that you're already familiar with: software-defined infrastructure, infrastructure as code. I hope I don't have to tell you why that's a good thing, right? There's some consistency and so on.

And the key element here is: let's apply that to our DevOps infrastructure. So that's what I'm talking about here, is software-defined DevOps, taking our artifacts, controlling them just the way we control the infrastructure where we do our build, test, and deploy. Right?

All of those same reasons for doing those apply, I think, even more in the DevOps space, and I'll talk about why.

Again, we'll talk about some of the tools and stacks. We live in a world which is everything's a mix-mix. That's just the nature of it. So we need something that works in a broad way.

In fact, I used to do a lot of work with design patterns, another phrase I'm sure you're familiar with. And a lot of people very quickly wanted implementations of them. And they would say, "That's a design pattern."

No, that's an implementation pattern.

A good architecture, good design, a good model, that can be reused in many places. So again, we want something here that works not just for one stack or one environment or, heaven forbid, one discrete set of releases of something.

We also have this enterprise lens that we have to put on top of things. And Gene had joked the other day about when security and, what was it, legal and so on get involved, you're like, "Oh, we don't want those guys." That's a common reaction.

But the reality is they're there to protect that enterprise so that the people working in that actually have a job. And when that stuff breaks down, it's a bad thing.

So we've got to wear that lens and look at this stuff, and find ways to increase, via automation, via tools, and so on, all of that security and governance that's important to an enterprise.

I'm going to mention technical debt. Is there anyone here who's not familiar with technical debt? Okay. And that was kind of deliberately designed so that nobody would raise their hand, so I didn't have to define it.

Okay. So you know what that is.

We also have that in DevOps, right? Every time you go into some template and make a setting, and it's not documented and nobody knows about it, and you say, "I'm going to come back later and build a script to do this or something," and you don't do it, you're building technical debt.

Every time you let a team spin up a new stack of tools and don't document it with any knowledge artifacts, and don't make any attempt to share that stuff, you are building more technical debt.

So I'm going to talk about DevOps technical debt.

Now, the people side of this is not new. People, whether they are devs or ops or DevOps or consultants or sales or whatever, they have a familiar set of guiding philosophy in them. Being human beings, they are what? They are opportunistic. I'm lazy. They like to reuse things. Right?

So here, we want to structure things so that they are going to fail safe. Meaning if they don't do something, if they don't follow a practice, if they don't use some guideline that we'd like them to use, by default, things will happen that will not hurt us.

So that's one lens here.

And the other one is, of course, leveraging other people. So that's, again, what we do all along, is we look for things, and we leverage.

There was another plenary session where they were talking about how lean manufacturing came from Deming and PDCA. And of course, I wasn't listening to the original part of that, but it goes back to Francis Bacon in 1620 and inductive reasoning, and you form a hypothesis and you test it, and so on.

So this is a very familiar process here.

So again, we're always looking to build and get greater scope, greater control, basically do more. Okay? And a big part of this is the open source culture. I'm going to have a couple of points on that, and I'll just leave that there for now. I'll come back to open source stuff in a minute.

So I have my new paradigm for engaging Dev and Ops. And this is a funny one because I'm hearing silos a lot. Silos are bad, right? We don't want... And of course, is it a horizontal structure? There's a variety of flavors of these silos.

Why are silos there? They are an organizational artifact. As an organization tries to, we used to say functionally decompose, you're trying to find ways to break work down, to manage it in an efficient manner.

We have, in fact, a fellow from Accenture did a talk last year about the two-pizza team, going back to Jeff Bezos's idea of that's like an optimum team size. And people would like to debate, is it one team, what kind of pizza and all that. But there's a sweet spot there.

So how do we carve things up and allow that team to do its thing without hindering other teams and so on?

So this is basically my paradigm here of using DevOps as a service.

Now, this will be anathema to the people who are being preached that DevOps should be a part of everything you do. You should just breathe in and out, and DevOps should magically flow in and out of your pores.

There's a certain amount of overhead, and there's a certain amount of skills and so on that are involved in doing that. And if everybody doesn't have to have that, one of my lenses here is we might even want to specialize that. And that's what I'm going to talk about with a shared service.

So let's go back and focus on technology first.

This is the technology adoption life cycle. This is also a thing that's been around for a while. And added onto here is Geoffrey Moore's chasm, the idea that it's... I don't want to walk off the stage here and get hurt, but crossing across that can be a transformative thing. Right? Something eventually becomes mainstream.

Many years ago, we started out with Agile. Agile kind of burbled along on the edge there, and Scrum came along, and Scrum was the thing that helped Agile cross that chasm.

And we've probably gone past that inflection point with DevOps by the fact that this is the third year for this conference, 1,300 people and so on. We're definitely not down there in the bleeding edge side of things.

Yet, we are constantly moving that new edge along. And who's moving that most aggressively? These are the companies that you're hearing about, the Netflixes and Etsys and Ubers and so on. They've built their entire business model on being on that edge. Right? And they are competing day to day with others on the edge like Amazon and its businesses.

So you first have to decide how close to that edge do you want to be. And it's a challenging place, right? It's a place where something like 90% of the stuff that you create gets thrown away. Is that going to fly in your organization? Are you going to get funding for things like that?

I'm going to say it's going to be a challenge because in the typical TDD thing of red, green, refactor, a lot of you guys were devs. When do you get time to do the refactoring? Do you take a whole sprint and say, "I'm not going to add any features and I'm just going to refactor"? That's not going to fly here.

So we have a huge push on us to be delivering that value all the time, and that's a bit of a challenge, let's say.

All right. This is yet another version of a pipeline and where a common set of IT methodologies kind of commonly fit across this. It's not a completely clean line. This is just an example of it. And I've got at the very bottom lean manufacturing because, again, that is a process that starts from an idea to delivering value out the other end.

So again, there's some grayness in here, and I probably should have had the arrows fade at either end.

So this is another term that you are almost certainly familiar with, the idea of functional and non-functional requirements. Right? We used to write specs. Specs had all of the functional things. Now we do different things, epics and stories and use cases and all that, and it's all about delivering that feature to the user. Right? That's where we get the value.

So what happened to these NFRs, non-functional requirements? I've got a list of kind of the most common ones there, and devs kind of have the idea that ops will take care of that. If I check it in and it gets out there, I'm assuming they're doing the right thing.

We know that that's not a great viewpoint, but we know that certainly is common.

So what happens is that it runs on my machine, I check it in, they do something, and it falls apart. And this is where it gets challenging.

What do you do when something falls apart? And I'm not talking necessarily about DevOps today. You have a couple ways to do that. One is you give it more privilege. Now, you know about this if you guys are devs, because as soon as you get a new machine or build a new environment, what do you do? You give yourself administrator privileges. Right?

Okay, so the devil's in the room, and how do we get that devil not out onto the production? So there's a constant battle here.

Now, one of the interesting things is I feel that most of these NFRs, which they're just the ilities, and we just kind of abstracted them away, these are the true functional domain of DevOps. DevOps has to help build all of this stuff in.

So, we don't necessarily, as DevOps professionals, get a chance to do the UI design. But certainly if something has to be reliable and resilient and all of that, that's firmly in our wheelhouse.

So we want to consider these. So I've got a few questions I want to ask you to consider about your environment in light of those NFRs.

How are you managing those continuous delivery artifacts? Do you even know what they are? Or are they just, you sign on to a web console, and you bring up a form, and you fill in some things?

Well, so the artifact's just some something that's stored somewhere. Now, some of the more mature systems, Chef, for example, they have recipes and cookbooks. Okay. I know. Those are artifacts, and maybe I can save them, and I can version them.

And I don't know if you've been doing this and working with these tools, but some of them have a fairly robust system for managing themselves. But so many of the systems that we work with don't. They just have a console. You set a bunch of settings. You save it.

It gets hard. It gets particularly hard with security settings, with basically all of the little permissions and so on.

I actually have a friend of mine, he's a CTO of a small startup in Seattle, has 25 people. They've been in there for three years, and they've got a pipeline, and it is so brittle, so dangerous for them to change things.

They're just drawing a line, setting up a new pipeline based on least privilege, which is a reasonable thing to do. And they're just going to say, "We can't figure out where everything is here. We will just start a new thing and do all of the net new here and hope we don't get killed by that."

So managing your artifacts, that's fundamentally what I'm going to be talking about here.

Are all your build pipeline settings stored in source control? If they're not stored somewhere, then how are you going to know if they changed? How are you going to know who changed them and what the settings are? I think this seems obvious, but we'll come back to it.

Can you rebuild your pipeline via automation? Now, I think that starts to get a little interesting and kind of aspirational. I'm not talking about rebuilding your entire source code control system, your entire world. But can you put together your CI/CD pipeline via automation? That's fundamentally where I'm going here.

Can you scale that pipeline? Now, think about that. If you have the ability to templatize it and automate it, we know that's one of the levers that allows us to get benefits from automation and benefits from the hyperscalabilities in the cloud. Right?

So the same thing here. How do you scale it by onboarding new projects, new companies that your large enterprises acquire, and so on? So, if you have something that's easy to apply, that's going to be an easier thing to do.

And now this goes back to some of the lessons from the DCSC. When you concentrate a certain body of practice together, one of our guidelines when doing large projects is we set up these centers of excellence, center of practice. I think you'll probably be familiar with that in your world.

And you generally would like your highest-functioning people sharing all of their best ideas, and then you're going to be leveraging. So the same thing here: how do you create new talent?

And the talent aspect of this is really big. So again, if you have a set of systems, the knowledge, and guidelines, and so on, it's a lot easier for somebody to get up to speed on your system.

Now this one, this is probably the most far out thing, is could you, if you had to, move your pipeline to a completely different stack or a completely different cloud?

Now you might say, "I don't need to do that." But think about the strength that comes from being able to do that, if you had to. If you cannot do it, then it's not an option. But if you have an option, and it's not as difficult, now you have a few more levers that you can play with.

You can do a private cloud and then migrate it to a public cloud. And this is where I'm ultimately going. You could possibly work with some shared service somewhere that helped you do this, as long as it's portable.

Being stuck in any particular repository or stack or tool or whatever, we know that in the long run, that's going to turn out to be a bit of a challenge.

So the final one is, how are you managing this DevOps technical debt? My gut feel is you didn't even know you had that until I started talking about it.

So the first enabler, now again, Accenture with large projects and large practices and so on, has a whole set of tools and practices and so on that we bring to each engagement. But I want to share a couple of these here, and these are actually public, and they are things that you could reuse. And I'm going to talk about how you might be able to reuse them.

The first one has not yet been made public, but there's a fellow here from Australia. Mirco, you want to raise your hand? There he is. He's right there. His hand's in the front.

And he has promised to work with me to get this public on the Accenture DevOps blog. He has his own blog as well. So I would expect in the next couple of weeks, this would be out there.

Now, an assessment framework and a model is, by itself, not a magic bullet. It's just a starting point.

And you're going to get the slides, I promise. And then the next one, the next slide is just... I imagine if you have an image-to-text, it's going to be a lot faster to still wait for the slides.

Okay. So we have words.

Okay. We have a plan here, a diagram on the right, which is talking about a life process cycle you're familiar with, and a couple of other lenses on top of it. So I just wanted to talk about these as being a set of domains to allow us to, again, kind of break it up and say, "How are we doing across this lens and this lens?"

With the governance, for example, continuous integration, continuous delivery. These are all mostly familiar things for you. And there's some things that you look for on the right, and again, you can't read that beyond the first half dozen rows.

But it's something that you want to start with to say, and this is a full-up model with, how do I gauge my position along here?

Now, the interesting thing is, if I took that leading to ad hoc and flipped it end to end, what would it correspond to that I've just shown you? The technology adoption life cycle, right?

That's the unicorns or decacorns out here on the right, high-functioning people right behind it, all the way down there. So it's basically just flipping that out because we want to be more to the right in this case.

So there's a number of, again, of artifacts and things that you can look at and some guidance to help you know where you are, which would help inform you of where you really should be spending some resources.

Now, I've got a humorous version of this that I want to share.

Actually, even before I do this, I want to say that you've got to take any of these things with a grain of salt, because there's a fair amount of self-assessment in here.

You know you can ask an organization, "Are you doing Agile?" And of course, they're going to say yes. But we know that there's some variation there. We also know that more than half of you think that you are above average intelligence of everybody in the room. You think you're smarter and better looking than the other average people, and we know that the average person is average.

My mom and dad used to argue about this. My mom would come home and say, "Oh, the average human is so stupid." And my dad would say, "No, the average human, by definition, is average."

She wouldn't give up. She would say, "But it's a low average, right?" They could go for hours on that.

So here's my model. How many of you like dogs?

Really? Only a couple? A couple? Oh, well, I was expecting more.

Oh, yeah. This is a Samoyed. Now, this is another learning moment here. I spend my whole life correcting other people. You think it's called a Samoyed, but it's Samoyed. Think Smiling Sammy.

And I have one, and I go to the dog park, and it kills me if somebody goes, "Is that a Samoyed?" And I have to fix that. So again, I've given you one takeaway. That's a Sammy.

Okay? That's an ad hoc, free like a puppy. What does that remind you of? Open source. Oh, that's a big part of our world.

Okay, so open source tooling, free and cute as can be. But right away, it's chewing your slippers, it's peeing, and so on. So we got to bring some structure to this.

And this is this next step, but we got to give it a little basic training, not to pee, maybe be able to walk on a leash without pulling and so on.

Consistent. Now we're getting, boy, it's off the leash there. They're sitting. That's great. Now you're really starting to get somewhere, right?

When you're optimized, well, you're running through the snow. It looks like fun to me. And of course, the absolute best ones become the breeders for everybody else. So anyway, it was a humorous example of a maturity model for dogs.

So now let's look at another enabler. Now, this one will probably be more interesting and probably more usable takeaway here.

There is a set of assets that emerged out of the shared service center where there's kind of like a minimum viable pipeline, let's call it. And it was something that was used inside of the company to train people on DevOps.

In fact, we're basically going to spin up another project for this to allow new people to come in and pick it up.

Now, I'm going to talk about, there's a couple flavors of this, and I don't want you to get hung up on any particular icon. Is it my tool or not? Think of this ADOP, this platform, as a reference or a sample, and feel free to use and modify it for your environment.

The key thing is it has tools in it for each of the elements across that pipeline. Okay?

And there's a link at the bottom, but I'm going to drill into this into a little bit more detail.

So this, I hope you appreciate all of the work that goes into automating this PowerPoint.

Okay. So when you look at it, it spins up a Docker image. We've got Nginx. We've got a bunch of tools inside of it.

And the version that is out there in open source has one, the most important aspect of everything in the open source version of ADOP, is that it's free. Right? It's all free stuff.

Now, compute isn't free, and you're going to need an AWS account or something like that to run it. But the idea here is with a very minimal investment, actually zero investment in licensing, you can get up a minimum viable pipeline and start doing your single-page apps or your proof of concepts or something like that.

Or you can train up people on a DevOps process.

Behind the scenes, it's not part of the smallest one out of the box. It runs Docker Swarm, so it gets all of those scaling things as well.

I look at you guys with your cameras. I do the same damn thing. And then I've got a lot of pictures, and I go, "What is that?" Right? It's so fuzzy.

If you have a boat and you ever take a picture of there's a dolphin going by, and then later there's gray and there's a black dot, and what was that? Airplane. Same thing.

Okay. Now, there's a lot of IP into something that is basically an additional thing in that stack that handles some of the larger sets of functionality around commercial off-the-shelf products, the SAPs and so on.

So this is not... We, Accenture, do not give away all of these cartridges. These are part of our engagements. But there is a sampler out there for building a cartridge to extend this, and I believe the Java cartridge is also out there.

All right. I'm actually going to jump forward one slide and then come back because I know you got very excited about any logo or any tool name. I guess I do, too.

So I wanted to show you one that has a little bit more fidelity to it.

So everything on the left is either part of or can be added to ADOP, and there's that dark line across there. So the stuff underneath, the Docker, Sensu, the ELK Stack, Jenkins, Git, Gatling, Gerrit, all of those guys are part of ADOP.

And if you install this thing, and I'm going to do only a very quick demo, not a very detailed one of it, you get free versions of all of that.

Now, part of the reason that they're there is that many of those things also have enterprise-licensed versions. So we have, in this service in the UK that has hundreds of clients, they're running the private SaaS version of CloudBees Jenkins, for example.

So I wouldn't get hung up on a licensing thing here, except for when you're trying to get something up and play with it. We all know that that freemium model of a tool is nice. This actually batches them all together and gets you going quicker.

I'm going to jump back one slide.

So the dedicated instance, which is the open-source project and so on, is a single-tenancy thing. So there's another kind of principle here of applying your brain so that every project doesn't have its own Jenkins pipeline, for example. You know that would be an anti-pattern, right? You want to have some shared services even in that area as well.

What else to say? It can be actually installed anywhere because it's also open source. You could port it to your private cloud if you felt like doing it.

The managed service, which is used by literally thousands of people inside the company to help deliver things to clients, is all based on just AWS. Okay?

So let me do one thing here, which is I'm going to jump to my web browser. I believe I have internet connectivity, and it takes about 20 minutes to spin up one of these stacks.

And actually, let me reauthenticate here so that when I go to do it, it'll let me.

Now, in the public space, here's a pointer to it. This is the ADOP Getting Started. That is the doc stuff off of the GitHub repo. So this will basically tell you how to get it all running.

We have an internal version because it's basically using our own shared AWS account. And I'm going to do basically the same thing here, launching this stack. And it's going to go build a little template from an S3 CloudFormation script. It's already got that guy filled in.

I've already got a version out there because I'm going to show you the one... It's like cooking shows, right? There's one in the oven, and then I'm going to pull it out. So there's that little bit, just because I don't want to spend 20 minutes on this.

So there's dash two, a username.

This is one of my favorite things. I've actually demoed this for some groups internally, and there's this requirement for a heavy password. Why might that be? It's one of my favorite things to ask.

People think about it and they go, "Oh, because you got to protect your data." And I'm going, "This is Hello World," right? Nobody's going to come in and steal Hello World. What are they going to do if they can get into your AWS account? Does anybody know what's going to happen?

No. No. They're going to start mining Bitcoin, right?

And again, they use automation. So if you check in a key or a password, GitHub actually has daemons that run across looking for this stuff. But for a long time, you wouldn't know it until you spent $10,000 on mining Bitcoin for somebody else.

So you want to be careful with these things. All right?

We have a convention where, because it's a shared space, we have at least one key telling us who created it. And I'm going to spin that guy up.

So we jump over here. Give me about five seconds or something like that. And I'm spinning up a new cloud here. Okay?

Now, the net result of this, and I'm going to go to the one that I built earlier, is that we do a variety of things. We actually build the cloud. We deploy into that cloud what that animation is, all of the pieces to build that, including the pipeline. And then we step through all of the pieces of the pipeline, build it, some variety of tests, and so on.

So this is the version that I built in my room like an hour ago. And it would be there, but I want to just kind of jump into it.

A few getting started points here, but the key thing is, if we look behind this, there's an already built version of a pipeline here, and I can just rerun this.

And this way it's configured, I have five pipelines in my workspace at once. So it's going to kick off a new one.

Okay, there's my new version of that. And I've got to go find my reference tab here.

So I usually keep... This is just for me. We get into that bumper page, and I'm always wanting to go back to it.

So the other thing I want to show you behind the scenes is some of the complexity behind Jenkins. So Jenkins itself is a configurable system, a variety of plugins, some that come with it and some that you can get somewhere else.

So if I say manage it and look inside of the plugins, these are all the ones that are in our version of it. Now you see a lot of them are red because we're going to have to upgrade this entire thing and all of its dependencies in one chunk, and that's supposed to happen next month. But this is one of our versions of kind of packaging stuff together.

The key thing is inside of any of these things is a set of processes. And the processes are templatized, and the templates have values, and the values have schemas, essentially. And it's what you might expect.

I'm going to go into the whatchamacallit, unit tests, for example, and show you what's inside the configuration for unit tests. And as this thing in a pipeline, there's just a variety of steps that we run, a variety of settings.

The more key thing is down near the bottom, what are the triggers that we do afterwards? And this runs another thing. It can also... Basically, a trigger could be anything. I can do an add build step here and go and invoke things. There's actually a few more in this. I have to kind of scroll down there. Okay?

So all of these settings, my point is, let's find a way to get some of these, which are very syntactically dense. Right? All of these settings here, I don't think you'd want to try and create the Groovy script, for example, from scratch.

So you're going to typically have some place where you store your scripts. And I'm talking about pulling all of that stuff out, putting it in a certain place, and automating it.

All right. So let me go back to my slides.

And so that's the second enabler that's out there that you can play with yourself. And again, that enabler is basically the same thing that's used inside of the shared service business inside of Accenture to help our clients.

All right. There's another element that they have, which is an organizational element, which is how they expose this inside of the company. And you probably have inside of your own organizations your own change management teams. You probably have done a variety of things like Agile transformations. So you have a lot of guidance here.

It just happens in this case, we organize it all under one kind of portal and lens for people. Because we're typically providing guidance about how to use this stuff, not just, "Here's a URL, go to town on it." Okay?

So let's talk about these trade-offs.

With a 10-plus year history of shared services, I asked Mark Rendell, who kind of runs that, and he's the guy that spoke last year about two-pizza teams, I asked him, "What have you guys taken away?" Not from the standpoint of necessarily running it for a particular client, but running it as a service with multiple clients. What might that look like, both the good and the bad?

Well, there's three big spectrums that when you group things together, you have the ability to move, I guess, the entire practice along the spectrum.

So there's three spectrums I list there. For standardization, if by definition everybody is using the same set of tools and templates and running it on the same thing, that's a pretty high degree of standardization. Now, you may not value that. I'll just say okay.

But putting that in one basket, it allows you to control that from a central place. This is very exciting to some people inside of an organization. It's anathema, of course, to, well, Alan Shimel, DevOps.com likes to call the ripped T-shirt DevOps guys, which is, you're impinging on my freedom if you dictate anything to me.

The next one is governance. Again, we can set up some policies on top of that.

And the third one is some of the processes involved.

So these are the lenses that, or the levers that we get to move everything as a batch. And that's a positive thing because it gives you standardization versus the fracturing and the, I guess, proliferation of other things.

So consolidating and simplifying a tool portfolio, the biggest advantage to this is you get a tremendous economy of scale with licensing. You must know this whenever you go to license something. If you have 20 teams doing something versus an enterprise license and so on, it's going to be a lot more cost-effective.

I mentioned the maturity assessment and coaching the teams only because that is something that they found that they had to do to make their teams successful.

And this is a key thing here: in our use of it, the teams are actually Accenture teams, not client teams. This is not a service that is commercially available for people to use, but it's a model that I'd like you guys to consider creating inside of your own organization to take advantage of some of these commonalities.

One of the big challenges, then, is getting feedback from people, right? Because the ultimate users are, in fact, a step removed. Inside of your enterprise, you wouldn't have a multi-hop process for doing that.

Geolocation for them, and it's turned into a bit of a challenge, is they're in the UK, and so that's Europe and a certain set of people that are located within a few hours' time zone one way or the other.

But you get to here, here in the US, and that's a bit of a challenge to work with Europe, with let's say US users, if the people in Europe want to have nominally 9:00 to 5:00 real desk jobs. So we've set up another version of that here in the US and Florida to help support that.

So what you would find is if you have teams that would be using this inside of your organization and they are spaced around the globe, that you might, in fact, need more than one service center because, again, people need access to things during their work hours.

The last one I want to mention here that is a particular challenge for them is that they're integrating with a lot of things like source code control systems that are inside private data centers, so they have to set up a lot of VPN tunnels.

And one of my favorite things with dealing with data centers, you go and ask an infrastructure architect, "What options are available for getting into it?" They would say, "I can't tell you that. That's a security violation for me to even document that and tell you that."

So it's a real challenge sometimes to tunnel into these systems and hook them up.

So how might you use this ADOP? I gave you an example, a place to go look at it. It's got a set of things.

Let's go see how my build is doing, by the way.

Go back into here and look at my stacks. This guy is almost done. Almost done with building the cloud and putting everything in it. It then kicks off the building the pipeline, then that's a separate process. Starts out at about 10 minutes, and then rebuilds are in the five-minute range.

In fact, let me go back over to... I had that guy running somewhere, didn't I? My reference app.

Hit refresh here. And this guy is well along. It's already finished rebuilding when I triggered off that step earlier in that pipeline. So this is in the five-minute range to get done. There are actually some times on there that are seconds to minutes and so on in each one of those steps.

So again, how might you use this?

We use it internally to help educate people on a consistent set of tools.

We have also been seeing an uptake of this for building, like I said, MVP, minimum viable product, proof of concept type stuff. Right? A lot of the engagements tend to be things like six months and nine months and so on, but clients are saying, "I need to see some value quickly," like a week or a sprint or something like that.

So having something that you can stand up with relatively little effort to help deliver that, and then, of course, you are adding some technical debt. If that's not your ultimate thing, you're going to have to move it somewhere else or say, "I'm going to absorb that of having it run in its own stack."

But anyway, that's a trade-off that you'd have to make.

And the third one is just use it as an example, as a reference of how to put something together and automate its creation. If you're using some other tool set, TeamCity, for example.

The other reason we have a version of this that runs in the Microsoft world, but obviously, the licenses there are not free, so that cannot be open source. But if you have an enterprise agreement with Microsoft, then you might want to build it on TFS and so on, and their testing tools and all of that.

So I don't want you to get hung up on an individual tool. I want you to think of it as a jump-starter and something that you could customize for your own environment. Okay? And a first tool for you to have in a shared services model if you wanted to set up a service center inside your organization.

Let's talk about the people. I've got only a few minutes left.

This I thought was interesting. So I'm sharing things with you, right? I'm standing here on the stage, and I'm sharing things with you. I'm competing with you, too. Every moment, we are all competing for the best talent out there. And that's what this was all about.

I didn't know when I put this in my deck that Capital One was also a presenter here and so on. I was at Jenkins World, and I said, "Well, what are you doing here? What's going on here?" And they, "We're showing a couple of open source projects."

So in the world of DevOps, in the world of open source, these are like honeypots. This is how you attract people that want to be a part of that culture. You have a thing, they can come up, they can contribute. It's a great onboarding for finding people that have an interest in doing the stuff that you're doing.

And so the thing I thought was interesting was it's entirely a jobs-driven thing. It says jobs@capitalone.com.

Now, I don't know if there's anybody from Capital One in this room. I thought it was a brilliant idea. You see somebody be a sponsor here as a way of getting ahead of everybody else in their space and so on, for that talent.

So that's the next point here, is there are people that are great at getting the best talent. The apocryphal example is Google, but a few other companies out there. You know they're getting the best talent.

So it's a challenge to find talent.

So I've heard, in the last few weeks, several people ask, "How much can we save by going to DevOps? How many people can we cut?" And I'm like, "Oh, man. That's not the right thing."

People are very expensive to get. Now, you could say, I could free them up and use them for something else. Is there something that has higher value? So that's the point of this.

If I can automate, and if I can free up some resources, then I don't have to go and buy a booth at a conference to try and attract new ones. That's my point.

So I've got two quick slides on people-related things, though the first bullet on here wasn't really a people-related thing. I just didn't want to create another slide to hold it.

But again, anything that you can reuse, that you can leverage and get more value out of, clearly is going to be something to aspire to. That's the point of everything we do with automation, is to write it once and use it many times.

And the same thing here of building a set of assets that can be reused. They're relatively expensive to build sometimes, but the reuse of them is often not free, but much less expensive.

So that was not a people thing.

But the other one is sharing the knowledge between teams.

So this could work in a world where you have your two-pizza team, and you're doing your own DevOps. That's certainly the way any of the internet startups do. They don't have even a separate ops team. So their devs are doing it and owning it all the way through.

But as these things scale up, you need to find ways to batch stuff together and allow them to proceed and not be blocked by things. And this seems to me to be a very logical set of NFRs that we can package as a service.

Okay? So what we get then is we concentrate our knowledge in one place.

Instead of having a user group inside of a company where it's, or a Yammer channel or something like that, where there's 20 people spread all over the globe, these people are more likely to be concentrated together.

Certainly a practice that's well-regarded in the development space, pair programming, team rooms, and all of that. You get some significant benefits from co-locating people.

You also get a lot of flexibility here when they can more easily shift to another project to help somebody out. Again, co-location there gives a lot more cross-pollination, let's say, between those things.

It also helps people cover for other people when they have lives and they want to take vacations.

When you have one guy that understands, or gal or whatever, Samoyed, in your organization that understands your DevOps pipeline, and that guy, gal, gag, wants to take a vacation, what do you do? You say, "Well, we'll just cross our fingers, and we won't touch anything until you get back."

Okay? That's not a recipe for success, but it's certainly a strategy. Not a good one.

Actually, that same company where my friend's setting up a pipeline, they have an unlimited vacation policy. You know what that means. The point of that is when you leave the company, you don't get paid for it. So there's the other side of that.

But it also means they won't let him take a week off because he is the only one that understands this. So it's a challenge.

This was also a big deal for them, being able to manage themselves. Now, this is probably a little different because it's in the context of Accenture, where most of the people are on the road at client sites doing engagements, and they roll off of that, and they can go do something else.

So they really aren't directed and don't have the sense of self-organization and self-direction. This did allow this team to do that. It was a beautiful thing. Also inside of your organizations, you're more likely to be able to replicate that set of values as well.

Now, the next bullet here is ironic. He said, "Beware of the HIPPO."

Now, I always aspired to be the highest-paid person in the room, and I don't think there's anybody here that's the lowest paid in your organization. If so, I'm sorry about that, and we're hiring.

But there's a principle there of allowing everybody to have a voice.

And it's been the experience inside of the DCSC that you want to suppress the ability... And boy, I used to be an enterprise architect, all these things. I love telling people what to do.

But his point was in an egalitarian world like this, where you're trying to get the best from everyone, just be aware of that as a possible anti-pattern.

Doesn't mean you shouldn't still try to be the highest-paid person. It just means you've got to listen to everyone else.

Again, they got a lot of benefit from cross-training. This is another Agile principle in the Agile Manifesto of cross-functional teams. And again, a team that's co-located and working and so on, they get a lot better at covering for each other.

And then he mentioned also this trust versus empowerment, and I've got a link to John Clapham's blog post about this.

Imagine a developer. You trust the developer, and if the developer breaks the build, what happens? They get a dunce cap. There's public shaming, and it's not that bad.

What if a DevOps person breaks the build? Not so good, right?

So you want to trust people to do stuff, but the cost of a violation of that really suppresses the fear of... All of those reasons why we want to do continuous delivery so we can release with confidence and so on, all of those same things apply to this DevOps world, where, again, everything is a one-off setting somewhere, and people are very hesitant to touch stuff.

So this article is interesting. If you get the slides and click through and take a look, he actually says that trusting somebody is a danger because you're inflicting that on them. If they make a mistake, they're going to be traumatized by it.

That was an interesting point of view.

Everybody would like to work with their coworkers. You guys all get to go to lunch together and so on, so there's a number of good things here.

But like anything else, a great team can become very insular, right? And there's like, "Oh, let's not add any new blood. This thing is spinning along just fine." And there's a challenge between learning new stuff versus seeing something ship.

For example, a lot of value in saying, "I shipped that," right? We all like that. That's the payback for being a developer is getting something shipped.

So these are some items that he wanted me to share with you guys.

So where do we go from here? I've got four minutes to go, and we're finally getting to the point of this, which is we've got to find a way to automate this.

Now, this is another anti-pattern to say, "You should," right? If my wife says, "You should," I say, "Wait a minute." And I'm saying you should find a way to put some structure on your DevOps process and artifacts and so on.

And that's part of the first thing, is you have some definitions, you have some standardization, and then you can automate. You can't automate it if you don't have a taxonomy and a system in place.

So that's why there's been this kind of mantra of define it, standardize it, and then automate it.

Now, there's two reasons why you want to do that. Even if you're running in place today and doing okay, that's not going to be good enough next year, two years, five years, and so on. You know that.

In fact, this quote, this is actually a quote here, it's the second line here, was from a book, In Search of Clusters. The cluster server, I don't know if you guys were around when that first came out. The guy had a really good idea in there.

What do you do when you hit the wall? You're running at 100%, 8:00 to 5:00, 9:00 to 5:00, whatever your work hours are, 5:00 to 5:00, some of you. And what do you do when you hit the wall?

Well, you run a little more. I'll work a little more overtime. You know that's not sustainable.

Work smarter. Okay, I'm going to go to a conference and get some ideas and come back. And what's that going to do? That's going to allow you to scale 10%, 20%, 30%.

And the ultimate answer is always get help, right? And either possibly get more people or any way to reuse stuff. And automation here is getting help. It's expensive to create, but then it allows you to scale on from that.

So I've always come back with that: work harder, smarter, get help. Those are really the three choices.

And it's funny, many situations have a limited set of choices, and they're all bad. You might have heard half of all marriages end in divorce. That's sad. What's the alternative? Death.

Okay, so there's not a lot of good options here. So the same thing here. You've got to pick your option, your appropriate one.

So this was harvested from a media client, and this is in a complex scenario. This is as built. It wasn't necessarily designed this way. Somebody sat down, and these are all of the tools and how they're all hooked together.

And I hope you can imagine the challenge with keeping this running as infrastructure for DevOps. And that, again, is speaking to everything is going to get more complex. There's going to be more demands and so on.

So the sooner you can get ahead of that curve by doing that standardization, the better.

So I wanted to come back and talk about those NFRs, those ilities and so on, all those great things as developers that we assume somebody else would handle.

Versioning, packaging, and dependencies. Now, they actually do a little bit of this with things like I mentioned, Chef recipes and cookbooks and so on. But just like you do this with your binary artifacts and your source code and so on, you need to consider doing that with your DevOps artifacts and batching things together and moving them along so you don't have 20 different versions of a similar pipeline, for example.

Okay? So I think that should be an obvious one.

And then there's the building of it, which I was showing with that CloudFormation template. Maybe jump back over here and hit this guy.

Oh, actually, it's already sitting there saying it was completed.

Now, we've got some daemons running in the background. If I don't log on to this thing after 20 minutes or something like that, it'll just say, "Oh, you really didn't want it." It's going to spin it down.

So Amazon's not going to help you here. You really need to control your costs in this kind of an environment. So we have a variety of tools behind the scenes. And this isn't production per se, but we scavenge these resources.

In fact, one other thing down here is we've got a Security Monkey role as well. So just like I said, you don't want to open up even your play-around environments to somebody coming in and taking them over. You need to set some guidelines on top of that.

So the Security Monkey isn't saying you have to do these things. It's always testing and probing for all of those security holes. It's one of the Simian Army that comes from Netflix, like the Chaos Monkey and Gorilla.

Okay. The other very important reason to manage this stuff in a consistent way is your certificates.

I don't know if you saw this, but where do you store your certificates? You probably have a variety of solutions for it. There's certainly an AWS service for it.

It's always a challenge to do roles versus it's easy to have a cert, and we just trust that everybody on the team isn't going to leak the private key. But really, in this world, in terms of automation, you can do better than that. So again, we need repositories for these things.

Now, this is where I think you could also put some pressure on all of the people in the expo hall to make their systems more open. We've certainly done that on a source code control system via Git, for example. There's been a lot of pressure to do that.

And there are some examples inside of pipelines. Jenkins, for example, pipeline as code. There's an add-in that you can do that, and it allows you to basically pull out and restore pipeline settings.

This isn't exactly the same. Most of the systems give you ability to back up. This literally would allow you to pull this stuff out into, I think it's YAML, something like that, some artifact that you can then control.

Now, this also gets exciting if you have some testers in your world: the idea of automated testing of your DevOps infrastructure. That would be really good to know before I make this change and push it through, that I'm not going to break something. Or if I do, that I can roll back to a previous version of that.

But if you're not managing those and allowing yourself to do that, you don't have that recoverability.

Now, this is my last thing, and this comes back to Joel Spolsky. He was at GeekWire, and we got together afterwards and I said, "Hey, I'm going to talk about software factories, and I loved your hammer factory, factory, factory thing."

This was 10 years ago. And he goes, "I didn't write that."

It was on his blog, but it was a guest post by this fellow, Benji Smith.

Now, this is a truly epic post. It's a guy walks into a store, he's looking for a hammer to build a spice rack, and the guy says, "Ah, you want a universal hammer?"

"No."

And then he, "We don't sell those anymore. You want a factory."

We've been down this road before. Increasing levels of abstraction, they sound great, but they have challenges.

And the biggest challenge of all of model-driven architecture and all of these tools for creating things was what? We didn't have full-fidelity round-tripping. It was code spit. You'd do it, you'd spit it, you make one change, and now I'm going to literally step off the stage. You've left the path.

You can't go back and use this model. You can't do anything. If you change it here, you're going to clobber that, because it's a one-way deal.

I think the next version of tooling that we can do around this can be smarter here and can allow us to make changes, inspect those repositories, and then update them back into our system.

It would be best if we allowed that to happen, as opposed to the only way to push a change through might be by rebuilding your model in a tool, for example.

Anyway, this is a great little story. There's a link in the back, and you should take a look at it.

So these are just the questions I wanted to leave you with from a technology and a people point of view, and I think you've heard each one of these things.

Do you have some standards for this? Are you trying to rationalize? Here is going back to math, where you have common denominators and you factor things out. You basically want to get to a sweet spot in terms of the tools that you are supporting.

DevOps technical debt. Like I said, I bet I just invented that. Well, I actually didn't. I Googled it. Somebody else talked about it, but I could have. I could have invented that.

But I think you can bring it back to your organization. It's something to think about.

And then this other one, how do you get the best talent? Did I mention that we're hiring? I assume that every one of you is as well.

I was at another conference, and every speaker starts with, "Here's our job link. We're hiring, we're hiring, we're hiring." That is the biggest challenge facing our segment of the industry, is finding and attracting that talent and then retaining them.

So this is just something to consider as far as the things I talked about for making that something that's better inside of your organization.

Finally, my last thing is, does DevOps-as-a-service make any sense to you? Or do you want to just come up and slap me?

I'll be here, and then I'll be at the party tonight, the election thing. I'll be around the next couple of days. Please come and find me. My email was on the very first slide here, which is going to be there, first name dot last name at accenture.com.

Happy to have a conversation about this.

And the very last takeaway is all of these resources here. So there's the pointer, the quick starts to things, the Joel Test, the frameworks, the maturity model.

But basically, the best place to kind of watch for new things to come out would be the third from the bottom, which is the Accenture DevOps blog. It happens that I have the top two posts on that, but that's accidental, and Mirco will write something soon and push me off that stack.

And there's his blog.

And then Mark Rendell is the guy in the UK, and he does a lot of stuff about organizing teams and so on, and I think he has some good practices to share.

That's it. Thank you very, very much.