Scaling Continuous Delivery Across the Enterprise
Like DevOps, continuous delivery is talked about everywhere. Like DevOps, continuous delivery is used all over. However, like DevOps, continuous delivery practices are often not scaled out across the broad enterprise technology portfolio or distributed teams. This session will focus on the opportunities and benefits of scaling continuous delivery across the enterprise and review some recommendations to achieve this.
Chapters
Full transcript
The complete talk, organized by section.
Scott Willson
Probably one of the first things to know is who I am. I'm Scott Willson, two Ls in Willson. If you don't get that second L, you're not going to find me anywhere on the internet, much less within my own company's directory.
There's my Twitter handle, email address, all that kind of stuff. I actually run product marketing for continuous delivery at CA currently. A little bit of background for me: way back when, when I started my career, before I had too many grays... I got my first gray when I was eight, all right? So you've got to bear that in mind.
But when I first started my career, I actually started as a software engineer, writing code, code jockey. Truth be told, I still do a little bit. I'm actually in the middle of writing a little college football analytics program, and I've recently picked up the little language of Python, which was pretty cool, doing some regex, trying to parse some of the stats. But anyways, enough about that.
Suffice to say, yes, I started as a software engineer. Now I do a little bit of marketing with that skill and that background. So let's jump into this, okay? Shall we?
First things first. Today is a very momentous day, and it's very apropos that we're in Las Vegas. I don't know if you guys realize how important today is, but it's actually pretty big. Today's the Mega Millions jackpot. It is the biggest ever: $1.6 billion.
I can't even imagine $1.6 billion. And it got me thinking, since we're here in Vegas and we're at the DevOps Enterprise Summit, what in the world could you do with $1.6 billion? What kind of DevOps things?
And I thought about DevOps-y, and call me crazy, and pardon my very quick distraction here, but it got me thinking a little bit about Dr. Seuss: "Oh, the DevOps-y things I could do with a billion or two. I'd put wockets in all the pockets and code words for nerds, too."
I don't know. I had an ADD moment. Perhaps most developers are ADD, but whatever. It just kind of cracked me up. $1.6 billion. We might change the whole DevOps world if you won. If you play, right? Because in theory, if you don't play, you don't win.
All right. So a heretical question here. You probably want to stone me for being a heretic even asking this question, but is DevOps the anti-pattern to digital transformation?
Just something to think about. I know most of you are like, "Are you kidding me? Of course not. DevOps is what makes digital transformation possible. That's what makes things go. Why would he even ask that question?"
I ask that question because of the many executives I talk to, the many companies I go and visit. I seem to see that this is a bit of an actual thing. Why is this? I think it's because we in the DevOps world are so busy with our heads down doing this DevOps thing that we fail to pull our heads up and take a look at the broader problems of the business, the broader context.
Particularly, a lot of DevOps stuff I find tends to be done in a siloed manner. The evil silos that we've been talking about in IT forever. But right now it's the cloud silo, because it's easy. You can do agile. You can do DevOps things in the cloud: containerized, microservices, great. And we get focused on doing this rather than taking a look to seeing, well, ultimately, why are we doing what we're doing?
And as I visit with executives and we talk about digital transformation and what digital transformation really means, and it really doesn't mean just going to the cloud, although I do find many CEOs get hired to take their company to the cloud and that somehow is digital transformation. But really that isn't. But the pushback we get is, "Yeah, we're doing digital transformation. We got this."
So here's to put it all together. Who's read the State of DevOps Report? Everyone? A lot of you have read it? Okay, key thought from that along these lines: doing continuous delivery for the sake of doing continuous delivery is not enough if you want your organization to succeed.
This is why I postulate that DevOps, at least for the sake of just doing DevOps, might actually be an anti-pattern to what you're trying to ultimately accomplish, where the business is ultimately trying to go. You've got to do more. You've got to answer the great why. Why are we doing it? What are we trying to accomplish? Those type of things. You've got to get at the root of it. Otherwise, you're just doing it for the sake of doing it.
Now, just to put this in another frame, I like stories. Who here has heard of Michael Jordan? Or perhaps LeBron James right now, moved to LA. Of course, I've been kind of laughing about the supposed disaster LeBron James is having just from one game in Los Angeles. It's a long season.
But ask yourself this: what if Michael Jordan just played basketball for the sake of playing basketball? Would he have ever made it to the NBA? Would he have ever been a champion? Name the athlete, Formula One racer, whatever it is, some endeavor. If they did it just for the sake of doing it, as hobbyists do their things, would they have ever achieved something great?
And that's kind of the point. I think that's the essence of what Nicole Forsgren found in her study this year. You've got to think of the broader context.
So here are six of the seven challenges that I identify in getting out and scaling out this thing we call continuous delivery.
First thing is value stream clarity. There have been several sessions this week. This week? I guess it's day two. It seems like Wednesday for some reason. I don't know if anyone else thinks the same way. But value stream is very important. Cash to concept. But not just per product, but really from a business standpoint. What is the value that you're doing? Why are you doing it? The great why.
How about cloud migration? I've come across many organizations I talk to, and they're going to the cloud. In fact, going to the cloud is such a strong thing, I'd almost say there's kind of a "we're going to the cloud" expletive. We're just going to do it.
And so there's this great thing that's called lift and shift that is happening. Who here is familiar with lift and shift? Anyone? Everyone. Okay. It's a disastrous idea, because what happens is, what I found at re:Invent last year is everyone just takes their big, monolithic middleware applications and they lift it up and put it in the cloud.
And you know the questions I heard as I walked around the booths and everything at Amazon re:Invent last year? "Hey, do you know anybody who can do job scheduling in the cloud? Can anyone do any kind of batch scheduling in the cloud?"
And why are they looking to do these things in the cloud? Because those apps have a lot of care and feeding, a lot of repetitive clearing. You've got to clear those darn logs, right? You've got to do all kinds of things, reset the caches, all the stuff that you did. And those things didn't really matter when all you had to pay is the electrical bill for these apps. Right?
But now that you're on the Amazon or the Azure pricing model, these CPU-heavy, memory-thirsty applications can actually cost you more money than actually keeping them on-prem. And indeed, many companies are finding this out.
If they just lift and shift, they find that they have to have the same processes and tooling that they had on-site. The world of the cloud isn't all unicorns and rainbows and four-leaf clovers. Turns out these apps still need the care and feeding that they had on-prem. All you've essentially done is just changed your data center, have you not? That's it.
Lift and shift means I now have a co-located data center. Hooray, that's exciting. I've got to manage two different places. I've got to somehow do continuous delivery across two different domain sets. That's not really what we want to do.
But the reality is, as far as cloud migration goes, most enterprises, at least the ones I talk to, what a lot of surveys and data say, is that the enterprise is going to have a co-located cloud strategy. And if for nothing else, for the data. The data's heavy. They're not throwing terabytes or petabytes of data up into the cloud. A lot of times it's just too much. They're going to keep it on-prem.
So at some point, you're going to have to contend with trying to do continuous delivery across multiple data centers, essentially, is what they work out to be.
What about fragmented tools and teams? Big challenge. Lots of open source, lots of cool tools to get the job done. But, well, the elephant in the room. I'm with CA. Who here has heard CA is getting acquired? Anyone? No? Yeah, lots of you. Okay.
M&As, mergers and acquisitions, is a reality of the world. So you get these nice tools that you do. Two years later, your company makes an acquisition, and they have their own sets of tools. Now, how do you reconcile that?
In fact, I have found at some of the companies I have gone to, they don't even need mergers and acquisitions. They just distribute. A big retailer, who I will not name, they have a big office in Atlanta, where I'm from, and they also have a group in the Midwest and somewhere out in California. And they found that these distributed teams have their own sets of tooling, and they're trying to create a continuous delivery best practice.
If they don't consider the larger use case and come to some type of consensus on how they're going to work together and have a standardized place, then really all they got is the Atlanta-based continuous delivery practice and the LA-based continuous delivery practice. They move at different cadences, different speeds, and they use different tools, standards, and so forth.
And executives, when you get up to a certain level, they kind of like standards. They like repeatability. And we're going to get into why that is in a moment.
Legacy versus modern. That's actually a real thing. I find a lot of times you go to conferences like this, we tend to think, "Oh yeah, we can do all this stuff," but the little secret is it's if you're using containers, microservices in the cloud. At the modern enterprise, you've got all this other stuff to deal with, and we're going to talk about that in a second.
Visibility and measurement, that's a DevOps thing. You can't improve what you can't measure. But how do you measure all this stuff? And even the stuff that you're not actually writing?
And then, security and compliance. You really don't need to say much about that. Obviously, security and compliance have to be first-class citizens. It can't be something you go do after the fact. Auditors like to have a nice single place to go see what actually happened. They like to be able to know what happened without variableness.
One of the things I've always said, something I've lived by, is that computers are great at doing what you say, not what you mean. Human beings, we're great at doing what you mean. And so security and compliance has got to be a first-class citizen. We're going to jump in on that in a second.
So I wanted to talk a little bit about legacy to microservices because it is a big problem. This picture, as simple as it is, symbolizes the enterprise.
You've got mainframe and, hey, mainframes have developers, too. Sometimes I think we forget this. To use perhaps a political parlance, the inconvenient truth is that there are these other little tools, commercial off-the-shelf things like Siebel, PeopleSoft, SAP. They too have developers, and they too have their own thing. Maybe we don't like to admit that they're actually real developers, but the enterprise has these, and they have their own cadence and their own things.
And then, of course, we've got the emerging stuff that we're all excited about. Yay, we can all do development in the cloud and it's so cool. I get Visual Studio Code, and with a couple of commits, I can push things up and it's so awesome.
But what happens if we don't respect the entire portfolio? If we don't pull our heads up and take a look at this horizon? What we have is the hurry up and wait.
I don't know whoever's been to a doctor in the last couple of years, but I swear a few years ago, doctor's offices had a really great innovation moment, and they realized people don't like waiting in the lobby. I remember this. It must have been about 10 years ago. I went to the doctor, I think with my son or something, and man, I couldn't believe it. We signed in. They pulled us into the back room within a couple of minutes.
But see, then the problem was I sat in the back and waited in the waiting room. Where I used to wait in the lobby and then be seen, I'd be seen immediately with the doctor. Now they just make me feel better, sort of, by having me go immediately to the treatment room and wait forever.
And that's what happens. If we just focus on just the cloud stuff, then every 50th, every 200th deployment, we now have to wait for those darn Siebel guys to catch up with us, or the mainframe guys. True digital transformation happens when you've got really continuous everything.
And let's be honest, the CEO lifespan is two years. So why aren't people just going to wholesale rewrite all of this stuff? We'll rewrite some. Some of you guys are actually rewriting some of your major apps cloud native. Show of hand, anyone? Several of you. Okay, so it happens.
In the mid-'90s, I was part of doing some of this too. I rewrote some new apps. I rewrote several apps several times. There was this whole COM, DCOM thing that Microsoft was pushing, and then it was Java, and we're going middleware, and it was SOA, and we rewrote things.
And what happened is we'd rewrite an app and write a couple new ones, and then over 10 years, you look back and your portfolio has kind of ballooned because you don't get rid of the old stuff.
Now, from a business context, why don't we get rid of the old stuff? I talked to a railroad company a few years ago. I don't know why in my head I'm forgetting who they are all of a sudden. But they took the unprecedented adventure, shall we say, to go ahead and rewrite their mainframe code. They want to get off the mainframe.
And I cannot remember how many hundreds of thousands of microservices that they had at the time that they explained this. But at the time of their explaining what their journey was, this gentleman tells me they had spent eight years and $200 million, and they weren't close to being done.
And I thought, "Eight years? How many CEOs was this?" If you're a public company, what board's going to go, "Tell you what we want you to do. We want you to ignore revenue and go ahead and rewrite all this legacy stuff for about 10 years, spend all of our money, and maybe we'll become profitable on the other side of this"? It's just not going to happen.
So we have to live with this stuff. Rather than trying to ignore that this is the reality, let's deal with the reality and make everything fast. Do continuous everything. Continuous delivery of the mainframe, continuous delivery of SAP, your Oracle systems, whatever it is. If it's an app, it can be delivered continuously, and we should be thinking about that.
And I think the time to consider this is now, because three-quarters of all companies say they're doing continuous delivery. But they're doing it, like I said, for mostly the cloud native stuff. So that's great. We've experimented and have some success doing this. So now let's take those lessons learned and let's work back, right? Let's go to this left and start figuring out how we can apply these things to the mainframe and to other legacy systems, not just the mainframe.
Security. I could have brought lots of different statistics, but it would have been too easy for me to get distracted because the numbers are something else. But we all know this, right? Application layer attacks are the leading cause of database breaches.
But the only thing I wanted to jump in here on security, because we all know, right? Security has got to be a first-class citizen with continuous delivery. But the key thought I want to share with this is that everyone is so interested in scanning their code. We got all these different tools that scan the code, and it goes out, and it gets updated, and we know that the code we're pushing is secure.
But I met with a pretty big shoe company, and I talked to their security auditors at one point, and I asked them just a simple question about security. I said, "So my experience, I used to run a small IT shop somewhere along the lines of my career. When we do deployments, I don't know, every third update or something, we're always changing the app, putting features and things. And every so often, some new feature required some new access, some credential to some other file system or some database or something, right?"
He goes, "Yeah."
I said, "So what do you do when that happens? You probably grant them access because the deployment fails, right?"
He goes, "Oh, yeah, it fails."
"And then you hurry and grant the permission so the deployment will work."
He goes, "Yep."
"And then you take that permission back, right?"
And the guy looked at me, he's like, "Well, no."
So in other words, you've got a God-mode user that's designed to be destructive to your environment that's just out there, should it ever get compromised, can wreak havoc on your environment.
So the idea is, what I'm trying to say here is it's one thing to make sure your apps are secure. It's another thing to secure the automation mechanics themselves. And we need to make sure that those are secured, because those credentials and those mechanics are designed to make destructive changes. Now, they do beautiful destruction and construction for us, right? But we don't want those to be compromised.
Okay, so some other thoughts. What about scaling things out? We already covered security. We already covered the whole portfolio. What about leadership?
In the book, I assume almost everyone has read this book, yeah? There was another key thought in here, near the end of the book, I thought was very interesting and very candid of Gene to admit this. But basically, he says that a lot of the change agents that made all this great progress in their company, they have an opportunity to go work somewhere else. And so they take it, and upon his or her departure, the company reverts back to where they were, and they go back to their old ways.
So that means the continuous delivery practice, and indeed the DevOps practice, was not sustainable. So to be a leader, we have got to focus on sustainability so things, practices, this goodness that we talk about here survives your departure when you leave. Otherwise, we're just doing change at the end of the day for the sake of change.
Another great book about leadership, it's a great book, Robert M. Gates. But he's talking about if you want to make change and you want to make sustainable change, then you need to focus on pulling people outside of their silos. The quote here is, "The best way to get access to and use internal talent and ideas to implement reform is to get people from different parts of the organization working together outside their normal bureaucratic environment."
We kind of heard that this morning, didn't we? I mean, we had a lawyer talk to us this morning. I've been, I think, to every DevOps Enterprise Summit. I've never heard a lawyer give a keynote before. I thought, "Wow, we have arrived. If the lawyers are talking about DevOps, maybe we really have arrived."
But that's kind of to this point. We're pulling people outside of their own individual silos. It really takes the expertise of everybody if we're going to do something in a sustainable way.
And when it comes to expertise and externalizing it, we need to avoid groupthink. DevOps has an opportunity, or has the myopic tendency, I should say, to become very thinking upon itself. Dev, no ops, just engineering best practices.
Now, according to this book, another great book, highly, highly recommend it, James Surowiecki, he says that we need to avoid groupthinking. Homogenous groups, he says, are dangerous, because they think and only see things within their own bubble. They cannot see, nor think, or comprehend anything other than themselves.
Sort of like those Uber driverless cars and bicyclists. Those AI machine learning algorithms with backpropagation only know the data that you teach them. And when the weird thing happens that they were not taught, like a bicyclist running in front of the car, disaster strikes.
The same thing happens with people. I'll read the quote here. It's really good: "Non-polarized groups consistently make better decisions and come up with better answers than most of their members, and surprisingly often, the group outperforms even its best member." This is because you have a diverse set of opinions, a diverse set of perspectives that are being applied to a particular problem.
The book goes in to understand how this works in the NFL, or actually does not work in the NFL, how it works in stock markets, and even in the political races. It's a really good read.
But another thought, another use case that he brings up is the disastrous shuttle crash in the atmosphere. The Management Mission Team said the performance of the MMT is an object lesson on how not to run a small team or group. Instead of making people wiser, being in a group can actually make them dumber.
That myopic groupthink, a homogenous group, is very dangerous because there's no diversification of thought. Now, what happened in this crash, when the shuttle went up, there was a group that looked at the foam they had found down, and they did the simple math. It's just foam. But at the amount of tonnage of foam that it was, they realized, "Hey, wait a minute. There actually might be a problem."
Well, the group that they kept trying to talk to did not give them equal footing to express their opinion, so their opinion got ignored. And of course, disaster struck.
I thought it was great to have a lawyer here today. You should have a business analyst on your DevOps team. You should have other people on your DevOps teams that have those perspectives. Non-technical perspectives can actually help avoid disaster.
I remember being part of a group and everyone was saying... Well, I won't tell you the group, because it'll hit a little too close to home. But suffice it to say, it was with this group, and as we're thinking about papers to write and what we were going to author, there was a concern with people, some angst, that we don't understand why businesses don't think it's okay for us to fail in production, and we should be allowed to experiment. It should be okay to fail.
I actually worked for a broker-dealer. And I remember, I reported to the CEO of this broker-dealer, and he said, "Scott, something you need to know about a brokerage."
I said, "What's that?"
"Something goes wrong, if we're out of compliance, I, me, personally, I get fined, not the company. If we're found fraudulent, whatever, I go to jail. The company gets nothing." He said, "I don't want to go to jail, and it's your job to make sure I don't."
Failing in production in that environment is actually not a good thing. There was an investment company called Knight Capital who had a failure in production. Some of you are nodding. You remember it. Within minutes, they accumulated a $9 billion position in the market. It was a false position because of some errant code that went into production.
So I say it's one thing to be able to fail, but you need to find safe environments in which to fail and experiment in. There are some environments, healthcare, airline industries, Wall Street, and others, where it's probably not okay to fail in production. Actually, things can go very wrong, very bad for people.
My last challenge here to talk about: who here has heard things like this? It's an application company. Application economy. No, it's a data-driven economy. I've heard that. Every business is a software business. It's about the app, stupid. Has everyone heard these kind of things?
I will tell you, I've run my own little small business and some other ventures along those same lines. I'm going to tell you a little secret now. You guys have heard what I'm about to say here, but it's not organized for you. So I'm going to break it down.
There are really only three things that a business cares about. Under this same guise here, I could tell you guys, almost sometimes what we talk about here at DevOps sounds like a manufacturing line. If I ever worked for an auto manufacturer thing, I was like, "No, every company is a manufacturing company."
I can even take it for every company is a cog company and a wheel company, right? A conveyor belt. Because the reality is if suddenly we couldn't make cogs, the world would stop. But how much value does the business put on those cogs? I mean, just think about that. The cogs make the conveyor belt. You couldn't even get an iPhone without cogs and gears on those manufacturing floors. But yet they pay pennies for these things.
What does the business care about? They don't care that much about cogs. But I'll tell you what they do care about. They care about increasing revenue, period. That's it. How do I make more money? In fact, the whole reason you're in business is to make money. Kind of obvious. If you ever did a lemonade stand, sold cookies when you were a kid, you didn't do it to lose money.
Although I did have a cousin who thought so. I remember real quick. Oh man, this kid, he's had an interesting life, but that's a whole other, perhaps comic, session. You can come see me at some bar tonight and I'll give you the story. But he said, "Scott, I got this idea, man. We're going to go buy some Kool-Aid and we can sell every glass for a penny."
And I worked out the math for him. I was like, "Dude, we're going to lose money on this." He just did not understand his efforts would actually lose money. But that's not good business anyways.
Next thing businesses care about: decreasing their costs. They want to save money, make as much money as you can, because the difference between these two are what? Your margins. And there's a reason why I'm telling you guys all this.
And there's one third thing that any business owner, any business executive cares about, which is mitigating and managing risk.
The reason I bring this up is because every time I come to these DevOps shows, when I speak, when I talk to you all, all these things, we speak geek. And we try to do this evangelist thing at our companies. I mean, Gene Kim talked about it. We'll come here and we'll hear all this good stuff. We'll get so excited and think we can last another year. That was his opening comment, right?
But the problem is we go back to our boss and we start speaking geek. "Oh, I got this tool and this widget. We can go so much faster. We can do this thing." And he's like, "These three things. How does that help me accomplish these three things?"
And you're like, "Well, I'll give you a quick example. Last two minutes. We can go to market faster." What does that mean? What does it mean to your business by going to market faster?
A very simple way to look at it is, if I were in the business of selling Snickers and I could get a Snickers bar out to market by September, how much more money could I make versus putting it out in the market in November? That three-month gap means what? I can sell more Snickers than I could otherwise.
You have to frame it in this kind of way. These three things. That's it. That's all any business is ever, ever doing. Just those three things. It's kind of simple, but having met with several C-level folks, this is what they care about. That's it.
So key thoughts to think about here as we wrap up, and I'll add one extra thing to this. I remember meeting with a CIO of a big bank, and we were talking to him about all these neat things we could help his company do, right? Make his people more efficient and this stuff. And he says, "You know, Scott, this is all great, but I got to be honest, and this is going to sound bad, but none of this matters to me."
I said, "What do you mean none of it matters?"
"This," he says, "my people in IT, their salaries are fixed. So whether they work 80 hours or they work 30, I don't care." He says, "Now, I know that sounds bad because I do care," right? And his chief technician was right there. He goes, "I do care about these people. I truly do." He says, "But from the balance sheet, unless you can address these three things against a fixed salary base, there's no business case. There's no sense doing it."
So key thoughts to walk away from. Be a leader. Persuade, guide, respectfully argue. Don't be afraid to argue your point. But be respectful. You've got to persuade people. You've got to kind of lead them so they understand and can get to where the business case is. The simple example I gave you about Snickers: going to market faster is not the end. Answer that financial question. What do you get by going to market faster?
Think more broadly about continuous delivery, the entire portfolio. Go for a sustainable practice, not just a siloed practice. Embrace the entire portfolio. I already mentioned that. Continuous everything, not just continuous something. Continuous everything.
Make the mainframe go faster. You'll watch how quickly things change. That's real digital transformation.
Secure the automation mechanics themselves. Think in terms of really including ops and making ops agile. Learn to speak the language of business: risk, cost, and revenue. That's all a business cares about ever.
The whole reason you can invest in Wall Street, because everything boils down to these three things. You can look at any company and figure out what are they doing with those three things and determine if you're going to invest in them or not.
DevOps plus multidisciplinary teams. Multidisciplinary teams are the key. I think that's important. Start including folks. The most efficient team I had in my career was made up of some developers. We had some QA people. We had some business power users and business analysts, and that was our team.
And man, we pushed stuff out so quick before DevOps was ever called DevOps, and it was always high quality because we had this diverse perspective. And the business power users would always find weird bugs that were not technically bugs, but they would always have this perspective on the business that would help us.
And then just last thing here in ending, it's a journey, not a destination. It's a continual cycle of continuous improvement. It's not just saying, "We did continuous delivery. Woo-hoo. We're done." You've got to continually tweak it. Sort of like working out, something you need to do more of. You can't just reach a certain point. You've got to continually work out.
And then here's just once again, contact information. And yeah, thanks for your time. Thank you.