Sanjeev Sharma

Welcome, everybody. We'll give a minute or so for people to settle in. Most of you are probably coming from the Disney talk. How do you follow that with all those video clips from all the movies? I thought, "I'll put up a video of people assembling data centers," but that isn't as much fun.

We'll give it a few seconds, let people settle in. Everybody got a copy of the book? I'm the author, in case you were wondering.

I think we'll start here. Ready? Excellent.

Good morning, everybody. My name is Sanjeev Sharma. This is the only slide we have. We're just going to have a conversation. I do have a few pointers, which I wrote down, and we chatted a little bit yesterday about what we're going to talk about, but this is going to be an open conversation about DevOps and scaling DevOps in the enterprise.

We do have a microphone in the back, so if somebody has a question based on anything somebody says here, feel free to raise your hand, and we'll invite you into the panel discussion. Let's make it as interactive as possible.

Let's start off by doing some introductions. My name is Sanjeev Sharma. I work for IBM. This is my old title. I changed roles around a month ago. I'm right now the Architecture Guild Leader for our cloud architecture practice. We work on large, complex, and first-of-a-kind migrations to the cloud in enterprises.

But I put up my old title because that has more street cred at this conference. So I'm like, "Let's stick with that for another month." If I put cloud architecture, you wouldn't be paying as much attention, I guess.

Let's introduce our esteemed panel here. We're very fortunate to have Mark, Chris, and Frank here because they have done what we're going to talk about here, or are in the process of doing it.

Mark Peterson

We're in the process.

Sanjeev Sharma

In the process of doing it, right.

Why don't we take a minute each and introduce yourselves, name and what you do at your company. But I also want you to take a couple of minutes to talk about why you are on this panel. What are you doing? We'll talk about more detail, but just as an introduction, what in your role, why are you at this panel to talk about scaling DevOps at a large enterprise? You're either at a large enterprise or you're working with a few of them.

Mark, why don't we start with you?

Mark Peterson

Sure. Mark Peterson. I've been in this role, or this title, for a couple of years now: platform services architecture. I have a long history in Wells and other financials, mostly working in the application support and support world, or engineering world.

The question really of what are we doing? I'll extend my role first. What we're really focused on at Wells is transformation overall. This isn't just a technology issue, but the bank as a whole is really fundamentally changing how it works.

In technology and HR and marketing and all of the service parts of the organization, we're moving from what has always been "run the bank" or "run it like your own" attitude in each individual product line to a much more enterprise-first approach towards banking.

Certainly, this has a little bit to do with the sales practices issues that were announced a few years ago. But it really goes back further than that, to the regulatory issues facing banking overall. Between the regulatory environment, just the business environment, and the need to be more efficient overall, it has really changed how the bank wants to operate.

For me, that's really the role I'm trying to take and be involved in in the bank: how do we make that transformation happen? What do we need to do?

DevOps is certainly a big part of it. Really fundamentally changing, standardizing, simplifying, modernizing the architectures and methods we use to approach such a large endeavor. How do you fundamentally do things differently?

That's probably the biggest part of why I'm here: to talk about what that means, how it happens in an enterprise at our scale. Certainly, the biggest issues have little to do with the technology. It has everything to do with the organizational and cultural models that have to fundamentally transform and change.

Sanjeev Sharma

Excellent. Thanks, Mark. Chris?

Chris Nowak

Thanks. That sound okay?

I'm Chris Nowak. I am Chief Transformation Officer at Kingsmen Software. It's a small startup where I run the DevOps advisory practice. We do a lot of different things, software engineering, but that's not really why I'm here.

There, that's a little better.

That's not really why I'm here. I actually spent 20 years in the banks. I was at Wachovia, Wells Fargo for a while, Bank of America. I've seen a lot of change. For about the last 10 years of that, while I was there, it was all about creating basically DevOps. 2006 was before DevOps was a word, but it was about automating and scaling what we were doing.

We started at Wachovia, which was acquired by Wells Fargo in 2008, 2009.

Mark Peterson

Yep.

Chris Nowak

It was sort of a startup situation where we didn't have funding. We had a little bit of executive support, and we had to grow it that way, then gather folks along the way and make some mistakes. So we had sort of a startup mentality there.

When Wachovia was acquired by Wells Fargo, it was, how do you cross that barrier to be part of a larger acquisition and be a surviving service? At BofA, it was this much larger scale.

Ultimately, at Wells Fargo, we did roughly 350 applications fully automated, tip to tail. Bank of America topped out around 800. There was a little bit of technology there. We had to rationalize things. We had 50 different tools. But it was really about the culture.

Anytime you install a tool, it's not the problem of installing the tool. It's about how do you integrate it with the ecosystem, both procedurally and culturally? How do you work with your infrastructure teams?

I would say that probably the biggest reason I'm here is to say it can be done in a highly regulated environment. Mark and I actually knew each other from Wells Fargo at the time. It can be done, and it's not easy, or it would have been done already. To me, that's the next big problem to solve.

Sanjeev Sharma

Excellent. Thank you.

Frank Canihuante

Can you hear me?

Sanjeev Sharma

Yeah.

Frank Canihuante

Frank Canihuante from Travelers. At Travelers, I work in the tools that actually make the build and deployment possible. I'm sort of the architect on that.

There we go. How's this? All right, I'll repeat it.

Frank Canihuante from Travelers. I work in the tools that do CI and CD, basically. That's kind of an interesting point of view because Travelers has been doing some great things in the DevOps space, and they've been doing it for a while. Perhaps going back 10 years, before it was even called DevOps, because there was always a need for someone to do configuration management, infrastructure builds, and deployments.

But it got to the point that the practices that everyone wanted to do, especially in the continuous delivery, continuous deployment space, were just not possible to do with the tools that the company had, which was an internally built system to do builds and deploys.

We went through a period that lasted about three years where we moved to a new platform that allows us now to do even more things than they've ever done. We're now at an interesting point because we feel that we can do more.

We are here, and I say "we" because there's many of us here from Travelers, and we attend all these sessions just to get inspired, really. To really say, "These are things that can be done because we've done them, and we want to do more." It's good to connect with others. It's good to see what others have done. We also want to say to others what has worked for us. Many of the things that Mark and Chris talked about have worked for us as well. So that's what I'm here for.

Sanjeev Sharma

Excellent. Thank you.

Let's move into one of the things you all said, and we are seeing this a lot. DevOps might be, what, five and a half years ago, since that infamous or famous meetup at Ghent, Belgium, where they coined the term DevOps. I always joke it's the third-best thing to come out of Belgium after chocolate and beer. That's how the term DevOps came about.

But in reality, we've all been doing what is today put under this DevOps umbrella for a while. What DevOps and the DevOps movement has done is given it some kind of organizational structure, some kind of a name. It's not a methodology like Agile, obviously, and I don't think it should be.

At the same time, it has also created a recognition in senior management in large IT organizations that we need to do something. But that needing to do something, that desire to do something, translating into actually executing into the kind of transformation you all are talking about, has to have some business driver. There has to be a compelling reason to act. Then we can take that reason and say, "This is the business outcome I'm looking for."

Maybe we'll start with you, Frank. Was there a compelling reason to act at Travelers which motivated you to make it more formal than that internal grassroots movement, which DevOps was before DevOps became DevOps?

Frank Canihuante

Yeah. Can you hear me? Yeah.

There was always the need from the grassroots movements, something internally bubbles up from the bottom to the top. But now what we're seeing is that the top is looking down, and they're saying, "These are great things that people are doing." It's not necessarily driven from the top down, but that is happening now.

As I was saying before, we're at an interesting inflection point that we can do more if we can meet somewhere in the middle with the business drivers. They're always better quality, release more frequently, and all that.

What are the things that we need to do? Not necessarily tools-wise, but what are the practices? From the human point of view, what are the collaborations and interactions that we need to have? What types of activities would help us do that?

To be honest, one activity that does help us is precisely attending an event like this, where there's greater awareness of the value that there is in DevOps in general.

Sanjeev Sharma

Is there a particular set of KPIs that you came up with? That's something I know, and maybe my brain is fine-tuned for it because of conversations I've been having with our clients. That seems to be a theme: "Okay, it's fine to do DevOps, but to what end?" What are the measures of success to say, "Yes, we have improved"?

It's not easy. Mark and I were talking about it this morning. It's not easy to identify those in a lot of cases because there is no baseline to measure against. Is that an effort at Travelers, to identify certain key metrics to establish a baseline?

Frank Canihuante

Yes.

Sanjeev Sharma

What are the ones which you guys care about?

Frank Canihuante

It's interesting. We now have some really lofty goals that we want to meet by 2020. We have a lot of things that we want to do.

For example, we want to fully automate things. That's kind of hard to quantify, so we put a percentage. We want, I don't know, 98% automation. But to your point, there is no good baseline. That is a challenge in and of itself.

However, the thought that you want to do that is a good thing. How you actually measure it will have a greater impact to see if you want to continue with this and be even more successful with some other metrics.

Yet another thing that we want to do is, at some point, we're going to tackle other environments that are even trickier to measure, such as the cloud. Right now, we say we want to deploy N number of applications with a particular success rate, and that also has to be done on different runtimes, but there are different challenges there. A lot of things need to be figured out.

So just some examples of the high-level things that we set to ourselves as to how we're going to measure our progress. But there are many more.

Sanjeev Sharma

Can we skip to Mark? Mark, Chris, I want to ask you the question a bit differently because your perspective is a bit different.

Mark, how about you? I know we chatted a little bit this morning, but let's share with the crowd here.

Mark Peterson

I'll start from a personal point of view about it and then move into more of the Wells Fargo point of view and why we're prioritizing.

In my view, the reason for the DevOps movement is the amount of technical process debt that has been built up over all previous generations of operational management has grown to the point that it's unsustainable.

Using Wells, every time there's an incident, a new process is put in place, to the point that we have just stacked process on top of process after process. To the point that we can't get a server installed in less than 10 months now.

The operational debt that has come up is just completely unsustainable. I think, in many ways, to me, that's really the reason for this transformation that has to occur. I suspect that's true in many places.

The given reasons at Wells have moved over the past few years, and I think there's been a bit of history. Certainly, a big driver was time to market, trying to improve these unbelievably slow delivery times across the board. That is still an issue, but general efficiency, cost efficiency, has certainly come into play.

But it is safety and soundness. How do we automate to make things simpler, secure? How do we do patching faster? How do we get ourselves out of cybersecurity issues? How do we deal with all these issues? That's probably the number one driver that we see this as an answer to at this point.

Sanjeev Sharma

I think it's very interesting, and I really appreciate your using the term process debt, because we hear people talk about technical debt and code debt. Nobody openly admits to the process debt.

One of the things we do at IBM a lot is we go into large organizations and do value stream mapping exercises, where we look at the process flow end-to-end for a delivery pipeline or a set of them. Where we find the major bottlenecks and inefficiencies are in the process. Even when you have tools, they're not being used to their capacity because of that process debt.

The numbers we found on average are like 65% of your time and money is spent doing process overhead, and only 35% is spent actually doing work which results in code getting to production.

Mark Peterson

That's exactly what we see.

Sanjeev Sharma

So my numbers are good?

Mark Peterson

Yeah.

Sanjeev Sharma

Excellent.

Chris, from your perspective, you are looking at it from a wider aperture because you're working with multiple clients. You've been in the industry, but now you're working with clients. Do you see certain patterns and trends of what the business drivers are and KPIs are? And what do you think they should be?

Chris Nowak

The question, is it on now? In what you guys just talked about, there's so many things we could pick up on and talk for an hour, but I'm going to add to not just process debt, but we'll talk about culture debt as well. You have to really address all of those things.

To that question, look at your technical debt, look at your process. These are things that we see as patterns across the board everywhere, and we know that. We talk about DevOps as culture, automation, lean process, metrics, sharing, all these things.

In terms of a metric, let me back up. What I've seen is really interesting, whether I'm working with a company that's 50 or BofA is like 180,000, you guys are like 268, something like that, everywhere in between, they tend to have the same types of challenges. It ultimately goes up towards never retiring the process. We always add process. We never prune it.

If you are familiar with The Goal, Eliyahu Goldratt, it's all about you should examine why you're doing something. There was probably a good reason for doing it at one point, but is that relevant today? Take some time and retire that process debt too, which kind of leads into something else, like measuring your cost of delay, which is a Don Reinertsen thing. If you read his book, it's great.

I guess one of the topics in that is you don't improve the point where you are doing something. You get rid of the delay in between. That's your first win. Think about it. You all know this is true. You deliver your code, it sits on the shelf, it's waiting for QA to test. Not really QA's fault, and it's not anybody's fault. It's the process's fault that has grown up over time. Look for those opportunities to just whack that cost of delay in between. That's common across the board.

If I were to pick one measure, I love what Nicole's doing with DORA and the four measures they pick. I think the one thing that I see over and over is, if you can just measure, which kind of goes up to cost of delay, your speed of deploys, from the time you check in your code to the time you get to prod, if you're going to do one thing, measure that.

Then you can lift the cover up and you can say, "Why is my build taking 70% longer?" Great, go do a spot treatment on that. You can say, "Why is this deploy taking so long? What do you mean it's going to four groups to get approved to go to QA? There's no integrated testing there. Why can't I just do that myself?" Get rid of that process.

I think those are just some of the top-level types of things you can look for, no matter what size you are.

Sanjeev Sharma

You made the perfect segue for my next topic. This is a theme we see over and over again. I've been to several DevOps Enterprise Summits and several other conferences, and of course, working with dozens of clients.

The cultural inertia which organizations have built, and it's not even homogeneous, because most companies we work with, including IBM, they are not homogeneous. They are different divisions, subdivisions, lines of business. They might have grown through acquisitions and mergers. We never truly integrate culture, so we have different levels of cultural inertia in different organizations.

Everybody has a pocket where there is some SWAT team or skunkworks project going on which seems to fly under the radar, has no cultural inertia, has no governance overhead.

How do you scale that? How do you overcome that cultural inertia? Does it require organizational change? Does it require bringing in leadership? Is it a grassroots effort? What have you all found as being the most effective mechanism in your organization? What has worked, what has not worked when it comes to overcoming cultural inertia?

Why don't we start with you, Frank?

Frank Canihuante

There may be different things that work, and perhaps different things that work for different folks in different situations. What we have seen that has worked at Travelers, at least in the past three years that I've been there, is collaboration. Actually looking at a problem and saying, "This is our problem." It's a common problem, and we can all solve it together.

The ownership of the problem by a group of people, and not necessarily one centralized group or a particular line of business, is what gives us that sense of community, that this is a goal that is enterprise-wide. Just to think about it in those terms makes you come up with solutions that are more collaborative in nature and far-reaching than something in isolation.

Then if you see success with that, you tackle something else and something else after that. We're at this point now that we start to feel that what we have achieved with this working together can go on to solve other problems.

One other problem that we want to solve is not necessarily the speed of deployments now, which we feel is under control, perhaps. It is: can we release sooner? Right now, many quarters, we release every quarter, but can we release every week or perhaps every day? If it makes sense, and it will not make sense for everyone.

That's what we have seen.

Sanjeev Sharma

One of the things you mentioned, that collaboration piece, I think it's that sense of shared ownership, which is a major challenge in a lot of organizations because we've built these artificial silos where it says, "Hey, my part is done. Not my problem anymore."

In fact, that's my pitch against creating these overlaid DevOps teams. I go into many organizations, they say, "We have a DevOps team over there." The problem is you're just making them responsible for the DevOps piece. You're not taking responsibility yourself.

That shared sense of ownership is very important. My best example of that, where it was done well, I can't name the company, but they had made these T-shirts with "Everybody is responsible," and they gave that T-shirt to everybody. They even gave it to the coffee machine repair guy. The idea was that if the coffee machine is broken and people have to walk to the latte shop, you delay production by 20 minutes. So everybody is responsible.

I think that's the sense of shared ownership. Anything you would like to add to that, Chris?

Chris Nowak

It's going to be different for everybody in every organization.

Sorry. We're fading in and out. This is kind of the Twilight Zone here.

It's the microphones.

It's going to be different for every organization, but at the higher level, it kind of is the same. You have to identify that place where you can share that success outcome.

Sometimes you're going to be doing it at grassroots. I've had to do it only at grassroots until the executive thought he could trust us enough to give us money to do it the right way. In other places, we've actually had the executive support with it, which was great, but it comes with a whole other set of challenges as well.

I think you've just got to figure out where it is. But the overriding point is: find a place where you can do what you need to do with people that are willing to do it with you.

Arthur Ashe has a quote: "To be great, do what you can with what you have, where you are." That's going to be for every one of you. Find that group you can do it with. Show some success. Put some points on the board early. It has to be continuous. You cannot wait nine months to do it.

Then all those folks that were maybe a little hesitant to do it, they're like, "It is working. Nobody's jumping down their throats for doing it. They're being innovative. They're being adaptive." It creates this excitement.

As leaders, I'm assuming everybody here has some role in there, whether you have a team or you're doing it individually, it is your job to inspire. By doing that, you do.

Sanjeev Sharma

Thank you.

We talked earlier this morning, Mark, when we were chatting, that in your organization, you wanted to get some quick wins, so you went with tools first and now are tackling the organizational challenges. Can you comment on that, please?

Mark Peterson

Sure. First, if anyone has any magic answers to this question out there, please tell us.

We did start with tools first. Really just agreeing that we're going to have an enterprise standard around tools, that we are going to drive an enterprise solution aimed at those tools and move all teams towards them.

Chris, as he pointed out, when he was there, there were 350 apps put on one of what I'd call the pocket toolchains in the bank. There were a few at the time. Those pocket teams in different lines of business are now being put together, not organizationally yet. Well, even organizationally, they're starting to be. But really, we started with, let's get them all together. Let's get the people that are experts in this space together to work on an enterprise solution.

It turned out that probably two or three groups came up with basically the same solutions separately within the bank. So it was pretty easy to get agreement on what solution we were going to go towards overall. Now they're really taking their ideas, sharing them, building across for this overall enterprise goal, and onboarding all the rest of the apps in the bank.

Just to give an idea of those numbers, when Chris was there, it was 350 just in one major business. It's probably around 500 at that point. Now we've probably onboarded across the bank about 1,200 to 1,500 apps, and we're aiming for 6,000 under the solution.

It has been an effort to really start almost grassroots, but we had the buy-in at the very senior levels that we are doing this as an enterprise solution. It's going to be a pure, centralized method with a federated approach towards those groups, towards the lines of business, to support them as a real service. Just like, if you can go out and buy it on Amazon, why can't you buy it internally as a service?

We've really been driving that idea for the culture with that executive support, and then making sure we had the right people across all the different groups to contribute, to be part of it, to build the solution together.

Chris Nowak

You mind if I?

Sanjeev Sharma

Please. No.

Mark Peterson

The less I say, the better.

Chris Nowak

I agree with everything you said, and it kind of took me a while, unbelievably long, to come to the realization that I didn't own a DevOps team. I owned a platform services team that provided DevOps services. Literally, I figured that out about six months ago after doing this for years.

The DevOps teams are what Sanjeev was referencing earlier. They are the teams, not just the application, but the teams that are coming together: your security, your QA, your dev, your ops, all those folks that are getting together, architecture, audit even. If you're really good at it, you're pulling in finance and HR.

Those are the folks that are saying, "How do we deliver this thing to the business?" I actually had, and you do, I guess, we have the DevOps support services teams. It took me a long time. We operate like a DevOps team on our own, but we're not the DevOps team.

I actually think that distinction is important to make, just to keep things clear about who's doing what and who provides what to whom.

Sanjeev Sharma

I think that's a very good point. Even at IBM, with very large, highly distributed development teams, we have what we call a DevOps Center of Excellence. They own the platform. They have a platform. They maintain the platform. They also provide DevOps coaches.

If a product line comes and says, "Hey, I want to start using DevOps to deliver something massive like WebSphere Application Server," they're onboarded to that platform. They hire DevOps coaches from their Center of Excellence who come in and then help them make that transition, almost like an advisory service.

Chris Nowak

Right.

Sanjeev Sharma

Then their job is to work themselves out of a job so that team can become self-sufficient to do continuous delivery.

We have DevOps on the mainframe. You already heard a few speakers talk about that, because that platform is truly very broad, and it's not one platform for the entire organization.

But I want to make a quick comment on something you said. You said 1,200 applications?

Mark Peterson

Roughly right now, yeah.

Sanjeev Sharma

Roughly. How about you at Travelers? How many do you have?

Frank Canihuante

Now about 3,000 apps.

Sanjeev Sharma

3,000. I have to say this: I love it. I go into large organizations, and they'll come to me and say, "Sanjeev, we don't need your enterprise DevOps playbook approach. We went to Silicon Valley. We talked to some startups. We're going to do what they do."

I always ask them, "How many apps does that startup have?" Two. I think Uber has three apps. Anybody from Uber here, can they confirm that? How many apps do they have? A large startup will probably have five apps. We are nowhere near the 5,000.

You can't do what a small startup, which is co-located in one building on one floor, which has no technical debt, no cultural debt, no architectural debt, you can't take lessons from them and apply it to...

Mark Peterson

Architectural.

Sanjeev Sharma

There are good lessons to take from them, don't get me wrong. We all want that nirvana. But unless you can start with a blank slate, you have to do what they did. You have to do what we did. You have to start from where you are and start eliminating that debt over time.

Chris Nowak

And those applications are interdependent too. It's not like you have, "Oh, you can just deploy this all day and it affects nothing," right?

Sanjeev Sharma

In fact, the business doesn't care if one app gets deployed. They care, can I deliver this business function? Can I process that claim? Can I process this loan? If my service works and something else doesn't...

Mark Peterson

It's even worse than that. There's no consistency in the app definitions. What is an app in one group is a completely different thing in another. You'll have an app that really consists of 100 different applications in many ways.

Sanjeev Sharma

Exactly. We have a question. Let's take that question while Mike is making his way there.

The whole conversation we have about what's a service, what's a microservice, that conversation itself can be a whole day-long panel. Yes, sir.

Q&A

Q: I don't know if this is a challenge to what you said, but I'd like to hear you speak to what we heard from Capital One. "You have to do this." It sounds a lot like top-down, and I don't mean to be insulting. I'm not trying to. This is a good thing to talk about.

It sounds to me like what specifically Wells Fargo, you're talking about, is a top-down, "Here's a platform. We're going to march everybody to it."

Mark Peterson

Yeah. That's not what I'm saying.

Q: Okay. But Capital One seems to have found a way to say, "Small teams, we're going to have the startup culture, and we're going to do it that way." So maybe that's what you're doing.

Sanjeev Sharma

I don't want to say, as I said, it's not like you cannot get lessons from the startups, but you have to figure out what works amongst them and how to apply it to you based on where you're starting from.

Let's take something, for example. At IBM, we've implemented the Spotify squad, tribe, guild model. Our application of it doesn't look the way Spotify did it. We got good ideas from it on how to organize our teams and have that shared ownership by having squads, tribes, and guilds. In fact, I told you my new title is Cloud Architecture Guild Leader. I have it in my official title.

But ours is different because we are several hundred times, maybe a few thousand times bigger than Spotify. They're a great company. We learned a lot from them. Also, we're geographically distributed over every continent except Antarctica. I can tell you, I've been to every continent except Antarctica to visit IBM customers.

You have to learn from that. And yes, I don't think anybody here is saying we are doing top-down only. We are saying don't ignore the top-down. Everybody's doing the bottom-up. DevOps always starts as a grassroots effort by people saying, "This doesn't work. Let me pick up the phone and find out who this other guy is I'm supposed to work with." They do it in an ad hoc way.

What we are saying is make it an enterprise-wide effort and standardize it so people are not doing it in an ad hoc way, and you're not dependent upon who you know. Most places, the way things get deployed and done are based on who you know and you building your own culture. I remember when I was a developer, I used to go buy donuts for the ops guys so they would take care of me first.

Frank Canihuante

I think that's one of the things to standardize, not to centralize. Some people think of centralization as a good thing. You own the tools, you own the processes, but it's not about centralizing, which has a negative connotation: you're taking power away from me. It's about standardizing. That's what allows you to scale: standardizing on the tools, on the processes, even this knowledge and the understanding of certain practices.

We're actually working on that precisely, trying to come up with standard definitions of what CI is and particular types of tasks within CI. That allows you to scale.

Mark Peterson

I absolutely think the same. What I left out in that description is probably 10 years of teams individually at the ground level creating solutions. The problem with that has been there's no way to get it to be consistent, to be standard across the bank.

You need that top-down-level decision to say, "Yes, we want to do this type of process across the whole bank," and really get buy-in to help deliver that. Otherwise, you end up with every group building its own solution, which is essentially what's happened over time. You end up with a product selection that literally, I can go to every single vendor, and they all have something here at Wells Fargo. They're in.

To manage that is not efficient at the enterprise scale. We're trying to become more efficient, which is not to say that it's a top-down decision then or approach. We're really trying to build the expertise at the platform level as a group that's going to say, "Here's the way to build this for the bank as a service."

I do come back to that idea of building this as a service. We were talking last night a little bit about the cloud, and I hate the word cloud at this point because when you just say cloud, everyone umbrellas everything into it. What's really needed in a place like the bank is a bunch of services. These are the services that are going to be delivered in a cloud-type way. It doesn't matter what the middleware is or what the component is, but it needs to be delivered as a service to the entire enterprise.

DevOps is, in many ways, the same thing. It's a service that needs to be offered as a utility. It has to be done as a group, and it has to be done with the right level of expertise in the teams. To do that, you need that top-level buy-in that it's going to happen.

The Cap One example, I think, is a terrific example. There are many great things that they've done. Wells is larger, and it has a whole different set of issues to deal with. We often go back and forth on, should we just do a greenfield approach? How should we handle this differently that would be a better way?

We try to learn lessons from places like Cap One and others, and we do. But it's really finding our way to do it.

Sanjeev Sharma

Absolutely, because everybody's different. The starting point is different. Also, within the organization, you're different.

The way I see it, to expand upon what you said, is the goal of the standardization, what you said, Frank, is to really put some guardrails around what you let people at the grassroots do and not do so that there's no chaos. But at the same time, those guardrails will need to be very selective about what you want to standardize on and what you do not.

You don't want to standardize on a programming language. You don't want to standardize on a build tool. Those are commoditized as a service. But you do want to standardize on certain areas like operational readiness, deployment automation, configuration management. You might have more than one standard because you'll have a mainframe. I'm sure you all do, right? On the same side, you have mobile apps also. You should have different standards for them.

Chris?

Chris Nowak

I know there's a couple questions. I'll try to make it quick.

I was going to say I was going to take a contrarian point of view, but I think you started to pick up on it. I'm starting to believe there's a limit to where you do centralize, but it depends on what you're doing. You might do some of these things and standardize it and centralize it, but I think people never really look at the cost.

You have a tool conversation and it stops at licensing, and that's the tip of the iceberg. What they're not looking at is what are your second- and third-order effects, the cost of delay, all of those other things. How long are you making the business wait?

If you're not thinking about that, which most people don't, then you have a tool war conversation and it means nothing. Standardizing that at the top level is fairly myopic, in my opinion, because what you're doing at that point is not looking at all of that downstream cost that ultimately affects the business, which is really what you should be aligning on in the first place.

Optimize for business value and take it to the right level. I do think that as you go up every level, you start at a team and then the division and then the enterprise, you keep hitting these cultural barriers. Every group of folks, really, it's going to be once you unlock the culture to allow the right types of choices around standardization, then that's where you can really start to excel past the folks that you're competing with.

Mark Peterson

I'll add one more to that. This may sound contradictory to some of the other things I've said, but what I'm recognizing is decomposing.

A lot of times people just kind of put everything, and everything ends up meshing in the SDLC process. Your SDLC policy becomes policy, process, procedures, standards, all comprised in one kind of document.

I've started to really try to decompose those aspects of it. There are principles, control points, policy, and process and procedures and tools, so that you can have ultimately the conversation of letting the right teams choose, at the ground level, the process and procedures and tools, while still having control points that are key to the organization as a whole and an enterprise as a whole.

Sanjeev Sharma

We have a question over here.

Q: It's a quick one. Thanks for sharing these experiences. I'm really interested in your perspective on speeding up some of the rates of adoption.

I want to double-click a little bit on the standardization part, and in particular, around vocabulary. I come into a lot of different organizations, and when you have small-scale conversations, meeting rooms under 10 people, people are talking about entirely different things, even though the words are the same. Like you mentioned, platform or services.

Sanjeev Sharma

Cloud. Define an application for me.

Q: Yeah. At a small scale, you can get that team to start aligning on the meaning and to have a vocabulary that's pretty consistent. But you guys have introduced, with DevOps in general, a lot of new words that are brought together, or terms.

How important is it to standardize the vocabulary when you meet with the broader teams or the bigger groups in their domains? When you bring up something, how do they respond? I guess the question is, how important is it to standardize vocabulary and broaden that?

Chris Nowak

I'm glad you asked it because I think that's really important, and I think I forget about it, and a lot of us do. If you're not speaking the same language on what you mean, you're going to have conflicts for the rest of the year. Absolutely.

What does it mean to be a build engineer? What does build mean? To me, build is not an artifact set. Build is the process of creating something that can be deployed in a run state. I usually actually start that. I'm like, "Build means something different to us. Let me tell you what it means."

You have to do that, or else your teams aren't communicating. You might as well speak different languages.

Frank Canihuante

Pierre, even words that are overloaded like pipeline. People think of that as a set of tools, big and small. Is it a set of processes, or is it a particular thing like a Jenkins Pipeline plugin?

To your point, if you don't have a standard language that gives you a common understanding, it's just hard to do. You're going to constantly run into issues of thinking about the same thing but looking at it differently, or looking at different things and saying it differently. It's just not going to work.

Q: Exactly. Can you share what has helped to align these?

Sanjeev Sharma

I can.

One of the things we do when we go in and start a DevOps transformation engagement is we say, "Let's set up a DevOps Center of Excellence, central competency," whatever you want to call it.

At a large bank where I worked, we did that with a Friday afternoon lunch and learn every week. The first lunch and learn, it was six people, and I was the speaker as an outsider. Today, they have over 200 people, 300 people on a web conference every Friday.

The idea is kind of a watering hole where people can meet, share their experiences, and then through that start harvesting. "Okay, this is our definition of what does it say that we are fully deployed or fully automated on deployment. This is our definition of a release. This is our definition of a build. If we are going to use containers, we are going to do X with containers. Everything is a Helm chart." Whatever the standardization you want.

These guys are deciding on the nouns and verbs of your taxonomy. It's very important, but it's done as a group. You cannot mandate that. But you also ensure that it's standard. Otherwise, people like us vendors will come and confuse the heck out of you because what I call something is very different from what the CA guys or the Pivotal guys will call it. We have our own definitions, trust me.

Within our company, we have different definitions. When I go to companies, I say, "What is DevOps going to mean to you?" Because we don't have a definition of DevOps. I think we all agree it's around culture, and that's probably where the conversation ends.

We have a lean agile practice. The first thing our person will do is she'll say, "What is the definition of done for you?" So do that.

Chris Nowak

Right. And is done deployed to production? Is done a user using it? Those are two very different definitions.

Sanjeev Sharma

Yes, sir.

Q: Can you guys talk about how you decide which one of your applications that you're going to move into the pipeline, the DevOps process first? Speaking from large organizations that have core business and then not core business, clearly core business gets done first, but how do you balance the cost of automated testing and getting that into the pipeline versus the benefits?

Mark Peterson

I can start on that one.

Sanjeev Sharma

Okay, you heard it first.

Mark Peterson

We've bounced around on that a little bit. I'd say our main approach is from a risk lens, especially around vulnerability patching and how to fundamentally get the environment cleaner.

We have a whole ranking system for all of our apps, and we call it the Critical Application Model something or other, CAMS. Every app gets a ranking. It's critical, high, medium. We've just been going on that ranking as far as moving them onto the toolchain, for the most part.

Where it gets interesting is we're shifting that a little bit now, because the model, and straight out due to Struts, due to Equifax, due to those issues, beyond just the toolchain and DevOps, we're really taking a greater focus on our publicly accessible apps, and we're moving them up into the list. Not just the toolchain stuff, but doing more work around how we want to manage those as a platform going forward.

Sanjeev Sharma

Let me pitch my book. You all got a copy coming in, right?

In my book, I describe a model which we have been developing at IBM, which is along those lines, looking at risk versus value. There are certain applications which are very high value to your business and have very low risk tolerance. At the other end, there are things with very high risk tolerance, but the value is very low.

You start off with where you can show quick wins but minimize the risk. That way, you can create a spectrum, just like you did with your rankings, not very dissimilar. I think you have to go look at it that way.

There are two. For one, you want to show long-term business value, so that'll come with the high-value ones, but those are the ones usually attached to the low risk tolerance. You want to start with quick wins to start building that momentum. Getting those quick wins is very important.

When I was talking about that COE example, the biggest win was the third meetup we had, the third Friday, where one company came and said, "Hey, we used UrbanCode Deploy, and what used to take us two days to deploy, we can do it in 20 minutes now." People went, "Whoa, what did you do?"

Creating those quick wins is essential to create the momentum and overcome those barriers. You cannot overstate that enough. If you don't have quick wins, money runs out in nine months and you're going to be doing something else.

Chris Nowak

We took the same approach, sort of. We would look at the entire application portfolio, and then we would take an easy one first. Let's just figure out if we really understand what we're dealing with, because I've lifted the cover on an application and found 40 underneath. It's not easy.

Do one first, but also what you get from that is a learn, and you get a relationship with the application team. I'll look for a team that's willing to play ball and then say, "Let's do three apps. Let's do an easy one so we make sure we understand the technology we're dealing with. Let's do a medium one, and then if we feel comfortable with the risk ratio, let's go after the hardest one. Because if I can do that, I can do the rest of your portfolio, and let's line it up."

But it's going to be different for every situation.

Frank Canihuante

I think it also matters what technologies you're using, because at large companies, you may have mainframe, desktop, mobile, regular Linux and Windows applications. If you already have some patterns that take advantage of some automated tools to do the task, it's really not that much more to bring yet another app, no matter what its value to the business is, and just add it to the pipeline as a fully tested app.

Chris Nowak

Right. As long as your pattern evolved.

Sanjeev Sharma

If some app really fits your risk-value criteria, but it's a total outlier in the technology stack, obviously don't do it first, because it's going to add no value. You cannot learn anything from it and apply it to others.

We go in and see there'll be an ERP app or a packaged application, very critical to the business, but it's one of a kind in that company. There's no reason to do that first because you cannot apply what you learned to other applications. Common pattern is also very important. Good point.

Frank Canihuante

Plus, just about every large company will have different business units or lines of business, and for each one of them, they'll have their own favorite applications that they think are the highest-value apps that need to go and be fully integrated into the pipeline, no matter what the technology is.

Sanjeev Sharma

We have a question.

Q: Just a quick question on how you go through coming up with a taxonomy and some of the principles while making sure you're not crushing the culture you're trying to create, especially as open source is challenging some of the way we traditionally thought about allowing software development to happen.

Because now there's incredible switching costs that come along with altering a pipeline. How do we make sure that we don't get ourselves in a situation where a year from now, we're spending a ton of money because we're not thinking about what it's going to cost to actually alter that pipeline?

Sanjeev Sharma

One of those money questions.

Mark Peterson

That's not a quick question.

I'll answer the first part of it. The first part of it is altering the pipeline. That, to me, is sort of a founding principle of how we want to manage the pipeline. We do this in a way that we can switch in and out tools.

If not, the history has always been, once you put in a tool like this, it's there forever. We are very much trying to be of the attitude and opinion that we will be agile with our toolchain, and be nimble and be ready to swap it out if we find the approach isn't working or a better solution has come along.

I think that answers the first part of your question. Open source in general has many other facets of interest.

Sanjeev Sharma

I think the catch to that pipeline question is, when you bring in a new tool, think in a lean style of thinking, the Lean Startup way, that I'm building an MVP with the tool. I need to be able to experiment with it to see if it fits my company. Run some POCs, run some pilots to see if it's a fit.

But at the same time, you have to work towards lowering the cost of adoption of the tool so that you can lower the cost of replacing it.

If I'm giving my child a bicycle, eventually, I want them to graduate to a car. But you have to start with the bicycle first. I can't put my 10-year-old in a car. They won't let me.

You have to say, okay, today I can run this set of tools, but once my organization matures, I can look at Kubernetes. I can look at containerizing everything. But I cannot do it today because I'm not ready for it. I'm still on a bicycle with training wheels.

If you keep that long-term planning in mind, and that's something where we can come and help you create that 18-month, 5-year, 10-year roadmap. Then there are the unknowns. Two years ago, nobody talked about Kubernetes.

Mark Peterson

I'll add to that point. I think there's one more aspect that I didn't really talk about, which is part of it is an organizational and a real cultural shift. What we had not previously done well or invested in enough is core engineering skills.

To do what I just described, to be agile in the toolchain, you really need that core engineering team capability that is strong and ready to do that. I think that's an area where we're certainly investing.

Sanjeev Sharma

Frank, this is your company, so you want to comment?

Frank Canihuante

I actually agree with what they were saying.

We think of the toolchain, and I like to think of the enterprise toolchain because there are hundreds of tools that Travelers and Travelers-like companies of that size will use, but there's only a handful that really make up the backbone. There you've got your SCM, you have your CI, and you have your CD. Perhaps within CD you have deploy and release-type tools, and perhaps a ticket and a binary repository.

Those are the main tools, and those are the ones that you like to keep as a system. They're really a system of systems. Understanding one tool does not give you an understanding of the whole system. You try to maintain it, you try to grow it together, and if you make one change here, you need to keep an eye on whether or not it will affect some other piece.

You integrate them, and you like to keep them loosely connected or loosely integrated. As Mark was saying, I'm always thinking, in fact, if this tool will go away, what would I have to do to the rest of the pipeline to keep the whole thing still intact?

That is a different way of looking at tools. At Travelers, we're starting to think of it as a system. Systems thinking, like lean. It will grow as a system, not as individual tools. It's a different way to look at it.

Chris Nowak

I've always looked at it like a framework, even from way back when. I've got a Visio on it. We always said we want it to be, in fact, this was our phrase, a pluggable, swappable framework. This was back in 2002, we were talking this way. It was exactly for the reasons we talked about. We knew it wouldn't last forever.

Think of it that way, and let's be real about it. Nothing's free. Open source isn't free either. There's always a cost. I actually had a situation where I was called up, and it was a Jenkins plugin, and they said, "My StarTeam plugin doesn't work anymore." I go, "Well, I don't know anything about your StarTeam plugin, but the guy in Sweden stopped developing it. I'm sorry, I can't support that."

Nothing is really free. It's just always a trade. Understand what your trades are, I guess, is maybe the bigger thing to do.

The other thing that I'm kind of thinking about lately is SDLC. We always express it linearly. It's not. I actually see deploy as the center of the universe because everything radiates out from deploy. To me, it seems to be a hub-and-spoke.

I can change work item tracking tools all day long, and with a little bit of configuration, it's going to flow into my deploy and catch all the data. Build tool, you can still kind of swap it in and out. Artifact management, same thing.

You get to deploy, though, and it radiates out to everything you touch, your orchestration, your provisioning. There are higher costs with certain things that you want to put more thought into upfront. To me, the deploy process is, at least today, I think that that's really what you want to pay attention to.

Mark Peterson

That's the hardest to change.

Chris Nowak

It's hardest to change. Yep.

Sanjeev Sharma

If you want to learn more about UrbanCode Deploy, you can come to the IBM booth and learn all about it.

Question over there.

Q: I wanted to go back just for a moment, back to the standardization conversation. I'm curious to hear your insights on how you balance individual DevOps teams, the two-pizza-team type thing, their autonomy when it comes to things like architecture and those standardizations, and the balance between an overarching architecture team or centralized ownership of architecture versus empowering those individual teams. I'm curious to understand your guys' perspective on that.

Frank Canihuante

We've seen that many times, that argument: centralized versus decentralized. If you have a large organization, there'll be multiple lines of business or business units, and they will have their own architecture, their own DevOps teams, perhaps. It's hard to say to everyone, "This is the only tool, this is the only process, this is the only way to do things." It kind of kills the whole idea of having a little freedom and creativity and all that.

But left to their own devices, everybody will go their own ways, and you start to lose enterprise value. Once in a while, at least, it's good to meet and say, "Which direction are you going with your tools and your processes, and does it make sense to continue in that direction?"

This is where it helps to come to events like this to see where everybody else is going, whether or not it makes sense for us to go in this direction. Or out of the three or five different ways that people are going internally, maybe one or two of those are the way to go, and not all five.

Sanjeev Sharma

We are wrapping up right now, so let me see if we can take one or two more questions. There's one over here.

How many more do we have? Any other questions? Let's queue them up so we can quickly take a few, because we have three minutes left.

Shout it out, sir. I'll repeat it.

Q: What do you see as, in a heavily outsourced environment, with both development and infrastructure, how do you see that affecting DevOps adoption from a job and security perspective?

Sanjeev Sharma

That's a tough one.

Here's the challenge with heavily outsourced. I've been in many situations. I've worked for an outsourcer, which has outsourcer. When the contracts and the lawyers come out, all things go to hell.

If you can get around that and, right at the beginning, give guidance to your outsourcer on how you want them to engage with the DevOps lifecycle, things will work better. It's when you're injecting it midstream of contract, that's when they say, "Okay, let's go to page 739 and see how change management happens." That's when we run into trouble.

But I'm sure you guys have your own perspectives. Frank?

Frank Canihuante

I think it's kind of a necessary evil that you have to deal with. At the end, you do want to control the direction where you want to be taking the enterprise. They're there to help you and, in some cases, to work with you and collaborate. But I think you have to have the vision of where you want to have them work on and what tools and practices you want to have them do.

Sanjeev Sharma

I think it's that whole point that next time you sign a new contract or your contractors are up for renewal, the standardizations you have done in terms of process, if you started that DevOps journey, they are engaged and aware.

I remember doing one of my DevOps value stream mapping workshops, and they kept saying, "No, that's the responsibility of the outsourcer." Two hours into the workshop, I said, "Timeout. We are stopping this right here. Let's reconvene next week. Bring the outsourcer in the room, because I cannot do this." How can I do a value stream mapping where there's one big black box in the middle, which you have no control or visibility into?

We have one more question. Let's take that, and then we'll call it a day. There are two people with questions right next to each other. Same question. Good.

Q: Thanks. Chris, I was really happy to hear you say that a leader's job is to inspire. I think the other part of that job feels like it's coaching and developing. As you think about the cultural elements of what you're doing with DevOps, what are you doing to ensure that your leaders are prepared to coach, develop, and inspire?

Chris Nowak

One minute? All right.

It's your responsibility as a leader to train them to take your job. Make sure they're there. Give them the clarity, and I'm quoting from a book, Turn the Ship Around!, it's been mentioned a few times here. Give them clarity, which means you have to provide the why as to what they're doing. Give them the guardrails and get out of the way. Then make sure they're competent to do it.

That's a cycle. You keep training them up more and more. To Mark's point, you need to continue to grow your engineering disciplines. At some point, step out of the way and let them go. Now somebody else is the leader.

I don't know how else to express it in literally one minute, but coach them, train them, and get out of the way.

Mark Peterson

I'll add one thought to that, which is, I think when you're trying to do transformation at this scale and at this level of change for an organization, it's constant reminders of the very fact that you are trying to transform, that you are trying to get really what the goal is.

Remind people constantly that you're undergoing a complete change in how the institution operates and what that ultimate goal is. Really try to be clear about where you're ultimately going. Constantly reminding people, because it's so easy for teams to fall back into prior patterns and be limited by what they've done. Just limit themselves. People are constantly limiting themselves.

So I'm always trying to remind people what we're after in the end.

Chris Nowak

Yep. And you have to support their decisions. Let them know it's okay. Because if they do something and they get their wrist slapped, they're not going to do it again.

Mark Peterson

Yeah.

Sanjeev Sharma

Before we wrap it up, I need to make an announcement. If you would like to continue the conversation, IBM is hosting a reception this evening. If you see Mary in the back, Mary is there in the blue T-shirt waving. She has some registration cards. We have very limited seats. But if you would love to continue this conversation, we'd love to have you there. Please see Mary and get the registration form.

With that, I'd like to thank our esteemed panelists. This was a great conversation. Hopefully, you all enjoyed it. Thank you so much for your time.

Chris Nowak

Thanks.