Ralph Loura

Good morning.

So we're going to do a little bit of a maybe different presentation. We're going to run through 30 minutes with three speakers and, I think, 30 slides. So it's going to move quick. Trust me, it'll be fine. Why don't you tee us off with the first slide?

First of all, HP: a big place. We're literally 12 days away from being live as two separate companies, so it's nice to be here instead of in a war room somewhere figuring out why something went wrong. So far, so good on that front.

This is sort of the numbers slide, the big scary "how big we are" slide. I won't drain it entirely, but there's a lot going on at HP. As Gene said, 300,000 employees, a massive amount of infrastructure, data centers, just a little bit of activity in the last nine months cloning and separating and splitting and re-IPing everything on the planet. So big scale going on in the background here.

How do you organize an initiative like DevOps in that kind of context?

One of the challenges we have is we're a big 75-year-old company, and we're organized like a big 75-year-old company. Yet at the core of our DNA is invention. In many ways, HP is the founding of Silicon Valley, right? The garage, Bill and Dave. This is kind of where it all began.

We believe in invention and creativity and being different, yet organizing 300,000 people against a set of work in a predictable way with rational outcomes requires some sort of structure. So we're very much a traditional kind of IT shop and a dev shop in the way we're structured organizationally, which presents a bit of a challenge.

Part of that challenge is kind of what we're here to tell you about today: how we address that and the progress we've made.

Olivier?

Olivier Jacques

Yeah. Indeed, like many of you, I guess we started with unicorns. We have many projects in HP IT that have been looking at doing things differently and looking at taking advantage of the cloud capabilities that we have in-house, taking advantage also of the new automation capabilities that became possible.

So lots of unicorns that actually started to assemble. What we started really to do then was to look at all of those unicorns and take them and assemble, define our new world. We knew that DevOps was the thing that we wanted to do for many reasons, but we didn't know really what it meant for us, for HP IT.

So we did this pilot and this kickoff, and we'll share more about that, to define our own HP IT DevOps manifesto. In Agile, there is a manifesto, the Agile Manifesto. It's very well known. I know that there has been lots of debates about having a manifesto for DevOps, but we thought that for us, we had to make our own and create our own manifesto, something that we can rally around and something we can understand. I will share also more about that.

So what's our new world? How do we enable this parallel world that enables DevOps? This was really our journey so far.

Where we are right now is that we are scaling this up. We onboard more assets. We are looking at actually having more teams joining this DevOps mode because it is indeed a parallel mode that we are enabling for now.

One way that we are looking at to enable that is what we call the DevOps Academy. The DevOps Academy is a mix. It's not something that is super formal. We have very basic 101 courses, like just get the basics, but it's all about enabling champions and enabling change agents, learning by doing.

I think that the presentation yesterday from Target was very interesting from that standpoint, but it's all about enabling this community of DevOps practitioners that we have within HP.

Ralph Loura

The basic idea is we started small. We didn't take 4,000 people in IT dev and reorganize them in a DevOps model and run them through training, right? That would be silly. I didn't take 1,000 projects and immediately apply a new model and say, "Let's go see how fast we can go."

So we took a few things that sort of fit. We did a kickoff and a pilot. We got a bunch of people in a room.

I remember back in the old days when Voice over IP was a novel technology, we had the IP stack guys, and we had the Signaling System 7 guys. They didn't talk to each other. The two orgs didn't trust or value what each other did. The only way we ever got Voice over IP off the ground, and unified communications, was the equivalent of organizationally taking both orgs out in the parking lot, letting them sort it out, and then kind of the winners came back in the room.

This was what we did here, right? This was our kind of parking lot conversation. We took the ops team, we took the cyber risk team, we took traditional developers, the database guys. We had folks complaining that, "Gee, that grid and that cloud is not reliable enough for me to run my app on it yet. So once it becomes five nines reliable for that particular node, then I can move my app."

And I'm like, "Well, then you should probably leave because we're never going to get there. The answer is, you need to fix your app to understand that you're managing in a grid-based world and not assume that you've got a fault-tolerant engine. So you don't get it. Let's move on."

So it was kind of this: get them all in a parking lot, sort it out, get them back in the room, and see who wanted to play for the future.

Rafael Garcia

I think one of the key things was that Ralph was actually in the room the whole time. Having that CIO sponsorship in the room with all the players, when those kind of misgivings start to come up, Ralph was able to say, "No, this is where we're going. This is the place." It made a big difference.

Olivier Jacques

Right. This really helped.

Throughout this, our experience report that Gene was talking about, we are going to share techniques and things that worked for us. But we also wanted to make it real, so we are going to talk about two of the applications that we are using and that actually went into this DevOps mode. There are more than that, but two applications.

One is a mobile app. This is an app that is in the pocket of all of our HPE vendors, or salespeople, sorry, and that's 10,000 salespeople. Any progress, any change that we make there is good for our revenues.

Ralph Loura

One of the keys here is you want to pick an app that matters, because you don't want to just do some little widget on the side that nobody cares about.

An old boss of mine used to say, in a very, I think, tender and endearing way, salespeople are coin-operated. At the end of the day, you want them to be, right? If you build your incentive plan correctly, then you incent people to go do the things that are going to grow the company and meet customer needs and so on.

MyComp Mobile was our answer to that. Before this, what we had was salespeople wondering, "Am I on quota? Am I not on quota? Where am I? Do I need to sell more cloud? Did I hit my All Flash quota? Am I going to get my payout? What percent?" They're spending way too much time with their personal spreadsheet arguing with sales ops on whether or not I made something.

This way, it's real-time. It's in their hand. They can check it on the fly, understand what do I need to do this week, today, this month to crush it, to kill it, to get paid, and meet the needs of my customer. So kind of hit them where the heart is.

Olivier Jacques

Right.

Ralph Loura

Or the wallet, in this case.

Olivier Jacques

Right. One of the other applications that actually is business critical is what we call support automation.

It's an internal application that is really a social network between all of the equipment that we sell to customers, so on-premises, and with various people. So the system administrators, the support analysts, the service delivery, the sales, so that we can have a mesh and really understand where our systems are breaking in the field and also predict when parts are going to break as well. So it's a very key capability that we have.

Rafael Garcia

I think one of the key points here was that these are just two examples of the DevOps ones that we tried out in the initial pilot.

The reality is we had everything from microservices built to monolithic solutions. We had mobile applications. We had every type of application that we wanted to be able to try this out with, versus just thinking it was the unicorn stuff, the web-based native apps.

Ralph Loura

And we actually started with SAP in a former life.

Olivier Jacques

Yeah, we have SAP starting now, too. I can share about that.

Sharing about all of the techniques and all of the things that we have learned so far, we wanted to highlight some key points there.

Rafael Garcia

One of the first ones we wanted to talk about was collaboration.

At first, we were wondering, we know we want to break the silos. That's the whole point of starting some of this stuff. But we were wondering, does that mean we have to reorganize in order to make that happen?

We'd heard a lot of people talk about a DevOps organization. We weren't sure that quite was right. We heard a lot of people talking about changing your org and combining people, changing management structures and the like. We weren't sure that was very practical for us. Like Ralph mentioned, we're a very large, distributed kind of organization. On top of that, we really didn't know where we wanted to go. We didn't know that this is the target organization we're going for, so it didn't make sense to make a big change.

Instead, what we chose was ChatOps as a mechanism to do that. I'm sure many of you know what ChatOps is. There's talk of it downstairs in the demo site as well. But if you don't, I would highly recommend going and finding out about ChatOps. For us, it's a massive enabler.

Basically, what it is is persistent chat rooms that are combined with bots, interaction with the system, the environment that's associated to the particular service or application.

Ralph Loura

Shameless plug: tomorrow afternoon, if you're really interested about this, some more HPE folks are going to be talking about ChatOps in particular and what we've done in that space. So you can catch that talk. I think it's at 2:00.

Rafael Garcia

It's 2:00 in this room, Room B.

Ralph Loura

In the afternoon. And it's a live demo, which I've not seen many live demos yet, frankly.

Rafael Garcia

So yeah, we're putting it all out there right on the line.

Ralph Loura

The other thing that's interesting: this kind of idea was frankly inspired by the old internet meme that on the internet, no one knows you're a dog. Basically, you can be anything you want kind of behind the screen of a computer.

The idea was, hey, listen, I don't have to reorg an IT org that's literally 8,000 people spanning the globe in order to figure out how to do something like this.

By day, I'm a third-shift warranty support operator. But by night, I'm a ChatOps expert that's driving code delivery across a certain pipeline. So on the internet, no one knows you're a ChatOps dog, I guess.

Rafael Garcia

Some of the other aspects of the ChatOps stuff: it's not just a place where people come together, right? The whole concept is that now you're bringing in the environment itself.

So everything from what's the status of the build, what kind of tests are passing, is there a production event, what kind of self-healing maybe is happening, all of that is coming into the same place with everyone that's a part of this ecosystem reacting or interacting real-time.

The other aspect of it is that that is what engenders the trust across these different organizations that, in the past, were really playing the blame game. Now you have everything very visible and transparent. Everyone's able to see and interact together.

Olivier Jacques

Right. DevOps is really about collaboration, and it's really about trust enablement. This brings a lot of visibility, and many things become transparent thanks to ChatOps.

The other aspect is that we are social animals. That's what we are and that's who we are. In ChatOps, you get the opportunity to talk to each other. You get to have fun. You get to ask the bot to make fun of someone else if you want to.

But the other thing is that you have the systems that plug into this social collaboration. So we are social by default, and on top of that, we have the system that does help us do work, which is not the other way around. We are not living our lives in the tools and having social on the side. No, we are social first.

Rafael Garcia

It's not just about the social aspect and the fun part. That's a really big one, an enabler for the trust piece. But by having the systems engage, you avoid a lot of the surprises that were occurring with us in our traditional model.

For instance, we would have someone from a support side reacting to a production event, unaware that something's happening on the development side, pushing into production. Miscommunication. There are supposed to be policies and processes to avoid that, but the reality is those things break down.

In this environment, everyone sees what's happening, and they're able to raise their hand and go, "Hey, why are you doing that?" That's helped us avoid a lot of surprises as well.

Olivier Jacques

Exactly.

The other aspect is pipelines. There are many talks about pipelines. It was really a key and a very important point in many talks. If you have not bought the book Continuous Delivery by Jez and others, it's really a must-read if you are technical.

But pipelines, it's a set of tools, and they are very important. We also think that pipelines, what we have seen is that we could come with a very well-defined pipeline, awesome, rock star. There are people in OpenStack that are doing awesome things with their continuous delivery pipeline, and say, "This is going to be the pipeline going forward, end to end, for all of HP IT."

We think that one size does not fit all. Instead, we are looking at having something like a flexible continuous delivery pipeline, and that's what we have enabled.

So we enable the teams to basically have whatever they want. There is a lot of flexibility. However, we have a few anchor points, and those anchor points are kind of non-negotiable anyway.

Gene was mentioning about the source code management, so like Google has one code repository. At HP, we also have one code repository effectively, and this is kind of non-negotiable anyway. We have those few anchor points that are very important to us and that allow us to allow flexibility while, at the same time, bringing ability to be different.

Rafael Garcia

This was particularly important for us because we had a number of pilot teams. The ones we chose to participate were already fairly mature teams. They were already doing continuous integration. They were starting into continuous delivery.

We didn't want to come in and say, "Okay, you're doing it all wrong. This is the set of tools and pipeline and processes you have to follow." It was more of, what is the challenge you're running into, and what can we do in order to remove that challenge?

Then from there, what we expect is some kind of commonality across these pipelines is going to surface.

Olivier Jacques

Yeah.

Rafael Garcia

Over time, we'll start understanding what are the choice points that are the best for us.

Olivier Jacques

Right.

In this pipeline, I was talking about source code management. Maybe a little bit of zoom on this one. One change is one deploy.

The cool thing about, we have two kind of models. By the way, the war is over. The war on source code management tools is basically over. Someone has won, and this is Git. If you are doing Subversion, this is hard for you, so I'm sorry about that.

In Git, there are different workflows. There is a Git workflow, and there is a pull request, GitHub kind of workflow. We have both. But bottom line is that each and every change is one deploy, and we have lightweight peer reviews.

I think what is also very interesting is we also have the ChatOps integration. Same thing, right? Every time there is a code commit, every time there is a review, every time there is a comment to a review, this gets integrated in our ChatOps system. So it's all very visible, and we can always refer to it, so that we continue to add on top of it.

The other big anchor point that we have is our change records. In our existing world, we have a big process that we call the RFC process, the request for change process. That's basically the CAB, very optimized in the manual world, very optimized when we have teams that need to be interlocked with each other so that it can get work done by another team.

But in a world where many things are automated, if not everything, this is not the best way of working.

So we have this change record mechanism that we use to basically fulfill the same needs, in terms of audits and traceability, et cetera, but we take advantage of all of the automation that we have put in place. That's our system.

Ralph Loura

At times at HP, the RFC is a four-letter word. If you're a developer, or even in the ops team, this request for change and this moving records through can be perceived as being somewhat bureaucratic and frustrating at times.

So A, this is a delight to move from the manual paper-based, committee-based process to one where I push a button and a predictable thing happens if I followed the right sort of policy. Then two, as we said, it's an auditor's dream.

At the end of the day, instead of kind of parsing through logs and runbooks and the minutes of a meeting and other areas where we store and record decisions, I literally just pop up the ChatOps. I can go see what happened, who did it, when it was done, why it was done, the dialogue around it. It's amazing.

Rafael Garcia

The other thing on that change record is that we involved the auditors right up front. As we defined what this service would look like, we didn't just go into a room and say, "Okay, we don't need all this request for change stuff. Let's go and decide what a change is."

We had the auditors in place, the SOX folks, the security folks, and started to really understand what is the bare minimum you need to understand about every change that comes through versus this six-week lead time, 40-artifact thing that we have to create every time.

Ralph Loura

Yeah, they were some of the people in the virtual parking lot in that first kickoff.

Rafael Garcia

Yes, they were.

Olivier Jacques

Telemetry and the feedback loop and the second way that is very well depicted in The Phoenix Project book is very important.

This is the telemetry for the, you remember the MyComp Mobile application? We can see where our users are going, in which screen, how they navigate between the screens. That's very interesting.

Lots of telemetry, I think. But the other one where we see here, it's our performance, depending on the phone that you have in hand. We know that we have an iOS 9 issue on the iPhone 5C. We can see it.

But the other concept that I think is extremely important is what we call the Fundex. The Fundex is something that, basically, it's a composite metric for mobile apps. But you can create the Fundex that you want.

In mobile, what you want is speed. You want an application that is fast. You want an application that doesn't crash, and you want an application that doesn't drain your battery nor use your data plan like crazy.

So we measure all that. We aggregate, and we use it as a composite metric that we call the Fundex. Because you can drill down, but at the end of the day, if we can just have one metric that kind of summarizes all of the aspects of it, that's really something that is very interesting for us.

Ralph Loura

That's one of the things guys like me look for, is a nice, pretty dashboard with colors and bars.

Olivier Jacques

Yeah. Looks clean, huh?

Ralph Loura

It looks good. It's green. I like it. It's good.

Rafael Garcia

The other aspect here is that this is just one example, right? The whole point is that the pipeline has a feedback loop built within it, and each of these teams have a flexible capability of building out the monitoring, the self-healing, whatever they need in order to apply to their particular situation.

Versus what our traditional models were, which was you establish some kind of central monitoring system with some basic functionality that works across the board, and anything beyond that that you need is like pulling teeth to be able to enable it. This enables the teams themselves.

Olivier Jacques

Absolutely.

Ralph Loura

One of the concepts we talked about, so we talk about pipeline, we talk about anchor points.

It's interesting. I think if you're too prescriptive, you turn off a lot of those people that, at the core of their DNA, look at themselves as inventive. If I tell you, "Here are the tools you use, how to use them, when to use them, where to use them," it feels too rote.

Frankly, a lot of our creative engineers, even if it's the best tool or the best process, almost on principle will go find something else to use. You guys, you get this, right?

Our philosophy is essentially buoys, not boundaries. We're not drawing hard lines, "You have to stay within the lines." It's really this idea of, okay, we're going to kind of cut a channel fairly deep. That channel is a kind of safe harbor. If you go down the channel, middle of the channel, you know you're going to make it safely into port, your code will launch, and you'll be fine.

If you choose to, you can kind of test the waters, right? You can get to the edge of the buoys. You can kind of maybe even go 10 or 15 feet past the buoy. There's probably a little safety zone where the shoreline tapers off.

So you can go experiment with different tools, different techniques, different approaches. As long as you're staying true to the anchor points, those are non-negotiable, and you're consistent with our manifesto and behavior, we'll let you go experiment.

In fact, how are we ever going to see the next interesting thing...

Olivier Jacques

Right.

Ralph Loura

...the next innovation, the next thing that allows us to get there faster or better or with a better customer outcome, if I'm not exploring and testing at the edges?

You've got to find a way to navigate the channel, mark the channel, and then give people permission to test where appropriate.

Rafael Garcia

That kind of goes to the last point here, which is about trusting your people, trusting the organization.

One of the aspects that we had was we worked more of a command and control kind of model, which I think is pretty traditional, and everyone's built into this, where you establish a central policy or a central process, and you're looking to avoid or prevent people from making mistakes, essentially. You're not trusting the people to have the right visibility, the right knowledge, to be able to make the right choice for themselves. So you establish a process that steps in and does that for them.

Instead, we moved more towards this concept of integrated and empowered teams. Integrated meaning that these folks had, for any particular application or business service, everyone in the team that actually would be able to touch or feel a part of that ecosystem. So that team is self-sustainable, essentially. It's not a team that has to reach outside to be able to correct an issue. That included DevOps, but then pieces of Ops, like support and network and DBA, et cetera.

This is integrated. They're actually working real-time basis continuously.

In addition to that, empowered. What do we mean by empowered? For us, it was getting the right skills in the room, in the team. Getting the right access, being able to access production, for instance. Getting right visibility. That's where ChatOps came in. ChatOps gave the transparency to what's happening in the environment to all of the players, so that way, the DevOps teams have the right visibility to make the right choices.

We brought this back to this whole concept then, if policy and process is not going to drive this, then how do we approach every time that we have a decision on what we should do?

Basically, we fall back on this minimum viable process. Instead of resorting to a process as the first default kind of mental reaction, it's empower the team. What does it take to give that team the ability to make the right choice? Because that team is the closest to the situation, the closest to the issue. They typically know the best thing to do for their particular situation.

We consider that so critical, that whole trust the people, that as we developed our manifesto of defining what DevOps meant to us, trust is at the foundation of that. You'll see it at the bottom, and it really means everything. It's the fallback position on everything that we do, is trusting the people closest to the situation.

Olivier Jacques

I think this is interesting. The manifesto idea was, okay, what do we have to do, right?

We were talking about all of the techniques, all of the organizations that we needed for the teams to buy into, and we thought that we had to have a vehicle to communicate what's going to be their new world.

We wanted to make sure that they took advantage of, they are really jumping into something that's going to be different. You are going to be trusted. Yes, the interfaces between the silos are going to be APIs, and it was an awesome discussion yesterday in the morning session.

So it's a new world, and it's a parallel world. This is the manifesto. This is the contract that you have to sign. And yes, you will be on call as well. So that's scary.

Ralph Loura

One of the key questions I get a lot as a CIO when I talk to other IT leaders that maybe haven't started the journey yet is, "Hey, is DevOps just another way of doing work? Isn't this just like another methodology? It's ITIL, it's Waterfall, it's Agile. What's the big deal?"

My answer is typically, yes, it is just another methodology. It's just another way of working, and it's a fundamental belief system that changes everything. So it's both, right? It depends on how you look at it.

It requires a different level of thinking, a different level of leadership, a different commitment to a process. That commitment has to be pervasive. It has to be vertical. It has to cut through the whole organization. It doesn't necessarily have to be kind of horizontal. It doesn't have to be everywhere all the time.

One of my favorite tech or sci-fi quotes is, "The future is already here. It's just not evenly distributed." That's probably true at HP, right? We're doing a lot of things in innovative and different ways. DevOps is one of them. But we're not doing it everywhere all the time in the same way, right? We want the freedom to be able to do things differently based on maturity of the team and appropriateness of the situation and the problem.

On the other hand, when we're talking to the folks that at their core believe this is a movement, it's very easy to connect with those people and drive that as a movement. But when I'm talking to the people who just want life to be simpler, just, "Can we just stop all this change stuff? It's kind of getting out of hand," I can say, "No, it's just another way of doing work."

If you think of the way you build code as, "I've got to think about what I'm going to do. I'm going to design what I'm doing. I'm going to code it. I'm going to test it. I'm going to deploy it. I'm going to support it."

When we went from that Waterfall, we went to Agile. Instead of doing all those steps in a room with a bunch of people reviewing at phase gate, I did it around a table, and we called it things like a scrum, and I did it in a few weeks.

Then in DevOps, I'm just doing it in a few seconds or a few minutes, and all those steps are happening as well. They're just happening in an automated way or in a collaborative way or a different way.

So if I'm talking to someone who doesn't want the world to change, I can talk about it in comfortable terms about evolution. And if I'm talking to somebody who's ready to change the world, we can talk about the cultural change that's required to make this all happen.

Our key takeaways, as Gene requested. We were overachievers, so we only came out with four instead of five. We kind of condensed this for you guys.

You guys heard that trust is essential to change the way we work. Without trust, none of this kind of comes together, right? Because if I have to think about it ahead of time, set policy, and do all this kind of heavy-handed work over top, it doesn't really open what we're trying to open up here.

Enable collaboration across silos using ChatOps. Again, if you're an org like us who can't essentially take everyone that's doing this work, put them in a room, and just reorganize everything around this work, you have to do this iteratively, globally, in silos. ChatOps, we found, is a great tool to kind of... No one knows you're a dog in ChatOps.

Encourage experimentation. Buoys, not boundaries. Really make sure you're giving people license. Yes, you have to have anchor points. Yes, you have to have clarity around pipeline. You have to have the religious debate and work through that. But encourage experimentation where possible, because that's how you find the next thing that drives value.

And then, of course, a balance between trust and control. Collaboration isn't a free-for-all. Trust isn't a free-for-all. This is about creating equity and balance in terms of how much risk I'm taking and how I'm countering that kind of work.

Olivier Jacques

I would just add for the key takeaways that actually we are collaborating a lot with our professional services at HP. We do actually that for our customers.

The journey that we are in, in HP IT, we are working with our professional services to also share it with our customers, and hp.com/go/devops is the place where we are condensing a lot.

Ralph Loura

Yep.

How can you help us?

When to apply DevOps, right? We keep getting this: do I apply DevOps to my ERP rollout? Do I apply DevOps to just kind of unicorn-style apps? Is there a method, a methodology, a filter, a rule set, a flowchart that helps figure out kind of what methodology applies best to what kind of work?

Then we're still struggling a little. You see we've got metrics, right? So metrics to measure DevOps. But it's really more from a leadership perspective. How do you create incentives? How do you incent teams? How do you measure and create incentives so that people are working in the right style and achieving the right outcomes in a DevOps-style world versus the measures relative to things like number of releases and so on, which are kind of easier to track?

So those would be our asks.

I think we're kind of right on time. Thanks, everybody, very much.

Olivier Jacques

Thank you.

Q&A

Q: Thank you so much. Actually, I have one question.

One of the things that really stood out in our prep calls was this phrase, Ralph was always relentlessly helping break down barriers and obstacles, right? Whether it was security compliance or the daily coffee. Can you talk about that? I'm not sure who wants to speak to this, but what did that look like? What did it feel like? How did it help?

A: Olivier Jacques: I think it was a lot about enablement.

We had this parking lot, and the fact that, yeah, there was a question about security. How do we access the system securely? How do we make sure that our systems are not going to break down everything?

The fact that you could have the point of view of the CIO that says, "You know what? This is going to happen. It's kind of not negotiable also. This is going to happen, and we just need to find a way to do it in a way that's right."

A: Ralph Loura: Maybe this goes back to my... In college, I was an offensive lineman, so maybe it's in my DNA that I go clear the path to let other guys come through.

Q: Awesome. Hey, thank you so much.

A: Thanks, guys.

Thank you.