Charles Nelles

So, context for me: show of hands in the room, who's actually from the infrastructure or ops side here? Anybody?

Oh, my people. Cool.

Does anybody else feel like a zebra in a herd of gazelles?

I'll vamp while we get the slides going. That was part of why I put this together and tried to get here. It looks like, one, everybody's way ahead of us, so I'm just going to get used to that feeling. But the perspective I'm hoping I can bring that'll be valuable to somebody is, first, we're old. I mean really old. HP said they're 75. Wait till you get to my how-old-we-are slide.

Two, I'm ops. Always been ops. I've got that on another slide, but it's really interesting to look at it from that perspective. And it's a great perspective-bender to come to a place where my entire career is reduced to the last bullet on somebody's slide. Like, okay.

And then we're also highly regulated. We're a fintech company, and so we don't break rules, and they're making new rules every day. And so we're definitely in a position where we could punt, if you will, and say, "No, we can't do this. It's impossible." So this is a really neat experience for me, to come and be surrounded by folks that are working on this and to even have a story of our own.

So there's my slides.

What I'm hoping you'll get out of today, at least one of these, is some level of energy, or hope, or a new perspective, maybe some kind of a tool. And if you don't, hit me up, and if I've got something to share, I'll share it.

Hopefully, I won't say it's a unique perspective. I see a lot of similarities across the different stories I've heard in the presentations. But certainly, to come at it from the ops side has been a little different. And then with some of our other factors, like the regulations and age, it's also added some challenges, but also some benefits.

So this is me: 20 years in ops, ITIL guy, highly regulated environments from HIPAA to HCFA to SOX, and FDA, and PCI, and all that good stuff. So just a whole resume full of, "No, you can't." Right?

One of my challenges in the ITIL and ITSM space has been trying to get people from, well, process means we have to have a lot of sign-offs and a lot of no's, to really trying to establish process that's as lightweight as possible. And even before the DevOps stuff, it just didn't make sense to have a lot of these overbearing processes. A lot of people want to go fast, whether it's DevOps or not.

And so that was probably what drew me to this community and this concept, was the thought that, yeah, we can build process that has lots and lots of checklists, and approvers, and wonderful flows and forms, and we slowly just drive ourselves into obscurity and extinction by doing it.

But it is interesting to come from that background and also be the guy that a lot of times audit is coming to, and trying to be an advocate for speed, and agility, and flexibility. And just how much of that relies on the relationships with your partners, I've found to be amazing as I went through this with my teams.

We started like everybody else probably started. I'm not sure from the dev versus ops side, but we hear all these stories, millions of deployments per day, and you got to do a deployment every time you go to the bathroom, and all this other stuff. And the monitoring tells you that something's about to happen before it happens. And we're looking at all these companies saying, "Wow, those guys are great," and the whole unicorn concept.

Then we turn to our history. And when I say old, I mean old. Right? So these are the founders. We're like Hiram, and Ezra, and Jeptha old. Founded in 1851.

The interesting thing about that age--actually, I forgot one. The guy to the left. One of our employees actually invented the light bulb. That's Thomas Edison. I found that out about a month ago.

But one of the things about this that is interesting is we not only have the technical debt, we've also got this weird sediment of people, and processes, and things like that. And so layer on top of technology that we might have been propping up for a while--a long while. A while for us is decades.

But you put on top of that people that, when I got there, I knew people that had been at Western Union since before I was born. Right? So it's interesting to have folks like that with that much perspective. One, you get somebody like that with that much perspective and flexibility, you're in great shape. They've got all the history. They know where all the bodies are buried. You get the other kind, not so good.

So that's part of it, is that sediment of, "We've always done it this way." We had a lot of growth by acquisition and merger. So you'd pick something up, and you're moving so fast, you don't integrate it at the speed you'd like, or maybe you don't integrate it at all. It just leaves it bolted onto the side. Right? And it becomes that no-man zone where every time you look at it to try to improve it or integrate it with your processes, everyone begins to panic.

So a lot of this just turned into kind of what I call self-pity phase one, where we're looking at everybody else, and they've all got whiteboards, and they're in their hoodies, and they're zipping around and doing stuff, and we're just like, "Oh, there's no way." Right? I got Al. Al's been at the company for 30 years.

There really was a little bit of that, and I think, honestly, some of that comes with the infrastructure and ops world a little bit. Because if you're not careful, you can get into this mode of, "Woe is me. They're always dumping stuff on me, throwing it over the fence, and I've got to clean up the mess."

So one of the things I learned as we went through this is you really got to watch that, because that's not going to put you in a mindset where you can go collaborate with anybody. If you're put upon and thinking that you're the victim, it's just not going to go well.

The other thing that was a big part of us when we started to look at this was we were transactions-based. It was all about how many transactions today, how many did we lose in that outage, how many dollars per transaction, and that was just the DNA of the company. It's how we were brought up.

Then two new guys come in. We get a new CEO, followed shortly by a new CIO, and they start using the T word. And we're talking about transformation and innovation and all this stuff, and that's good. That's probably the fourth time that we've heard that. But we really started to do it. They changed how we were measured. They started talking about Net Promoter Score, and who's going to recommend us as a customer.

In addition to that, we had a healthy tailwind from the regulatory agencies that were saying, "You know what? You got to know your customer. You got to know your agent. You got to know who they've transacted with, when they transacted, what they're doing maybe on social media. Are they a risk to health, human safety, the national security?" All that stuff. There's just a ton of pressure when you're moving money cross-currency, cross-border, to know that stuff.

And so it was a complete mind shift for us to say, well, one, we need new systems that will tell us about our customer. Two, we need to partner with those folks that have been screaming customer, customer, customer for so long, and make sure we're bringing them that data.

But it also just created this kind of environment of, oh, okay, well, maybe we can switch the spots for the stripes. Maybe we can evolve out of the pond, whatever.

And so that was kind of the seed that started this for me. And then immediately I kind of got into The Phoenix Project, and we shopped that around with our ops guys and having everybody read that. And it was really the realization that we're not optimal. We're not horrible. We're not a crime against humanity. But we're certainly not optimal compared to what's going on today.

I think that was also a clear message from our leadership, which was a benefit, because while we don't--here's self-pity phase number two. You read the book and go, "Oh, well, I don't have a CEO fairy godmother to come and take me on the catwalk and tell the board what to do. All I've got is me and a bunch of outages."

So that was a big realization. It's like you get to the end of the book and you go, "Okay, this could work." And then you're like, "Oh, yeah. We don't have a wand. All we have is mess."

And so that was a big thing. Okay, realizing this is probably not where we want to be. Two, that I know Brent and Herbie. I don't know if everybody's read both or one of the books, but Brent's basically the one guy in The Phoenix Project that every project goes through. He knows how to fix everything.

We actually had two flavors of Brent. We've got the new Brent, which is he came on and he's Johnny-on-the-spot and fixing everything and not documenting anything. And we had the old Brent: "I've been here 30 years, and I don't care what your process is, and I don't care what you say, because if I leave, you're in serious trouble." And so that was part of our sediment.

But the other one is Herbie, who is the little scout in The Goal, where they're all marching to the campsite, and Herbie is three foot nothing, and he's got his pack full of anvils and soup cans and things. And so what they both represent is that first bottleneck, or at least that's the way I looked at it.

As I read The Phoenix Project, it was way too emotionally connected to my life, and so I didn't quite get it. It was a great book, read through it, but I didn't get all that I could out of it. I went to The Goal, which was about manufacturing, which I just don't care about at all. And then it was like, oh, okay, I see a whole lot more of this picture now that I'm reading about something that I have no understanding of, as opposed to, yeah, I know that guy. I hate that guy. That guy's killing me.

So that was part of it. And part of it is just looking at that and getting the concept that, well, we can sit here and whine about it, or we can do something.

And this story of the stonecutter--I really wish I could say that it came to me in a moment of inspiration, and I remembered a Chinese fable and it was meaningful. It was really listening to my peers whine. Everybody was griping about, "Wah, development, wah, wah, wah." And after I sorted this out, then the stonecutter thing came in.

This is this story about a guy's chipping away at the base of a stone tower, and he sees an aristocrat riding down the street on his horse, and his servants are fanning him. He's like, "Man, I wish I was that guy. I wish I had more power." And now he's the aristocrat. And then the aristocrat's riding on his horse, and now he's sweating. He's like, "Wow, I wish I was the sun. That would be so powerful, to be the sun." And now he's the sun.

Well, fast-forward through the sun, and the cloud that was shading the sun, and the wind that blew the cloud, and then the wind can't blow the tower. Now we're back to the tower. And lo and behold, we get back to the stonecutter. And it's kind of that realization that you could probably start this from anywhere. If we can get traction and momentum, anybody can. It's just a matter of figuring out how.

So that became a huge step, to get out of that kind of self-pity phase, that, "Oh, they'll never listen to us. We can never do this."

We started to look at certain things where we could make some traction. And also, we found how important it was to get to the full picture of what I call the assembly line. I think a lot of projects in IT today fail to get the results they want because they're dealing with that first bottleneck.

So you get whatever. You unload Herbie's pack, now somebody else has to carry that weight, but Herbie's still slow. Or you do the first step with Brent, or whatever it is, you remove one bottleneck, or you accelerate that, and now you just load up the station down the line. So until you get to that view, I think it's in The Phoenix Project, where he takes him up to the catwalk and sees the whole assembly line.

That was an important realization for us, because a lot of projects, you go in and say, "We're going to get this tool, apply it to that problem, and we're gold. Everything's going to be great." And really all you do is you move it down the line. So you end up spending that money, or at least this is what happened to us on certain occasions. You'd spend that money, and then you'd find out, well, wow, we didn't make it any better for the customer. These guys now have a lot of spare time, so wow, I can reduce their numbers or something cheerful like that.

But it was a challenge to say, "Well, let's look at that whole line." And then we fall into self-pity number three, which is, oh God, look at that whole line. We'll never get this done. So that was an interesting little evolution that we had going through there.

The next phase of it was trying to figure out, well, can we do anything? Should we just lay down on the ground and wait for the birds to come and eat us because we just give up?

And it was really thinking about what are you controlling? In my world, I have deployments, among other things. I've also got some ITSM process stuff, which was perfect, so we didn't need to change that. But the deployment stuff, we thought, well, okay, I can change that. And it seems like it's fundamental to this concept to be able to deploy quickly.

So what we did was we started working on deployment automation, but we started working on it from my perspective. It wasn't a grand, "We're going to do DevOps." It was, "I'm going to save $5. I'm going to go from two hours of work to 15 minutes of work."

Now, unfortunately, that two hours of work was part of a 16-hour chain. And so now instead of sitting around for six hours and then doing manual commands for two hours, we're going to sit around for six hours and click one button and then report that we're done and have everyone on the phone say, "How are they doing that?" But that was kind of the catch. We automated our stuff and started to get attention for, "What are you doing? How are you doing that? How do I get some?"

The other wonderful quote that I can't say came to me in the moment, but came to me later, was this concept of, to change things, don't work on the old stuff, work on the new stuff. As I get to the end of this, I'll do the big reveal, but it kind of began to paint a picture for me that became much clearer at the end.

But it really was looking at what we were doing today and going, "Oh, that's tough. How are we going to do that?" And maybe looking for a different option. How do we do something new?

And it's really about resistance. Because if I've got a massive manual process that I'm going to try to change, there's a lot of resistance there. It's going to be very difficult: people, processes, all that stuff.

And so the combination of focus on something new where there's low resistance, and also don't waste any energy, because we're in corporate America. Time is money, and money is precious. We have very little energy to waste. So you can't have a long, drawn-out project with a final reveal anymore. You've got to have incremental--it's much like the development stream. You've got to be producing value with every step. So those kind of started to sink in.

And then the last piece of this is data or get the heck out, or something else.

Data was currency for us. I think it is for everybody. That was the whole previous presentation with, I think it was Mark. It's just so valuable, and it doesn't have to be all-inclusive.

What we found was with our little tiny automation project, we took one process, we cut 85% of our time out of the picture. My CA guy's in the front, so I got to plug just a small amount. So using CA's release automation, we pulled 85% of the time out of the process.

But even more importantly in my book, we pulled 97% of the manual steps out. We reduced our opportunity to make a mistake, and we lowered that, I'll call it the hate threshold. "You guys suck." We lowered that because we're not making mistakes anymore.

But this became so important because this is something you can show to executives, you can show it to different groups upstream and downstream and say, "Hey, we got this. I don't know if that's valuable to you at all, but it's interesting. You want to come take a look?" And absolutely, ears perk up, and people want to talk to you at that point.

So we started there, just kind of that little drop in the pond. And then we started thinking about, how do we partner up with somebody to expand this? And for us, it was about, well, who's listening? Let's not start a conversation with a group who has no idea who we are, because, again, that's resistance. We're going to have to explain to them what this is, who we are, why it's important. Let's circumvent that. Let's not head into that resistance.

We actually picked the wrong partner first, not because they didn't want to achieve the same goal, but we didn't have the relationship we needed with them. And so their version of us helping them was them kind of crossing their arms and saying, "Yeah, you can help us if you want. We'll need to put this saddle on you and beat you extensively with this riding crop, then it'll be good."

They wanted the same outcome, but the relationship was so damaged from the past, it just wasn't going to be healthy. So we went through it. We automated what we could. We immediately got to a point where, "That's great, you automated our process, but we arbitrarily changed our process in the middle, so now we have to adapt." So it was kind of, okay, well, that wasn't great.

But we then started looking around for, okay, well, who do we have where we both have investments into the relationship? Who's our buddy? Right? And we found a group where we had just been, and all the IT ops people, we'd been in the foxhole together. Right? Outages, what do we do? DBAs on call, fixing things on the fly. We had just made it through one of those evolutions of, "What the heck was that? Thank God we fixed it. I love you guys."

Perfect timing. What do you guys think about CI/CD, doing some automation?

They're like, "Absolutely. You guys are the best. Let's do it."

So again, where's the least resistance? These guys love us right now. Let's grab those guys and get going. So that was a key factor.

And to be honest, our next step with these guys is we've been doing some back-and-forth use cases and stuff like that, and we're going to do a session here on Thursday with them and really dig into, okay, how do we solve some of this stuff? And to have a team that will come to the table with us, with the grouchy ops guys, and be able to say, "Well, let's throw anything on the board. Everything's fair game. Let's just see what we can do," is perfect. Right? It's that least resistance. It's the hole. We're going through the line, right?

The other thing we learned in searching for this partner and from our first partner is maybe don't try to help this guy first. Right? Don't help that guy to organize his books. Start with that guy. Right? These are not alphabetical. We'll help you with that.

But it's important, again, it's that resistance concept. If you go for too much, it's like you lose your momentum. You slow down. You're not delivering value. It's the, yeah, exactly, small batch size.

So the next place we kind of went and are going is this concept of teaming with these guys and having this level of trust.

We want to bend rules, right? But we cannot break any rules. Highly regulated. We're not breaking any rules. The rules are the rules. The key is to, one, respect your freedom. Right? That's coming from the ops guy. So yeah, respect your freedom. Don't go crazy out there with the code.

But to find a balance between complete order and control and some level of flexibility that doesn't fade into pandelerium. Right? Pandemonium and delirium.

But it took a lot of challenging to folks. In just the initial discussions to even get to the table, it was like, "Okay, well, let's pretend we could do that. What would that look like?"

"Well, if you guys would only let us do X, then we could do Y."

"Great. Done. Let's talk about that. Maybe we can let you do X, maybe we can't."

One of the big things was understanding the concept of rule and principles. I always made a joke when I was teaching ITIL: rules are for kindergarten and prison, because they don't have that level of responsibility right now to handle a principle. So in kindergarten and prison, I don't say, "We should be good to each other." I say, "Give me the scissors. No one gets knives."

In the principle side, you're actually setting out the ideal. Right? Let's try to achieve this. And what we found is in process as well, for the change process, whatever, you can go with principles. Then you can get away from, "Well, I need a VP's approval on any one of these."

Excellent. I've got all the VPs on speed dial. Let me just war dial them until I find somebody who'll say yes without thinking about it.

Shift that to, no, I need the right person to approve this. And if my guys don't agree it's the right person, then we can't approve it. If we do, then it's a five-minute approval. So that concept of the rules, which are the absolutes, never, always, whatever, versus the principles, which are, we're trying to achieve this. Right? The manifesto, it's principles. It's not necessarily rules.

And then the water analogy. I love this analogy that Bruce Lee had, actually. It was, "Be like water." Right? You must be shapeless, formless. When you put water in a cup, it becomes the cup. When you put it in a bottle, it becomes the bottle. Water can drip and it can crash. Become like water.

The cool thing about that--again, didn't come to me at the time, but you know Google, great--the cool thing about that is it tied it together for me. What we were doing through this whole process and continuing to do and having success at is finding the path of least resistance.

And I kind of backed up to a systems thinking level and thought, well, if you think about it, systems in nature that change, water breaking through a dam, river cutting through a canyon, that's all they're doing, is they're looking for the path of least resistance. And it kind of has become a principle for this after the fact, that we realized, wow, that's what we were doing when we were finding our friends to work with us. Right? When we were trying to don't bite off too much. Things like that were really just avoiding resistance and trying to figure out where we can make this change, and then continue to find the least path of resistance by encouraging people with data, all that good stuff, to continue and cement the change.

So we're still very early, obviously. But the fact that we've been able to get from warring tribes--and I mean warring tribes--to we're now going to sit down around a table and start working on some of this. We've also begun automating portions of it and showing that initial value is huge for us, especially with our level of sediment, and our level of technical debt, and just the past relationship. And that also seems to be a key to this, is that relationship is so critical.

Takeaways for me is we have a challenge with people stopping at the problem. Everybody wants to point out the problem. Trying to get people motivated off of that to say, "Good. Great. We got it. Write it down. Let's move on and fix it," or, "Let's avoid it altogether," was a challenge for us, but I think it's critical. You have to get to that point where you can go right to the solution. It might not be an easy solution, but the more time you spend wringing your hands over the problem, every second is wasted when you could've been moving towards the solution.

Finding your lever and bringing the data, got to have both. You got to figure out what you have influence over and bring data to prove it, because that data speaks for itself. That I don't have to sell. That gets out and people hear rumors and they're like, "Wow, 85%? They took it from two hours to 10 minutes? That's crazy. How do I get some of that?" That's what you want, is that rumor mill working, as opposed to the opposite effect.

And then for us, we learned a big lesson in finding that partner. We needed somebody who was least resistant, friendly to us, somebody who we had some credibility with, where we had made some investments, they had made some investments, and now they're coming wholeheartedly.

The other piece that I didn't mention there is there's another side of the partnership, and those are the vendors, and getting the right vendors at the table. Generally, the battle is a lot of the vendors will say, "Yeah, we can do all of that."

Yeah, but should you?

And our vendors have been very good after some back-and-forth conversation about, let's be realistic about what we can do versus what we should do. And let's come together to solve the problem and decide where can we help, and where can we back one vendor off, bring another in, or have no vendor.

And then for us, the other thing that Gene asked for was, where do we need help? Our biggest challenge right now is we're trying to kill Rube Goldberg. When you get into automation, the ops guys, we naturally gravitate to, let me automate this whole thing in a way that it can never be used again because it's so specific.

And we're having to fight with our own engineers to say, "Okay, I understand you're solving this problem very well, but we can't solve this problem anymore. We have to solve problems in the future." So trying to build a framework that we can hang some of this stuff off of--because as a shared services org, I have 50 development teams, and these guys do it completely different than these guys. I've got to serve both. And so trying to get my engineers and the automation guys to really turn that into a reusable framework is our challenge.

All right. I just got the wave, so that's it for me, unless anybody has any questions. And like I said, if you didn't get something out of this, jump me in the hallway, and if I've got something, I'll give it to you.

Bravo.

Thank you, guys.

Q&A

Lunch, though, so if we want to hang around, ask questions, we can.

Q: My question is, how did you deal with pushback from up above?

A: We were stealthy.

Because if you remember, we started as, we're just going to automate some deployments. And then once we got that deployment automation capability, then people started saying, "Well, I could hang test automation off of that and other automation off of that." We have another tool where we're using pre- and post-deployment to check configurations and watch config drift, and we could use that.

So it was really under the cover of, we're just automating our work, we actually established a beachhead and then began to get interest. And so if you go and say, "We're going to change the way we do everything. It's going to be great," that's where you get a lot of resistance.

If those folks are going to their leadership and saying, "I need some of what he's got. Can I..." Then it's a lot easier to break through that. So that's what we're seeing, and that's how we got these guys to the table later this week. They're one of our most critical apps in the company. They're part of our risk decisioning stuff. They've got 14 terabytes of data that they're going through every time we do a transaction. So big-ticket item in our world, and we were able to get them to come to us.

And it's funny because they came to us and said, "Well, we know you guys won't want to do this or this or this, but if we could talk about it."

And we were like, "Let's do all of that. As long as we don't break any rules, violate the Constitution, whatever, let's figure it out, see what we can do." And I think that's a big difference as well. If you go into it with that, well, what can we figure out? It seems to lower the resistance.

Awesome.

Thank you.

Q: Thanks, everybody. I just had a quick question.

A: Oh, yeah.

Q: The session you had tomorrow, was that you or a vendor, or are you doing another one tomorrow?

A: I don't think so.

Q: Okay. I'd like to mention it to my--

A: I'll meet you out on the street corner because I'll keep going.

All right, then. Thanks.