Host Intro (Gene Kim)

The next speaker is John Rez, as we call him. He is from KeyBank. He is the SVP and Director of Continuous Delivery and Feedback. This is his third time speaking at the conference. He attended the conference in 2015, and based on what he saw, he did something utterly jaw-dropping and amazing afterwards. Then he shared the story in 2016.

He presented again in 2017 with Stephanie Gillespie, who leads development for the digital programs to support all of KeyBank’s retail bank. This year, we asked him to present alone because we want him to talk about his views of next-generation infrastructure and operations leadership. How does he define his job, and why does it always result in so many incredible and admirable things? I think we have so much to learn about how he views his role. Here to share that with us in our closing keynote is John Rez.

John Rzeszotarski

So, do you know what is awesome about following Jason Cox’s awesome videos? Nothing. Absolutely nothing. So my brain is mush. Your brain is mush. Let’s get through this. It’s called "Breaking Bad Leadership." Hopefully you get the joke. There is no periodic element for LE, by the way. And it’s a little misleading. What I mean is breaking perceived bad leadership, but we’ll talk about it.

Real quick, my name is John Rzeszotarski. I go by John Rez. I would not ever make anyone pronounce that last name. It’s ridiculous. I work in the infrastructure group for KeyBank. A little bit about KeyBank: we’re located in the Midwest, the Northwest, and the upper Northeast, but we have commercial locations throughout the bank. We have 20,000 employees. We have $137 billion in assets, so it puts us about the 15th-largest bank in the United States. We have anything from mobile banking, online banking, deposits, credit card, equipment finance, real estate capital. We have tons and tons of products.

It didn’t start that way. Our roots go back over 200 years. With that complexity came a lot of acquisitions, a lot of new services and new products, a lot of new systems that we had to build, a lot of new people and processes, and lots and lots of layered security. For an operations resource, it can feel like a ticking time bomb seeing all of this glued together. Then as an operations manager, your team starts sending you really interesting texts in the middle of the night.

On top of all that, we have to deal with something else that’s happening in the industry, and that’s all of these great startups building all these amazing things, releasing every day, building amazing new fintech technologies and companies. But you know what? We have to do the same exact thing, and we have to deal with all that complexity. Their job’s easier. Our job’s a lot--come on, we have to fill out 30 forms to get that server. You just did it by clicking a button. Come on, our job’s way harder.

This is the complexity that we have to deal with. This is just an example of the infrastructure systems that sit on the top, the monitoring systems that sit underneath, and then the teams on the bottom that utilize those. This is one of those optical illusions. You have to stare at it and make your eyes fuzzy, and you can actually see the silos within the organization.

We feel like our head is above water. This actually was most notable in 2014. We suffered a very significant outage, and we said, "Look, we’ve got to do some different things because we just can’t understand and unravel this complexity."

This really all culminated in us going into DevOps Enterprise Summit 2015. There’s Russ Clanton right there, and that’s him up on the screen. Their talk was amazing. It’s so relatable. You’re identifying these other organizations that are talking about the same exact problems you have, but they’re getting over the hump. They’re figuring it out. So we came out of here extremely inspired and saying, "All right, let’s put together a pitch. But the first thing we’ve got to do is make sure we understand those problems."

I think one thing that I’ve heard a lot in this conference is that it’s very contextual. It’s all different based on what your organization is and what they’re doing, but there are some major themes that are very similar. One, we knew we had a siloed organization. We knew it took us too long and had way too many manual processes. We had tons of tribal knowledge, and that was creating bottlenecks and all kinds of other problems. We weren’t talking among other teams. And most importantly, we weren’t actually trying to fix these issues. We were almost just admiring them.

We went out and picked a specific use case that we wanted to go work on. It was perfect timing because we were in the process of rewriting our online banking and mobile banking solution, and we wanted to partner with them. I think a lot of people will say, "You know what? Go pick low-hanging fruit to get started with." But if you want to make a meaningful and impactful transformation in your organization, pick the use case. These are our flagship applications. These are the most important applications we have in the bank today.

So we chose those, and we said we’re going to automate and re-platform the infrastructure components, and we’re going to move fast with them. We can’t move fast unless we test fast. One thing we also noted was, yes, we were building this new platform for our new application to run on. We also had to do the same thing for the testing platforms, and that was something we had to manage and take care of from the infrastructure team’s perspective.

Some of those test results that were so impactful were really, really helpful. We more than doubled our test coverage. What we used to call test automation took 20 hours, and we would only run it at the end of a cycle, and we never found any defects. We were running way more tests in less than 12 minutes, and we were finding defects anywhere from 10 to 20 to 30 during the middle of that development phase. Finding it way earlier in the process really allowed us to move fast.

We delivered the application early, and we had to because we were also in the process of acquiring another bank called First Niagara. We had to onboard 500,000 new customers. The one problem we had was we had this one enrollment page, and we screwed up. It was complex. It was convoluted. We just didn’t build it quite the right way. But this is where DevOps sold itself. We did 10 production releases in the first four days. We did them in the middle of the day. We did them during the highest volume of traffic KeyBank has ever seen. And we did that all with zero defects and zero customer disruptions. That’s something I’m extremely excited about. Thank you.

This really sold DevOps for us from a leadership team’s perspective, and a lot of teams came knocking. That was really part of that big transformational change that we chose. I’m really proud to say that Docker, Kubernetes, and a lot of these patterns and practices power all of our customer-facing applications today. To say a 200-year-old bank that’s the 15th biggest in the country is doing those things, I think that’s something I’m really proud of as well.

One other big success story we had was our account originations application. This is the front door for any new customer, and it continually changes and continually gets tweaked, but it’s also the front door for fraud attacks. This is something we always need to be able to change and need to be able to change quickly. This team is extremely appreciative because what used to take four hours on a weekend during off-business hours now takes less than five minutes, and we really changed that application. Big success story.

When I was trying to put together the thoughts for this talk, I was thinking, well, we could talk about some of the design patterns we’re doing in the infrastructure space. We can talk about how we’re using Kubernetes in all kinds of different ways. But then Gene had a different question for me. He said, "What would you say you do here?"

I wanted to answer this kind of matter-of-factly and say, "All right, my job and role and responsibility is I run our code and release management organizations. I run container orchestration. I do that not just for software development, but also for infrastructure development." Then we’ve gone on to include things like monitoring and feedback so that we can capture all those analytics and create those feedback loops, build new capabilities with cloud engineering, automate provisioning processes as part of cloud engineering, and then build IT service development to remove the bottlenecks, build IT self-service capabilities so we can move fast.

Just like Jason Cox said, we’re a platform team. We federate out to the rest of the organization. We’re the evangelists as part of that, but our job is solely to try to continuously improve the rails that the bank’s running on and building software on.

But that’s not what Gene was really asking about. What Gene was really asking about was, what is your role? I honestly define it as: our role is to make the bank better in every possible way. It doesn’t mean exactly where I sit within the organization. I might sit in the infrastructure space, but my job is to federate out and identify areas all throughout the bank and every single vertical within the bank. We do that through a couple of different ways.

Number one, we’ll go out and target specific candidates, and we’ll be very outward-reaching. We do that because we know some areas will very much take a lot of benefits to following some of these great principles, practices, technologies, and capabilities.

The second thing we do is what we call utilize modern tech to mark-to-market expense. We’re measured by an efficiency ratio. Normally, when we want to do new things, we have to keep the cost cheaper or keep them flat. I think a lot of people in this conference share that same sentiment. We’re at an interesting time, and a lot of people talked about it. Topo talked about it, but open source can drive that mark-to-market and get you fair market value for what you’re doing. In an enterprise, you would normally get strapped by what a vendor was forcing you to do. Now it’s table stakes. Now we can actually say, based on subscription licensing, based on the power of open source, how easy it is, the barrier of entry is extremely low to start utilizing these, and the abstraction and design patterns that you can bake into your solutions allows you to easily flip-flop a lot of these different solutions and platforms.

Lastly, we always are striving to identify bottlenecks and inefficiencies, and I’ll give you one example. One of my peers runs the help desk, and she was reviewing the list of top calls into the help desk. The top three calls were all related to password resets. We said, "Hey, you know what? We could probably throw together a quick chatbot to see if we can automate that specific use case." Once we got it up and running over a weekend, it wasn’t much longer after that that we were actually able to roll that out and reduce the counts. But then that kind of took off and spread a little bit more like wildfire, where it’s, "Well, hey, we have all these other great use cases," and everyone’s throwing out all these different ideas. Now we just kind of created a bit of a problem because we had to solve how we are going to capture the right intent or the right capability that they’re striving to do. So once again, we just kind of put on our smart hats and said, "Hey, let’s look at natural language processing and some of the AI capabilities that you can call with the cloud services." This is very easy stuff. This is not complicated, and it’s so easy to just go in and start utilizing services like that. That was another big success story.

The more I thought about it, there’s another big piece that’s missing, and that’s really about the power of persuasion. The other thing I think we do really good at the bank is persuade other leaders, or leaders that we might not think are doing the things we want them to do. Our CTO calls this the departments of no, and it’s things that we have to change and we have to break through. At the end of the day, though, it is a bit of perception because what got a leader in a position of leadership is what they trust and what they know that works for them. So you have to be empathetic in order for you to respond back and use that as part of your power of persuasion to change their minds.

I think it kind of goes into all of these different tendencies, and these tendencies can be representative of things like ego, poor planning, holding people accountable, or just enforcing and crossing standards. I thought what might be useful here is if I take these and wrap them up into some of the personas that we deal with and show you some of the DevOps case studies that I’ve used in the past to help that power of persuasion.

The first one I call the general. I think we’ve heard the theme of command and control a lot, and that’s Frederick Winslow Taylor. He wrote The Principles of Scientific Management, and Gene referenced Taylorism earlier on. It’s really around the power concentrated at the higher levels of the bureaucratic structure, and that’s where decisions are made.

But if I put myself in an empathetic position and say, look, I’ve lots of times tried to put a decision out to a group, and they can’t get an answer. That’s where command and control worked for me in some instances. So I need to think about that, but I also need to change that story a little bit and say, rather than command and control, it’s empower and delegate. There’s no better use case, in my honest opinion, especially for resources struggling with this, than Stormin’ Norman Schwarzkopf, who wrote the book It Doesn’t Take a Hero. In there he has the 14 rules of leadership. One of them says, "Don’t tell them how to do the job. Simply allocate resources, set standards, and the results will exceed your expectations." This guy was huge at driving autonomous teams into the military, and it’s an amazing use case and story for you to help the power of persuasion with some of the leaders that you’re working with.

Another persona is one I like to call the boy scout. These are folks that follow standards on a metaphysical level. Not really. That’s not really possible. But common sayings would be, "I know it’s not right, but it’s the policy," or, "What exactly is your charter, Mr. Rzeszotarski?" Anytime you get that one, you’re like, uh-oh, this is not going to be a good conversation.

So if I put that empathetic position back out and say, okay, this is the persona that wants to continue to utilize policies to secure the bank, to secure the organization, and it’s the right thing to do. That’s exactly right. But I think another way to flip that persona is Dave Farley and looking at the fact that I’m going to continually, safely test our controls through experimentation. The reason that’s important is because Dave Farley says, look, humans are always going to make errors. To err is human. We know that we’re not going to understand the complexity, and the only way for us to drive that accurate success is through experimentation. Lots of times, I just point people to videos of Dave Farley because I think it’s a great persuasive story in order for leaders to see the importance of experimentation.

The last persona I’ll talk about is the skeptic. Now, I use Scotty because he couldn’t bend the laws of physics, and he must have been skeptical about that. Common sayings might be, "We would have to engage," and then list several departments, or a persona that says, "That’s not something we have funding, time, or resources for." Hopefully you guys are relating to some of these stories if you’ve been working with leaders like this.

But this is the guy that actually knows all the requirements that drive your change. He’s your subject matter expert. He’s almost your product manager for change because he knows exactly what you’ve got to do and what needs to change. He just needs to understand that disruption is coming one way or the other, and that’s been a big theme at this conference. We’ve talked a lot about the importance of disruption and how it drives into every organization because every organization is a technology organization.

One other big part about disruption that I think is neat is if you look at things like improvement, and some might even classify innovation, it could be defined as doing the same things better. But disruption is doing new things that replace the old way. Honestly, I think that’s where DevOps actually sits. It’s also using DevOps to create more disruption in your own verticals. That’s the one thing I think people really need to understand. We’re not improving the process. We’re just doing a new thing that’s making everything else go away.

So I never really answered the question to Gene, what’s different at Key? Number one, safely empowering our employees, and it starts with our CIO, Amy Brady. She very much embodies embracing change agents, driving continuous learning and continuous improvement, and then understands the fact that we need new models. We talked about Mr. Kuhn up there, about how with paradigm shifts, you cannot follow those same old models. Whether that’s the same old networking playbook from an operations perspective or the same storage playbook that companies have followed for years, there’s a new model that we all have to get on board with. She wraps all of that in making sure that we all understand that we are the CEOs of our own careers. This really gives us that enablement and that ability for us to go out and operate with autonomous teams.

Another really cool thing that I think we do is transformational architecture. John Willis and I were talking about this one night, and we were talking about the role and responsibility of enterprise architecture and how in many organizations, it’s really just a governance mechanism. It’s around, are you following our tech standards? Are you not following them? I think what’s different at Key is we don’t approach it that way at all. We don’t have enterprise architecture; we have transformational architecture because we understand that we have to keep up with the times. We have to embrace these new things because they’re only going to help us. When you do really cool things with a lot of your systems, you get asked to collaborate on a lot of really cool things.

This slide was actually presented at Google Next, and we’re collaborating with them on the alpha implementation for Google Kubernetes Engine. It’s something my team is really excited about because what’s coming in technology is coming fast, and it’s great.

The last thing that I think is really, and it’s a no-brainer here, is people. It’s people that have been at the bank. It’s new people we’ve hired at the bank. Our people are willing to experiment, they’re empathetic, and I think one of the themes that I picked up on here was, you can be a half-cup-full guy or you can be a half-cup-empty guy. I think we’re more half-cup full, and that really is a testament to celebrating small victories. If you want to drive culture change within your organization, you have to celebrate every small victory that’s out there. That could be that one guy finally checked in a piece of code into source control. That’s a victory you need to go out and celebrate.

Because of the world we live in, I love the talk with Microsoft that talked about we now have ambiguous roles. We have to get comfortable with being uncomfortable. I think that’s so important. But the most important thing about that is you have to be courageous. You have to have courage. I love this quote by Meg Cabot: "The brave may not live forever, but the cautious do not live at all."

So if courage is part of the solution, then what I want to know from the rest of the community and the rest of the conference is, what are you afraid of? Because I think that’s what we ultimately get excited about. Sorry, my kids really wanted to be in the deck. It’s an embarrassing photo. I don’t know what else to say. But that is one thing we would love to hear about: what fears does your team face and your organization face, and how have you conquered it? Anyways, I went fast so I could make up for Jason. Thank you guys very much.