Host Intro (Gene Kim)

All right, the next speaker is John Rzeszotarski. I met him in 2016 because of something remarkable that he did. He attended this conference in 2015 as one of the SVPs and director of continuous delivery and feedback at KeyBank, and after that he did something so utterly jaw-dropping and amazing that it blew away the entire programming committee. He shared that story in 2016 on the plenary stage. I love those stories because it shows how quickly ideas within this community can be shared and how quickly similar outcomes can be generated. This is one of the key marks of an incredibly productive scenius, which I talked about yesterday.

He later presented in 2017 with Stephanie Gillespie, who led the digital program supporting Key's retail bank, and he also spoke in 2019 about this crazy idea about automating technical governance.

Since then he's moved on to PNC Bank, serving as their SVP of technology infrastructure. I'm so delighted that he's here to speak about where he's taking those ideas, which is at the heart of the Investments Unlimited book, which he co-authored. He will be co-presenting with Michael Edenzon, one of his fellow co-authors. Here's Rez and Michael.

John Rzeszotarski and Michael Edenzon

01Opening and the end of Investments Unlimited

John Rzeszotarski: Awesome. Thank you, guys. Okay, so we're going to go fast and furious, because otherwise they're going to turn off our slides like they did Jason yesterday.

Real quick, I'm John Rez, senior vice president of infrastructure at PNC, and this is Michael Edenzon with Fianu Labs. Mike and I have worked together the last three or four years at PNC, and we've done some amazing transformations and a lot of awesome work out there. One thing incredibly close to our hearts is automated governance, which culminated in us being co-authors with some awesome other co-authors on Investments Unlimited. The story we're going to tell today is: what happens at the end of Investments Unlimited? The transformation occurs, something big happens, but now you've got to adopt, onboard, and get the rest of the organization to buy in. We're going to use some of our own personal experiences and a lot of the research we did in writing the book.

Quick disclaimer: we heard the law of mobility yesterday. This is an enterprise conference, so we're going to create a quick addendum to that with words and things. If you're in a meeting and asking yourselves, "Is this meeting about words?" and if it is words, you might want to consider utilizing that law. We also want to end every meeting with, "Let's go do the things."

02Why now, and why regulated industries are different

John Rzeszotarski: Why now? Since 2014 or 2015, enterprises have been carrying a lot of DevOps capabilities across the industry. But when we go back and think about it, these original things came out of big tech companies focused on building the big web, and they got a ton of awesome benefits: 10 deploys a day at Flickr, et cetera. One big problem was regulated industries. They got to take advantage of a lot of it, but they didn't get the value that the big tech companies were getting, because they have many lines of defense and many problems to solve, and there has never really been a great playbook for that.

We also come across a lot of people who say, "We do this already. We do secure coding standards. We do pipeline integration. We have automatic change record creation. We have a million tools that go out and do this." Our argument is different. This is a big transformation. It reaches across the aisle, talking to a lot of different teams and groups. You're going to have to build provenance for ten different runtimes. You're going to work on distributed and non-distributed systems. You're going to deal with a ton of legacy code bases. This is not a cookie-cutter solution, so whatever you do, don't go buy DevOps with a side of automated governance.

Like I said, Mike and I have been working together for three or four years, and we had this main goal to do continuous delivery at a large financial services institution. We wanted to do 10 deploys a day.

03The superhighway vision and the reference architecture

Michael Edenzon: When we stepped back and looked at it, we had a lot of procedures, policies, best practices, and things we had to follow. A lot of those turned into requirements for something we needed to engineer. Basically, we're sitting in a conference room and a voice calls out into the speakerphone: "If you build it, they will come, because engineers and developers absolutely want this." For those of you who have seen Field of Dreams, where Kevin Costner hears the voice in the cornfield, our voice came to us in a conference room. It said, "If you build it, they will come." So we asked ourselves: build what? What are we building?

We had this vision of a superhighway. The superhighway would allow a developer to go from new code to production without any human intervention, without any manual work, fully automated. I've talked to some of you at cutting-edge tech companies and you say, "We've been doing this for years." That's true. But those of you in regulated institutions, like banks, defense contractors, and healthcare companies, know that there are a lot of manual processes built up over the years, all related to regulatory requirements. That makes it really difficult to do this in an automated fashion.

So when we had this idea and this vision, we asked ourselves, "How are we going to do it?" Where do all great stories originate? A paper from the DevOps Enterprise Forum. In 2019, John Willis pulled some of us together. We got to write a ton of things around controls. We talked about what evidence that cannot change, ephemeral evidence, really meant and how you needed to quantify it. We talked about a lot of best practices associated with what we needed to build. That gave us our blueprint.

When we were talking about the automated governance reference architecture and what it would mean in its final state after we built it, John and I looked at each other and said, "Wow, this is going to have a huge impact on the developer experience at our company." We had to be careful. We didn't want this to fall into the wrong hands, because this gives a lot of power. It could improve developer experience, but used in the wrong way it could make developer experience a lot worse.

04Three beliefs

Michael Edenzon: In the beginning we set forth our beliefs, derived from this community. The first belief was developer autonomy. Developers should have autonomy and freedom to choose things like their IDE, build pipeline, builder image, and even the ability and freedom to prioritize their own worksets with the business, free of administrator oversight from someone who may not be on the team.

Second, we believed in carrots instead of sticks. I don't need to explain this too much: positive incentives instead of punitive justice.

Lastly, we believed in no downward pressure. We didn't want to do anything that put an undue burden on the developer experience.

05A simple shared language

Michael Edenzon: Our strategy was to create a simple shared language. We were going to take the principles of the automated governance reference architecture and boil it down to a very simple shared language. A given control in the development process can have four states: pass, warn, fail, or not found. Passing means you're good to go. Fail, obviously, you're not. Not found means we don't have evidence for that control, so we're going to deem it a failure.

We built this product and rolled it out. We were excited. We said, "We built it. We built this superhighway." The problem was that we built it, and they didn't come.

06Choice one: autonomy or standardization

Michael Edenzon: Why was that? Onboarding failed. When we dug deep, we realized onboarding failed because you can't onboard what you can't see. This challenged our first belief, and we had to make a choice: do we promote autonomy or standardization? I don't want to make it seem like those are binary. I'm not saying you can't have standardization while preserving autonomy. But in our case, we had to put in some serious guardrails. At that point developers could have any permutation of build patterns, and it made it impossible for us to see what was going on. We couldn't onboard what we couldn't see.

We standardized. We actually used our automated governance tool to standardize, and we could talk about that in a different conversation. But we standardized, and onboarding took off really quickly. We looked at each other and said, "All right, we're on the right path now."

Then it quickly fell flat, because compliance didn't improve. We had all this great observability. We were producing immutable attestations and giving great continuous feedback to developers, but they weren't using it to get compliant. If they weren't getting compliant, they couldn't do automated releases, and automated releases flatlined. That was our entire goal.

We dug deeper and realized what we sold was really different than what the developers saw. We sold a superhighway driven by compliance. What the developers saw was the reality of software development: their apps were not compliant. We had a long way to go as an institution to get these applications to the point where they could be released.

There was a disconnect. We had to decide whether to feed them a fish or teach them to fish. As the proprietors of the data we were giving to developers, they thought we were responsible for remediating compliance. We had to teach them: no, you own your own compliance. Only you, the development teams, can be responsible for your applications.

In summary, our first carrot didn't work. We had to try something different. They weren't using the carrot of automated releases.

07Choice two: what should the data be used for?

Michael Edenzon: What about scorecards? Horrible idea. The scorecard would be simple: take controls, bundle them together, say A, B, C, D, F, aggregate, and give a score. When we talk about scorecards, we can't avoid showing Rez's favorite movie.

The scorecard gave us another question: what should the data be used for? Do we package this data into scores and throw it over the fence and say, "Don't talk to us until you're compliant"? Or do we give all the troves of data over, at which point we risk someone misusing the data for a game of gotcha?

All the scorecard really is is an abstraction. It's packaged information. It's in the aggregate, and it abstracts the details so you can make a directionally accurate decision about how compliant something is. The problem with directionally accurate is that it's not sufficient for automated governance. We need granularity in our data.

When we provided the details, it presented a different problem: with details, great power comes great responsibility. There was a huge risk of the data being taken out of context and misused.

John Rzeszotarski: You have to remember, when working on automated governance you're going to be reaching across a lot of aisles. A lot of the groups you're working with are not technologists, developers, or engineers. When you say things in procedures and policies, it can have a very ill and negative effect. In research where we talked to a couple of other organizations, we found things as simple as a unit turning off unit tests during a build process being treated as an infraction of policy; in a couple instances, that was termination-applicable. You really have to think carefully about applying things like this. Jason said something yesterday that hit home with me: proximity. You need to stay very close to the risk units that are going to enforce the procedures and policies you write, because this could be incredibly detrimental to your culture and your development community.

Michael Edenzon: In short, we tried the scorecard and it didn't matter. It didn't make a difference. There was a nominal increase in compliance, but it got us nowhere near where we needed to be.

08Choice three: downward pressure

Michael Edenzon: We had to try something different again, and this challenged our third belief: no downward pressure. We had the mother of all downward pressure. We had this data, and we could start blocking deployments and interject ourselves in the release process. I'm sure some of you relate to this: you go to release, and someone uses that opportunity to say, "I found all these new vulnerabilities; you can't release." Downward pressure is the ability someone has to interject themselves directly into the workflow of a developer because that's the most convenient time to get the developer to do what you want them to do. When that adds up, it becomes a real bottleneck.

So we asked: are we sure we want to do this? Just because we can, does that mean we should? Our belief came from Sidney Dekker in Just Culture, when he said anything that puts downward pressure in your organization on honesty, openness, and learning is bad for business. We believed that.

But we decided to do it anyway. We put in automated enforcement. We alert you throughout the build process and throughout the pipeline process, warn you when it comes time to deploy to a lower environment, and if you still go forward with your production deployment, we block you.

09Game theory and why automated enforcement became the game changer

John Rzeszotarski: This really hit our beliefs because this is not something we wanted to implement. We wanted to build a high-trust organization. We had to convince ourselves why this was the right way to go. Andrew Clay Shafer often uses a picture of the Gatling gun and talks about finding the thing that changes the game. It's applicable to whatever situation you're working on.

We tried to run this theory about game changing, because we thought automated governance was our game changer. When we look at players in this game, we have product owners, developers, and engineers on one side, and governance, risk, and compliance on the other. If I'm a product engineer, or if you read the book, Bill Lucas, the product owner, says we want to do features. Features are the most important. They're the Wild Wild West; that's what we want to get done. For governance, risk, and compliance, the optimal outcome is a very lengthy review process. We are going to check this thing to death before we let it out there. Not surprisingly, the non-optimal outcome is the exact opposite for these two players, which is why we're stuck in this game.

What does the industry do? It falls back to a very subjective change process. I'm essentially going to fill out a mortgage application as my change record with all that documentation. Then I'm going to try to get through approvals depending on who likes me that week. Hopefully I can get my change approved and put into some type of change window.

The ironic part Mike and I struggled with was: why isn't automated governance the optimal outcome for both parties? It ended up being the non-optimal outcome for both. For developers and engineers, it was yet another thing they had to implement: add that to the backlog. On the GRC side, small-batch release is not an easy thing to swallow for a governance, risk, and security organization producing change that often and frequently.

The prisoner's dilemma teaches us that the non-optimal outcomes for the two parties playing the game can actually be what's best for the entire organization. That fit. Now we needed a game changer for our game changer. How do we change this game to get buy-in for the non-optimal outcomes for both parties? That ended up being automated enforcement, because it removed the other outcomes that were even possible for these two players and forced everyone down their non-optimal solution.

10Rollout tactics and results

Michael Edenzon: Real quick, we'll hit some tactics. We started with simple controls and soft gates, meaning failing teams get a warning: coming up soon, we're going to start blocking you for this. Then really quickly we move to hard gates. We recommend that because you want to send that shockwave through your organization so people know this is now how things are going to be.

Then quickly move to more difficult gates, all the way up into your most difficult controls. I think you'll see developers make the adjustment very quickly.

I will warn you: we did not make any friends in this process.

John Rzeszotarski: Mike has to walk me to my car.

Michael Edenzon: Exactly. We're not trying to scare you, but we want you to know this will not be a favorite among developers at first. But we think it's worth it, because what we saw after implementation and after developers started to change their behaviors was extraordinary.

We saw a dramatic uptick in engagement. That was encouraging because engagement meant developers were starting to care and take ownership of their compliance. They were trying to figure out how to remediate failed controls. They started helping each other, which we didn't expect. It created a sense of community, and that engagement was encouraging. That's when we knew we had it.

Also, a lot of stakeholders were worried at the beginning that we would cause production incidents by blocking deployments, because teams would miss their change window and it would have a cascading effect. In our research and experience, we found zero instances of that happening. Teams took it seriously, and they know what they need to do to make sure they don't cause these incidents.

So if you're going to try this, go into it with a lot of confidence. Know that you're not going to be a fan favorite, but that it will work.

Most importantly, we reached our goal: compliance shot up and automated releases followed soon thereafter. We were able to realize our vision by pretty much abandoning all the beliefs we held in the beginning.

11Closing

John Rzeszotarski: Regulated industries are inherently consequence-based models. Those consequence-based models go all the way down into the application teams, and they permeate and drive through the organization.

We understand that a lot of people aren't going to agree with some of this. In fact, we've had people tell us very much to our face that this is the wrong model and we're doing the wrong thing. It's important to understand the contextual nature of what you're trying to do and what industry you're working in. We would love to hear other stories associated with adoption and onboarding something like this in your industry and how that is accomplished.

Hopefully you learned a few things, and we can all go do the things.

Thank you, guys.