Mark Schwartz

My name is Mark Schwartz. I am an Enterprise Strategist for AWS, and what that means is I'm part of a small team of people who were senior leaders in IT organizations before they joined AWS, generally CIOs, CTOs, who pulled off some sort of a digital transformation, moved their organizations to the cloud. And what we do is we work with AWS customers, large enterprises generally, to help them get over the impediments to transformation that are typically non-technical things, almost always cultural change, organizational structure, getting the right skills for their people, figuring out the right investment models, and overcoming bureaucracy sometimes.

One of the topics that comes up a lot when I'm working with customers, very often they'll ask me when they hear about our transformation, I tell them at USCIS, where I was the CIO before I joined AWS, U.S. Citizenship and Immigration Services, part of the Department of Homeland Security. And I tell them we moved from a release cycle time on the average of about 18 months to deploying code, in some cases, three times a day, four times a day. We moved into DevOps. We moved everything to the cloud, or almost everything. We started refactoring everything into microservices. We made a lot of changes to how we did procurement and investment management. And the question seems to often be, how long did that all take you?

And I think when I hear that question, if the person who's asking it is trying to justify it taking a long time in their organization, right? So I love to tell them that actually it took us two days.

Now, I know a lot of organizations can't do it in two days because they don't have the advantage of bureaucracy like we did. But if they did, then they would be able to do it as well. And in order to explain what I mean by that, I'm going to have to give you a little bit of context first. And by the way, there are a bunch of people who worked with me in the audience here, so if I call them out by name or something, you'll understand.

So first, what was the impediment to transformation for us? And it was a document. It was a wonderful document called MD-102, Management Directive 102, a DHS policy on how essentially all IT investments were going to have to be overseen and run. So it was essentially the SDLC that we had to follow.

And last night, I double-checked the last version of it. This document, MD-102, called for us, for each system we created, to write 87 documents and to go through 11 gate reviews and define 21 different oversight roles for people who could say no to things. So it was an awesome thing. It called for us to do a system definition review that checked to make sure that all of the requirements had been written down and fleshed out. And there was one great, what was it, a test and integration review or something, where you had to get approval to integrate the code in the system and to test it.

So clearly this was set up to be a waterfall process. And as we started our agile transformation, we had the idea that maybe it wouldn't fit too well with this MD-102, which was officially the policy.

So we gave it a little bit of thought. We had a lot of arguments around this: how can we be agile with this wonderful piece of bureaucracy? And finally, we realized something which I think is true of a lot of pieces of bureaucracy. There's usually some sort of exception process, right? The people who wrote it aren't dumb. They know things are going to happen.

So there was something, in this case, called a project tailoring plan, and we could use the project tailoring plan for a particular project to explain why we were going to need to follow a different process for the special needs of that project. And of course, the special needs for our project were that it was a project, and projects should be done in an agile way, right? So that was essentially the argument, but we created this tailoring of the official process that essentially said we're going to do the opposite of everything that the process says. And it's justified because we shouldn't use the process, right? Essentially.

So that was kind of a little bureaucratic move that you might want to learn.

Now, there's a little snag with this, which is that somebody had to actually sign off on our plan, and that was a little bit harder to master. But the way that we did it, another good learning, is we took the project that was failing the most and that nobody could figure out how to fix, and we suggested that we use the tailoring for that. And since nobody else had any better ideas, I think, we were able to get an official approval to try it out.

So we chose, in fact, the project, if you listened to the USCIS presentation the other day, our enterprise transformation project, and that was the project that had already spent about a billion dollars and gotten nothing for it. So the idea that we might try a different approach was something that people bought into.

Okay. So with that approval, we started doing a Scrum process and doing our two-week iterations. Actually, I think we started with three-week iterations and standing up every day for 15 minutes and retrospecting and all of that. It was all going fine. An interesting conversation took place at that point between me and the people who were in charge of this MD-102 thing, where I said, "The agile way of running IT projects is actually much better than the old waterfall way, so why don't we change MD-102 to say, do things in an agile way?" And they said, "Well, there's no need, because if you want to do things in an agile way, you can do it under MD-102. You just need to do a tailoring plan." And I said, "Yes, but maybe it's better if the official policy doesn't say you have to do 87 documents and 11 gate reviews, and endorse that kind of behavior. So why don't we just change it?" And they said, "Ha ha. We have to be responsible around here. No. But we don't see what you're complaining about because we can do this in an agile way."

So I want you to remember this conversation because it's going to come back in a minute.

So here we are doing our little agile thing and suddenly another snag arises. And what that was, because this project was so high profile, because we had wasted so much money on it, we were of course going to be audited by the Inspector General and GAO, the Government Accountability Office. And, of course, we thought since we are now actually producing stuff every couple of weeks, that they're going to say glowing things about us, right? I'm looking at my colleagues in the audience. So it turned out they didn't say glowing things.

Now, what they did say though, was a bit of a surprise. Essentially what they said, we sat down for a debrief with them, and the auditor said, "You're not agile enough." And that, I thought, required a little bit of explanation since I had been doing agile IT initiatives since long before I joined the government. And, as far as I knew, GAO didn't know anything about IT development, let alone agile approaches.

So I wanted to hear their reasoning, and it went sort of like this. They said, "You said in your project plan that you're going to do Scrum, and we checked what the practices of Scrum are." And, yep, there was a list of practices and they said, "You're not doing some of those." And I said, "Well, number one, don't confuse Scrum with agile. And number two, you're supposed to retrospect and do continuous improvement and change your process in ways that make it better." And we made a few changes, especially around product ownership, which wasn't working well for us the way it was visualized in the books on Scrum. And they said, "Well, you're not doing what it says." And I said, "Well, agile means you change." And they said, "Well, you said you're doing Scrum, and we read in the book where Jeff Sutherland says you can't change anything in the Scrum process or you're not doing Scrum anymore." Which he did. And so we had to admit that they had us on that one, right?

So it was a failure that we learned from. If you think about it for a minute though, I just want to point this out: we had been defeated by bureaucracy, but it was not government bureaucracy that defeated us. It was the bureaucracy of Scrum, actually, in the end.

All right. But this gave us the perfect idea because we realized that GAO, when they audited us, they were going to compare what we said we were going to do to what we were actually doing. And so all we really had to do was to say we were going to do the things that we wanted to do, which is obvious if you've got bureaucracy behind you. So we decided we would write a new policy that said you have to be agile.

And so we started to go down that route and we found out that actually, writing policy is difficult in the government. It's actually in law how you have to go about writing policy, and you have to put it out for public comment and all of this. It takes a really long time to do. So our next idea was, why don't we create a management directive, just like MD-102? And it turned out that we couldn't do that either, because a management directive, somebody in our group probably knows better than me, but I think it has to come from somebody in a management directorate in the parent DHS department. So we couldn't do a management directive, but fortunately we had some good bureaucrats on our team, and they said, "You know what? You can do a management instruction." Win, right?

So we knew that for anybody to take our management instructions seriously, it was going to have to have a good title, sort of like MD-102, a compelling title like that. And, since it was the first one we'd ever written, we were going to have to come up with it. So we invented MI for management instruction, dash CIS, the name of our agency, dash OIT, the Office of Information Technology, dash 001, and figured that would make people take it seriously. So we created MI-CIS-OIT-001, and it said everybody has to be agile from now on. And we published it, and there was an effective date. So as of that date, everybody was agile. So that part of our transformation took one day.

All right, there's the first day. And the policy, actually, it was pretty clever. We didn't want to have that same fight with GAO again, so we listed the eight practices that we were going to call agile. It was things like frequent delivery of value and retrospecting and things like that. And then five other optional practices that were sort of advanced agile. And so boom, we were agile.

A little bit later on, we learned about DevOps, and we decided we wanted some of that, and it wasn't going to fit under that. So we had to write MI-CIS-OIT-003, which essentially said everybody's DevOps-y now. And so we had transformed to DevOps, and that was the second day of our transformation, when we issued that policy.

Now, there are a couple of problems. Maybe. You probably didn't realize that there were any problems. But the first problem we figured is it conflicted with MD-102, and MD-102 was written by somebody much higher in the organization. So maybe that would be a problem. And so my team's worried, "Boss, you keep telling us do things this way, but somebody is going to come in and say you're not doing what MD-102 says." And so I said, "Right, that's an impediment. My job is to remove the impediments. You follow my policy, and I'll deal with it."

I was pretty sure I could deal with it because if you remember back to that earlier conversation, the folks who were responsible for MD-102 had told me there's no problem doing agile under MD-102, even though it says exactly the opposite. All you need to do is tailor it. So I figured, what are they going to do, tell me I can't be agile? They just told me I could. All right. So that was an easy one.

The second little challenge was as soon as I would meet with a bunch of people in the organization and talk about doing DevOps and being agile, they would then go out into the hall, yes, you did, and say, "Oh, he's crazy. What are we supposed to do? What is this thing?" Right? But I figured eventually, if they were going to try to do it, they would figure it out.

In fact, what you need to know is that my position is what's called Senior Executive Service, an SES. Technically, an SES is supposed to be the same rank as a two-star general, because they wanted to make an equivalence between military ranks and government civil service. So you have to imagine what happens when essentially the chaos monkey becomes a two-star general, right? This is the situation here. So when I said everybody do DevOps, it was like General Chaos Monkey saying, "Everybody do DevOps," and of course, the whole team tried their best to do what I said.

We brought in some wonderful coaches, some of whom are here. And over a little bit of time, people kind of figured out how to make this work. And they started to do these frequent deployments. They learned how to deploy things with no downtime and how to roll back if anything went wrong, and they built a CI/CD pipeline and started doing frequent deployments and all of that.

And then it was time for the GAO audit again. And we knew that we would pass with flying colors, even though in the history of the universe, GAO has never said anything nice about anything that they've audited. But we figured this time we got it. We're doing exactly what our policy said. And it turned out that they didn't agree.

They said the problem here is that even though people are doing what your policy says, you don't have a mechanism to make sure they are. Auditors love that. You got to be able to prove that you're actually compliant. So MI-CIS-OIT-004 is the policy that says we're going to have our independent verification and validation team look at what all the DevOps teams are doing and make sure they're doing DevOps. And that was pretty brilliant, right? They expected us to have a policy that said if they find that somebody's not doing DevOps, they're going to take them out and shoot them or something. And instead, our policy said if they find somebody who's not doing DevOps, then they'll work with them to try to help them be better at DevOps, which was a little bit of a surprise to them.

But anyway, we now had this policy. So we had a policy that said everybody's got to be agile. We had a policy saying everybody's got to do DevOps. And then we had a policy saying we're going to check to make sure that everybody's doing DevOps. All of them were beautiful pieces of bureaucracy that had compelling titles to them. And we could do all of this because we had bureaucracy behind us. It was gorgeous. So obviously then it was time for me to leave the agency because my work was done, right? So we were there.

So a couple of things. I want to sort of step back and take a few learnings out of this. And the first point that I want to make is everybody's got bureaucracy. It's not just the government. Every company that I talk to, when I mention bureaucracy, they say, "Oh, come on, we need help with that." It's actually going to be the topic of my next book. And every time I tell people what I'm working on, they get all excited. "We could use that. We need that one." So I'm going to be writing about bureaucracy, but everybody's got it. And, in fact, as I said, it was the Scrum bureaucracy that killed us.

IT organizations are great at creating bureaucracy. We make standards, and then we make a process for making sure people are following the standards, and an exception process for if they don't want to follow the standards. Request it in triplicate and send it here and send it there.

And then we've got this DevOps thing, which is, in many ways, the extreme of bureaucracy. Think about this. One great piece of bureaucracy we used to have is after you finish coding some new capability and you want to deploy it, you had to go to the CAB, or the change control board or something, and you had to submit all this stuff, and they thought about it. It was always a group of people that really had no way of knowing whether you should deploy this or not. But what they would do is they would look at the paperwork and they would say, "Yeah, let's check the test reports. Oh, it looks like they ran all the tests and all the tests passed, so I guess that means they can deploy it."

Beautiful bureaucracy, right? That's really a bureaucratic structure to it. And that was how we used to decide on whether something could deploy. What do we do in DevOps? Exactly the same thing, except it's automated, right? We set up the pipeline so that you can't actually deploy unless all the tests have passed and it's gone green. What's the difference? It's really the same concept. Concept being, let's create rules to encapsulate good practices, and let's enforce those rules in every case. No exceptions allowed unless you fill out the form in triplicate.

So, in a way, DevOps is all about automating bureaucracy. I'm not saying anything negative. When I think about what it is about bureaucracy that drives us crazy, I think it's not the nature of bureaucracy itself, it's two characteristics of bureaucracy. The first one is that often we see bureaucracy that is not lean. It's wasteful. So it says if you want to deploy a piece of software, you have to write 87 documents and go through 11 gate reviews. What's the problem with that? The problem isn't that you have to have some structure. The problem is that 87 documents is ridiculous. It's just extra work that you shouldn't have to do. When you have to fill out the forms in triplicate and take them from window one to window two and window three, it's just extra stuff, and it's really frustrating because it gets in your way and you keep feeling like, "Just let me deploy." So the fact that bureaucracy often is not lean is the first thing, I think, that drives us crazy.

The second thing is that bureaucracy tends not to learn. So you have a set of rules, and that set of rules doesn't change very fast. In fact, it changes slowly. So MD-102, perfect example. Here are 1,000 books that say it's better to do things in this agile way, you're going to get better results. By the way, the waterfall way has never actually worked, and if you look at your portfolio of things that you're overseeing, you'll see that every single project is failing. So maybe we should change MD-102. No. Bureaucracy tends not to change.

But those two characteristics, the waste and the resistance to change, those are not actually essential to the idea of bureaucracy. In bureaucracy, you apply rules uniformly to everybody without exceptions. That doesn't mean that the set of rules that you're applying can't change.

And there's no reason why you can't take a bureaucratic process like MD-102 or any other one and look at it as a value stream and look for where the waste is in it and reduce that waste, either shortening steps or taking waste out of them or taking steps out entirely. Any bureaucratic process is subject to exactly that same approach as we do with any lean approach to a process. So there's no reason why a bureaucracy has to be wasteful or unchanging.

You can have lean bureaucracy, and you can have a learning bureaucracy. And in fact, when we set up this MI-CIS-OIT-003, we deliberately set it up to be learning bureaucracy in the sense that the policy itself did not define what anybody should do. It defined the objectives that we were looking for, frequent delivery of value, et cetera. And then it said, "The practices we want you to follow are in Appendix A." And Appendix A has what we think today are the best processes or best practices, but we expect that Appendix A is going to change over time as we learn new things. And so we deliberately separated these two concepts of the outcomes we were looking for and the best practices we were going to use to get there so that we could keep changing those best practices as we learn new things. Why can't you do that in a bureaucracy, right?

Okay, it's a little bit arrogant, but I think this was like the best piece of bureaucracy ever written. This was art, right? So, I contend that you can actually take bureaucracy when you encounter it, and you can take advantage of the little loopholes and exceptions and so on. But what you can also do is take that same lean approach to bureaucracy that we take to all of IT these days, look at it as a value stream, yank waste out of it to reduce lead time, and just keep doing that. And at the same time, establish a feedback mechanism so that you can learn as you go, and you can alter the bureaucratic rules based on your learnings.

And in a way, this is what we're doing in DevOps, where we take what could have been a slow, wasteful bureaucracy and automate elements of it and eliminate elements of it and set up ways of establishing compliance through automation and through artifacts that are generated automatically to demonstrate compliance and things like that.

So bureaucracy, to me, is not an immovable impediment. It's something that sounds terrible. The words are terrible. We use the words to describe anything that's frustrating to us. But the words bureaucracy and the fear of having to deal with politics and controls that are coming from outside yourself, that's just a fear. It's not an actual impediment. It can be removed as an impediment. It can be worked through. It can be altered and improved and even used as a way to establish controls where you do want to establish controls in cases where you want to have standardization, let's say, or centralized control, shared services. There are all these applications in the world of IT where we want to put some structure and some formal process around things. But when we do that, we have to be very careful to make sure that we're not creating bureaucracy that's going to resist change and that's going to add a lot of cost and lead time as people try to follow it.

So my concluding message is that bureaucracy, painful as it might be, and as much as you might feel like it's getting in your way, is actually something that you can tackle and you can work with, and you can use bureaucracy to get better impact, just as we use bureaucracy to actually drive a transformation, strangely. So good luck to everybody with your bureaucracies. Thanks.