Log in to watch

Log in or create a free account to watch this video.

Log in
Las Vegas 2022
Share

Inferno (USAF Kessel Run) — Navigating 12 Parsecs of Bureaucratic Hell

This will be the first of three parts of Bryon’s Enterprise Comedy, a journey of the soul toward innovation! Inferno (USAF Kessel Run) describes the recognition and rejection of bureaucracy along the 12 parsecs of bureaucratic hell. The allegory draws on Bryon’s experience co-founding and scaling the Department of Defense’s premier DevOps organization, Kessel Run, and now studying & helping other enterprise software organizations make the journey. And of course Divine Comedy… and… Eminem’s Rap God?

Chapters

Full transcript

The complete talk, organized by section.

Bryon Kroger

Good afternoon, everyone. I'm Bryon Kroger. Welcome to part one of Enterprise Comedy. It's my take on Inferno Dante's Inferno and my journey through 12 parsecs of bureaucratic hell, so I wanted to give a different kind of talk today than I normally give. I want to be a little bit more forward about the fact that bureaucracy is a hell and with no redeeming qualities, right people say, oh bureaucracy serves a purpose. I would argue that it doesn't at least not in an innovative world.

So what are the sins that land us there and keep us there suffering. I'll use my own Kessel Run journey and observations with other organizations have now had the privilege of working with and hopefully this talk is equal parts fun and educational and I want you to walk away recognizing these sins for what they are. Like, they really are the sins of bureaucracy and also a few tips. This would be a departure from Divine Comedy give you some tips on how to escape them and make your way through this bureaucratic hell, so

I joined the Air Force in 2009 with a strong desire to make things better, but seven years in I looked around and realized that somewhere along the way. Falling off my path many others around me might not have realized it. I don't know that I had even really but it started innocent enough at first right? I started to believe the bureaucracy had a purpose. I began to see some of its worst sins as being necessary or not that bad. I'd really fallen off the path at the same time though for those first seven years. I used really terrible software that the bureaucracy had provided the outcomes were terrible. I was a targeting officer. I watched some of the worst things in the world happen stepping into the office was like stepping back in time. I used to make a joke here about Microsoft Office, but it kind of belies the gravity of this situation bad things happen when humans try to do critical missions at machine speed not only that but we risk disruption not disruption in the Fortune 500 but on the battlefield and not to our profits but to our entire way of life even seeing that though I couldn't get back on the path like Dante my pride and my Envy kind of blocked the way to my own growth along with my ignorance. I really needed a guide.

I didn't get a Virgil though. Like Dante I got to wake up call when we struck a Doctors Without Borders outpost in Kunduz, Afghanistan that pain of watching that happen watching those doctors, international doctors sent there to help people die that pain became my guide it humbled me educated me and it got me back on my path but to get to Purgatory though, like Dante I had to enter the belly of the Beast. This is the defense acquisition system folks. I'm sorry for putting it in front of you. It's the worst slide I have I promise but I applied for a journey through hell and assignment as an acquisitions officer in the Air Force. In fact, I'm pretty sure that my orders actually said abandon all hope he who enters.

And

I actually want to read this summary of the intro of Inferno that I found. They enter the outlying region of Hell called the anti-inferno pictured here where Souls who in life could not commit to either good or evil must now run in a futile chase after a blank Banner day after day while hornets bite them and worms lap up their blood now that is like pretty dark but I think it actually describes the majority of federal software procurement quite perfectly.

Perhaps the state of many software organizations that y'all are a part of not necessarily people committed to doing bad things either right? That was the whole story of the anti-inferno. It's just this sort of weird status quo that punishes them horribly. It's no way to live a work life work life can be better than this and this is where I learned my first lesson about empathy. I had started empathizing with the bureaucracy, but empathy should never become excusing understanding how a thing came to be doesn't mean we should excuse it.

But I have been doing just that it's really easy to feel bad at times for the people that are stuck in the bureaucracy, but there's actually a problem in not choosing and so empathy beautiful thing key to Enterprise transformation but moving into excusatory is not okay. To air as human but to Nair is anti-human. You have to make a choice you have to do something. We owe it to ourselves and our fellows to decide and to commit what we believe is right.

In the Air Force we have this Rebel hero named John Boyd who taught us that in life. There's a roll call to be or to do you have to stand up and decide if you want to be somebody or do something that's the first step in the path to Innovation to decide to do something for yourself for others for your world for your country. Whatever it is that you do but unlike Dante's Inferno these people stuck in the anti-inferno. They're not dead. They're very much still living and you can lead them out of their plight too. You need to to get out of the anti-inferno though. You need a burning platform and I had one right? This was my roll call.

Our approach to software including the processes that were actually meant to reduce risk. We're failing our users putting our most critical missions at risk and demoralizing the employees who are building them. What's I going to keep climbing the military ladder or was I going to do something about it? The painful realization led me to my new assignment the belly of the Beast to start the journey on the day that day that I showed up. I called Defense Innovation Unit Experimental and linked up with a man named Lieutenant Colonel Oddy and a series of events. That would then become known as the Kessel Run. Some of you may have heard that story a couple times at DevOps Enterprise Summit. It's been told a lot.

So I'm just going to keep it very short here several of us got together in a room. We came up with a strategy to smuggle DevOps into the Empire. We needed a name for it.

So we generated stickies and like all good agile practitioners. We thought voted and Kessel Run one. This is a real picture of that voting. We cracked the code over the next two years on continuous delivery in the DoD. We became the premier DevOps organization in the department shipping warfighting apps to the field daily at secret and top secret levels. Of the most stringent GRC that you've ever seen and it's created an entire movement across all of the military services, but in the beginning we didn't have that Fame and we needed people to follow us out of this anti-inferno the burning platform wasn't enough.

I learned from Enrique that it also had to be coupled with a vision kind of a beautiful work life, which you see there on the right where we can continuously deliver valuable software that users love and then drive relentless progress towards a better future. That's how you get people on the journey Adam Furtado who's actually here right over there. He jokes that when he called Enrique to ask what this was all about when we asked him to move his new family just had a baby all the way across the country to San Francisco with like five days notice. He said hey, what are we doing in Enrique said we're going to change the air force that was actually the entirety of the conversation. I think Adam joined us on the journey.

So I'm pretty sure it works or Adams just a crazy leap of faith kind of guy, but at the end of the day, there are a couple lessons here already right? Number one when you're starting these Journeys you have to have a burning platform.

But with that don't wait for an invite. Nobody invited me to start the Kessel Run. Nobody invited Enrique. Nobody invited any of these people. Nobody gave us permission for sure.

And then use the burning platform don't wait for your pitch to get funded. Like I'm gonna go to the Enterprise Shark Tank and then once my idea gets funded then I can do it. No figure out how to get money and last but certainly not least. Please do not go alone. We still haven't even entered hell yet.

First off since the theme of Kessel Run instead of the nine circles of hell. I have 12 parsecs. I borrowed nine from Dante, but I added three of our own what are the sins that keep us stuck in bureaucracy. I think I need to go at light speed now. All right limbo.

In my journey, I found that a lot of people were just waiting. I've got four kids.

So I read a lot of Dr. Seuss and it always makes me think of the waiting Place very familiar with the waiting place where people are just waiting waiting for their hair to grow all sorts of weird things the burning platform in the vision. They don't get these people going you need something else.

And the first folks to come along are always the innovators, right?

So there's this innovation distribution curve that we always forget about. In fact when we talk about the frozen middle, it's not actually middle management. It's the middle of the Bell distribution. It's the early majority in the late majority, but the early and late majority couldn't be talked into coming. They had to see it even some of the early adopters are that way and so one of the most effective ways that I can tell you to solve this problem is to employ what I call the NUMMI maneuver this case study gets talked about a lot in the DevOps community. I first heard about it from Jez Humble Toyota wanted to enter the American auto manufacturing market informed to joint venture with GM who gave them their worst performing plant in the United States so bad that people were drinking and gambling on the job all kinds of other wild things. The plant had recently been shut down GM offered to help them take on the union so that they could get new employees and Toyota said no, thanks. They took them to Japan put them through the Toyota production system. They didn't give them a bunch of re-education and tell them how to think they just said hey come and work with us in a different way and when you see the Of working with us in a different way that will change your values and attitudes that will change your beliefs that will change the culture. They came back and did the same thing at NUMMI and in a few months a few months. It was the highest performing auto manufacturing plant in the United States same people same people that were drinking and gambling on the job doing things a different way in a different architecture. You can do this too. Just do it one slice at a Time start with your early adopters, then your early majority your late majority never crossed two groups at once because they have very different expectations.

Finally at Kessel Run, and now with my company we use balance teams engineering product design and we use pairing in every single position to employ this NUMMI maneuver and it's very effective.

So if you're just starting your journey, I would say start with one to three teams full of people from the innovator slice. Once you have real working software deployed to solve a real problem for a real user in a real Ops environment, then you can start bringing in the next group and pair with them and then split the team and pair again and keep doing that until you cycle through the early majority in the late majority.

And as you iterate build more complex system, it's an outcomes because you'll need that as well to pull in the late majority.

When we got to this point at Kessel Run, there was a lot of pessimism about our approach focused mostly on people to borrow another great take from Jez Humble. They say it won't work here because our people are too stupid. This is probably the most repeated things in the halls of government. We often saw that the organization would then try to hire external Talent from the valley. We need to hire some Silicon Valley folks to come into the Pentagon to come into these programs and fix them or they would do it via contracts with consultants and Outsource development. You can't hire your way out of a problem that you have bureaucracy your way into you have to grow out of it and we all need to because the most important ingredient in Enterprise transformation as opposed to smaller things on the edge is context and context can't be trained and when it walks out the door, it's gone forever.

So this is kind of my one two, three recommended reading to get you out of this trap first. We recognize that Talent, at least in the traditional sense, is overrated great book on the concept of deliberate practice and putting And you know Malcolm Gladwell's 10,000 hour rule like put in the work and and we needed to get our reps in and to get our reps in we needed learning velocity. We needed a path to prod we needed to be able to talk to users and we could learn and we could grow but to go that route. We needed to Foster an environment where employees with a growth mindset can learn and Thrive finally it is hell.

So we needed grit passion and perseverance passion as I like to say is developed not found you can help others develop it while creating a culture where when you fall down seven times you rise eight.

The flip side of this that I would often see is that people didn't think they needed any help at all. Deliberate practice requires not only a growth mindset and grit it also requires a coach how many sports or activities did you do growing up where there was no coach at practice get coaches, especially for leaders. We got coaches for both leaders and practitioners, but I really love the example of the trillion dollar coach Bill Campbell coach to Collective trillion dollars and value at one time including the likes of Steve Jobs and Eric Schmidt.

So if those Titans need coaches, I think we all do

At Kessel Run. We also use the consultancy called Pivotal Labs. It's gone now, so I'm not Shilling for them, but they used the same pairing model that I now use at Rise8 to coach practitioners and it's highly effective. Right Kessel runs simply would not exist. If it were not for pivotal labs and the DoD DevOps movements likely wouldn't exist either that pairing model of training people and building them up as practitioners. I do really want to emphasize though.

Leadership coaching leaders are the ceiling on your DevOps transformation. If you want to scale DevOps, you need to scale leadership ability, and you need to scale it first. Okay.

So now you've got a great team some great leaders all of whom have great coaches, right? You're good. No.

The truly educated never graduate as my Mom liked to tell me even with our gritty growth minded team and our coaches. We still didn't have all the experiences. We needed to sustain our fight against the bureaucracy as the Saint Mattis, you know, the patron saint of Quantico said if you haven't read hundreds of books, you are functionally illiterate and you will be incompetent because your personal experience is alone aren't broad enough to sustain you ladies and gentlemen, my favorite Marine General get your whole organization reading make it a part of your culture have book clubs and ceremonies that reinforce it you are going to make a dozen mistakes. In fact, you're making dozens of mistakes right now, but you don't have to learn the hard way because somebody else already probably did Discovery is great, but it's super expensive costs a lot of time and it costs a lot of money to learn the hard way research is better and cheaper leaders a readers.

So read more books lust now everyone in our Enterprise. This is about the time, you know, 2018 that everyone started lusting after Google, Kubernetes all the things SREs. everywhere and that made it really hard for us to get funding for simple applications to solve real user problems. The next most valuable things instead. They wanted us to build AI, ML, blockchain, quantum or a giant kubernetes platform to rule them all. That one's not a joke. In fact, it's kind of a joke.

Now that you know, this is just how you get funding.

So if this is the world we live in how do you get funding for things that actually matter? Well, you have to leverage those early wins with the customer gather data on how good it makes the company look and how good it could make the P&L look or if you're already in Ops doing real stuff how good it is making the P&L look get leadership aligned on those same objectives and make sure everyone's on the same sheet of music of what those objectives are. Everybody's oriented around the customer and make sure that those balanced teams we talked about earlier have direct access to the customer and can ship to them on demand. Once that's in place. You can use growth boards in place of your normal quarterly events where funding decisions are made based on problems and teams and their ability to solve those real problems instead of crazy ideas and solutions about blockchain and Quantum. This not only makes it easier for you to get more money, but it can also make it harder for lower performing teams and initiatives to get their money and at some point scaling your organization and your movement is require a reallocation and you don't want to wait too late to have that conversation.

Moving on to gluttony sometimes before you start other times. Once you deliver the first thing someone wants you to do all of the things at once this happened to us not only with our stakeholders, but actually with our users as well with users. I'll say if you just continuously deliver continue delivering value. It tends to build trust and that works itself out.

But with stakeholders, you need to take some time and space for your evolution of your capabilities one really effective talking point that we used is Gall's law. It states that a complex system that works is invariably found to have evolved from simple systems that worked. The inverse is also true complex systems designed from scratch and never work and they can't be made to work. You have to start over with working simple systems. A lot of times. It's really easy to point out several if not dozens of projects in your organizations that tried build a complex system from scratch and failed in the DoD. It's 94% of IT projects behind schedule or over budget and 40% never deliver a single thing.

So I had lots of ammunition ask for time to scale your simple system into a X1 yeah, the first thing you do is not super valuable. It's not the biggest thing. It might have been a low-hanging fruit.

But you delivered something as opposed to a bunch of people have been delivering nothing and it had some value instead of no or negative value.

And so use that to give yourself some space Also find modern architects who operate more like gardeners and really embody this modified quote from F. A. Hayek. The Curious task of modern Enterprise architecture is to demonstrate to the Empire how little they really know about what they imagine they can design. He's got the lightsaber Spotlight saber earlier.

Traditional architectures in our org they could design some Stellar systems on paper right lots of great box diagrams, but we know that Central planning and complex distributed systems doesn't really work that way. You can't just hide all of your complexity in a box labeled ESB you you have to be empathetic and kind what I just said is kind of mean you should be empathetic and kind but you have to point out that this is no longer the way that we do things.

But again this only buys you time eventually you do have to deliver on larger outcomes on systems and systems of systems.

And so give yourself self the space to do that and then create a flywheel start small and scale fast build great products create a great customer experience leverage that to get a shareholder experience that gets you more resources.

So you can build a bigger team and continue building more products and keep turning that flywheel and building and building little things can add up to something really big now greed shows up in the Enterprise just as it does in Inferno, we've got hoarders and Spenders based on two incentive structures, right you get rewarded for cutting costs depending on what part of the functional you're in or you get rewards for spending a lot of money both appeared in our journey hoarders want to cut costs and spend as little as possible in order to get recognized and promoted or in some cases because they want to be good stewards of the taxpayer dollar or the organization's money, but the simple idea that while we good development costs a lot bad development costs a lot more. It really helped us get folks out of the mindset that we could save our way to success.

So we had to invest and invest a lot but the cost of value ratio actually went down in the process and so collecting that data and being able to show the price per production outcome versus the price per hour or the total burn is super critical and with the Spenders. Well, they're promoted based on how much money they manage and the more they spend the bigger their budgets get every year. It's a lot like death stars though lots of outputs, but in the end the outcomes can be completely undone by a few Rebel ships helping these people plan their budgets then for outcomes over outputs and showing them where they can make more with more so that right more with more they have more money show them how they can do more with it and that gets them aligned to their incentives but also to ours

People definitely got mad. I am sure you've experienced this. For me, it looks like this they got mad when we moved our facilities off base. They got mad when we didn't wear uniforms when we weren't doing waterfall believe it or not. When we chose our name Kessel Run when we had stickers when we had t-shirts, I got an ear for earful more than once. They sometimes felt like they're just waiting for us to mess up and then I get an earful for that and threats of defunding and other things it wasn't a safe environment to fail.

Hopefully you've seen this from Google's Project Aristotle today. I'm just gonna highlight how important the number one finding was you're transformation will fail if anger makes its way into the team. If they don't have psychological safety, if they can't feel safe to take risks and be vulnerable in front of one another and you might be the person where the buck stops.

So either the leader of the transformation like myself Adam and a few others or a sponsor or a champion like a four-star general is real nice. They can create a bubble for you.

But same thing here, it's just creating space going back to this diagram as you scale. You'll need more Champions and more sponsors and I'd be remiss if I didn't mention that you're going to need really good PR for that and sales and marketing to shape the Enterprise and experience and create loyal brand Advocates a brand is a perception and those can be shaped to influence behavior, and that's what's needed for success.

It might feel uncomfortable especially in like government to be using branding and marketing and PR internally, but it has to be done. It's the only way that you can build the transformation. now

I don't see why you can't do it like this and some other program right doing the agile DevOps. There's always some new heresy and it was never actually a heresy we've never like forward looking. It was actually always backward looking they would tell us to do like SAFe or worse, you know, they'd say like DevOps is so outdated you guys need to do the DevSecOps and the BizDevSecOps with tests and you know, they throw like a million things in there. They started this isn't a joke. They started calling it DevStar.

So that's that's real.

So this is a parody from scaledagiledevops.com. Bryan Finster speaking elsewhere. He runs this website, but I really feel like it could have been real for me. There would be these Enterprise moves to adopt safe for similar and they'd want you to stop whatever you were doing and go get certified as a sad M effort with a few sad AF facilitators and do some fractal development.

How do you address this though? Since it's probably somebody's sponsored initiative. Probably somebody important the best antidote here is to make work visible and measure what matters I always recommend DORA is a great starting point, but know how to use them and like how not to abuse them, please but when everyone is getting measured on these things in prod it becomes apparent which ways of working are producing the best outcomes or which ones aren't at all the more you can dive deeper then into making work visible underneath these top very surface level metrics here the better story you can tell there's some great books that I'd recommend if this is a pain point for you, I'd start with accelerate really understand DORA metrics at a deep level make work visible to go deeper on what's Happening. Workflow then project to product which can really help you translate that to leadership and move the whole organization in this direction and finally measure what matters one of many books on OKRs, but I think a really great goaling system that can help you out here. I lumped a violence and prod together because I would run out of time if I didn't but also because I find them to be two sides of the same coin of creating an exercising power in the Enterprise. That is to say they're both very Machiavellian people use them people who use them believe in this statement that power is the pivot on which everything hinges. I do not but inside bureaucracy people use violence and fraud to gain power to try and enable this way of working one of the first things you can do assuming that your senior leaders aren't the machiavellians is to create contracts on your team and between teams we do this for APIs, so why not do it for team interfaces human interfaces have a culture manifesto with defined values and principles don't overlook. This don't overlook documenting it in the process of writing it and getting everybody on board talk about it a lot make people sign in agreement saying that they will forgo dishonesty power Etc as tools at work. I'm serious and do this with my employees Paul Puckett, Army Enterprise Cloud Management Agency giant organization. Runs, all of army Enterprise cloud has his employees do this. It brings us serious to the situation that's really important doing this with other teams you work with as a community can be really important, too.

Another way in which these manifest is as mandates on the violent side and as theater on the fraud side, so both of these either end or corrupt learning respectively so building the metrics that you've now established building on them by measuring what matters making work visible you can push now for what I call metrics over mandates right using data-based decisions instead of emotional pleas to drive mandates and adoption. You can also push back on the thespians by pointing out that it didn't really happen #ProdItDidntHappen. You should use that one. I really it's my favorite.

Finally we come to treachery treachery in my experience came from Mostly the other Rebels to be honest in the book Rebels that work they point out the difference between good Rebels and bad Rebels. I'm not gonna go into that. I'll just say expect that bad Rebel treachery is coming for you happen. It's gonna happen except it.

But the natural inclination then is to compartmentalize information and go into secrecy do the opposite be a good rebel. Secrecy is actually what creates the opportunity for betrayal instead give ultimate transparency access people with weaponize it at first but over time as you produce outcome after outcome after outcome and continue to push metrics and Alignment that you've built you'll get there but one caution don't give credit where credit is not due. We did this early on it. We would give people credit to get them as sponsors and champions. Oh you did that. It's its own form of secrecy who really did a thing. It can therefore be betrayed and credibility can be stolen which who cares? I don't care if somebody else gets credit. That's what I used to say, but that's a problem for two reasons because one the bad Rebels can use that credibility to undo you and they can also create a problem where you can't really Escape bureaucratic Hell Without credibility.

So my final 12 plus 1 it's a theme in Inferno. If you're not familiar the plus one quick trivia on Inferno. Did you know Lucifer is actually not portrayed as being in charge of Hell being there only had the power they were allowed but faced with Summa-verminoth like this one. We often feel powerless bureaucracy superpowers making you feel like you're alone in powerless, but you're neither the ultimate betrayal is accepting powerlessness. That's another talk into itself.

But this kind of brings things full circle. Don't wait for an invite don't ask for permission. Don't wait for a funded pitch, but at the same time don't do it alone get people to come with you stare Summa-verminoth straight in the face and walk right on past or go at light speed.

I've got a quick recap. I'm not going to go through it. This will be available in the slides after I will say if you're in charge of an organization and you want to set conditions for Success. Definitely check out Max reels talk tomorrow on that takes a slightly different angle on this and do I have I'm out of time. I'll take Q&A in the lobby. I'm gonna do something really fun.

And I close with something fun since this was more of a meta take on Dante. I'll close meta. It's jarring some what bizarre sobrace yourself any philosophy students of religion in here any fans in the house? Not very many. It's gonna make this even more jarring.

So I'm a huge Saint Thomas Aquinas fan. I'm actually not very religious.

But huge Aquinas fan for those of you that don't know Divine Comedy is actually it drew so heavily on Summa theologica that it's actually sometimes referred to Summa in verse so imagine how happy I was when I discovered that the beast in the Kessel Run was called Summa-verminoth. That really worked out and then where else had I seen Summa in verse? In Eminem's Rap God disclaimer, I do not condone the lyrics of the song. There's some pretty horrible ones. Catch me at karaoke club after a few drinks though, and you might just hear me. Attempt this at JJ fad supersonic speed but fun fact the music video actually features Inferno on the shelf that got me wondering is Eminem secretly a DevOps Divinity sent to guide us through the Enterprise Comedy.

Does he really have the key and the secret to DevOps immortality is the blueprint simply rage and youthful exuberance find out in Parts two and three follow me on LinkedIn. Hopefully next time we see each other you'll be able to say you did nothing but shoot for the moon since. Thanks, everyone.