Host Intro (Gene Kim)

[00:00:11.015] So here is another forum team readout from the Checkbox Project, which I thought was so mind-blowing to me. It actually shows up in Wiring the Winning Organization, which I co-authored with Dr. Steven Spear, who you heard from yesterday.

[00:00:25.785] They did such an amazing job describing what the world looks like when you have such high coordination costs that all you are doing is coordinating in layer three instead of doing actual value-creating work in layers one and two.

[00:00:38.335] So here is Dean Leffingwell, Steve Pereira, Amy Willard, and Chris Hill, or some subset of them. Take it away. Thank you.

Steve Pereira

[00:00:59.165] Thanks for joining us. Hopefully Dean gets plugged in here in a second, but let me give a little overview of the Checkbox Project for everyone, and then I will get into some Q&A with Dean and Chris to bring you behind the scenes and give you a little bit more context. There is stuff that did not make it into the paper, and I think some of that is really valuable.

[00:01:27.165] The Checkbox Project is basically a case study that amalgamated a bunch of our experiences in very large organizations trying to get seemingly simple things done. In the case of the Checkbox Project, it is the idea of adding a checkbox to a customer bill that essentially turns on a partner product sale, meaning you are opting in for a service that somebody else is going to provide.

[00:01:59.365] All you really have to do is facilitate that interaction, essentially firing an API call. But what that really looks like in a highly interdependent, global, distributed organization is this cascade of work and meetings and coordination and collaboration that is required to check all of the subsequent boxes required to do that work.

[00:02:29.215] We borrowed from the brain trust in the room. There were more folks in the room than were able to contribute to the paper, but they contributed to the concepts, ideas, and principles that made it into the paper. Chris, you are working inside an extremely large, complicated environment. How often do you see and hear about this type of thing in organizations like yours?

Chris Hill

[00:03:01.585] It is a good question, Steve. I see it all the time. For me it correlates to the type and makeup of what the company sells and what their identity is. I happen to work for a major telecom. We are highly integrated here in telecom, just as we are in other industries, and what we sell is an integration of sorts, which means the majority of work is going to be cross-cutting.

[00:03:39.855] The biggest challenge I see that we face is that we, along with other industries, are constantly caught flat-footed with incoming work. This process is over a hundred different teams. Why were we not better prepared for this? We are constantly chasing and trying to predict what type of work is coming in. We are constantly trying to morph how we are structured so that the new future work is something we can plan for. Hopefully I answered your question.

Steve Pereira

[00:04:11.115] Definitely. In your context, I think that makes a lot of sense. But I think we could make some implications or assumptions about any large organization. Dean, you work with a lot of super-large organizations. Is this something that you hear and see in the broader industry as well?

Dean Leffingwell

[00:04:36.255] Yes, absolutely. As you recall in that room, we had organizations that were structured quite differently. Some were quite vertically structured, with teams organized around architectural elements and backlogs that flowed down through the architectural elements, a fairly traditional hierarchical organizer around architecture.

[00:05:01.205] We had other groups in the room, big companies that I would call ad hoc. They had literally hundreds of agile teams empowered to do the right thing, working largely independently. The interesting discovery about the Checkbox Project is that it is really hard to do cross-cutting work either way. In one way, at least you have the handle to do it: you can mandate that we are going to do this, and it goes in your backlog, and you disturb those teams. In the other way, how do you even influence a hundred teams to determine that is the right thing to do?

[00:05:27.525] It is a ubiquitous problem. When we started looking at it, we started looking at the nature of cross-cutting work as different. We all try to think it would be great if we were completely autonomous and had basically no interdependencies and were all independent and could just deliver value. The fact is, the larger the enterprise and the larger the application, the larger value pile comes when things are cross-cutting.

[00:05:53.565] You have to do your local work and make sure each product in your SKU set is a good product, but the reality is the customer is going to ask for integrations across those products all the time. It is an endemic problem and a big problem. That is why we looked at what the problem was in this case. The implication is that it takes forever. What is wrong with it? The reality is these are really hard problems and there is no ideal solution.

[00:06:17.065] We brainstormed a lot about what we could suggest that would help, and then we devolved to principles that could apply universally. It is an absolutely endemic problem. By the way, not only large enterprises have this problem; a few hundred people can have exactly this problem as well.

Steve Pereira

[00:06:35.385] Right. Once you bump up against Dunbar's number at any level, you have these coordination, collaboration, and communication issues. What stood out to me about what you just said is the idea that you have these challenges whether you are in a deterministic environment, where everything is very controlled, or in an emergent environment. Along that spectrum, you will see it manifest differently, but it exists regardless of where you lie.

[00:07:14.945] The other thing I am pulling from this, that may not have made it prominently into the paper, is that this type of thing is not uncommon. We always think of these nightmare scenarios or really challenging situations as outliers, things that will happen once in a blue moon. But this kind of pattern repeats itself often, and it can be very hard to spot, because each individual effort looks unique and might play out slightly differently.

[00:07:53.345] You need this ability to see what your obstacles and communication issues will potentially be and be able to navigate that. There is a big difference between stepping on Lego pieces in the middle of the night when you cannot turn the lights on and when you can turn the lights on. Why go through it if there is an alternative?

[00:08:21.945] This paper spent time on the problem state and how you might approach it, but with a degree of flexibility to accept that we have this spectrum. Each organization is not only unique, but the efforts, divisions, and teams within those organizations are also unique. Is that part of why the focus on principles, Dean? Why are principles valuable in this context, and why might they be useful to organizations dealing with this large-scale coordination?

Dean Leffingwell

[00:09:04.645] The bad news and the good news is the principles are abstract, so you have to apply them in a particular context. Things we determined, like beware of initiatives that change the business model: that little checkbox project changed the business model. Revenue flowed from different sources; it went to different places. It required partners, business partners, customers, and support. Whenever you touch the business model, you are basically touching fire.

[00:09:31.825] Secondly, you have to be able to see the work. Simple principles of lean, like visibility: if you do not know where that work is, unless you have some kind of connected Kanban systems or flow-based system where you can really see the work, you do not even know what is happening. If there is a key component that has to be touched in order to deliver the end-to-end value, you cannot even see it.

[00:09:56.565] If you start to think in principles, you can back up and ask: in my company, do I have the visibility? Does this initiative change the business model? Do I have the architectural APIs to support it? Will my architecture even support this thing? As you back up to the principles, you start to get a handle on it no matter what your company is like, deterministic or ad hoc. The principles apply in either case.

[00:10:17.545] That is what Deming and Reason brought to me: if I can get back to the principles and drive them, then I have a chance to provide guidance for everybody. As a group of strongly motivated and strongly opinionated stakeholders, we even struggled to get to that. But we eventually did, because in all of our organizations this applies. You have to be able to see the work, respect cross-cutting work, make sure the architecture supports it, and be aware of things that change the business model. The set of principles is warning and guidance that anybody can apply if they are going to be faced with this kind of work.

Q&A

[00:10:52.825] Steve Pereira: Chris, we got a question from the audience. I am going to throw it to you first. It comes from Alex: do you find that this problem is easier when architecture is aligned to business domains?

[00:11:05.605] Chris Hill: It is a really good question. I think the obvious answer is yes. No matter what, though, architecture gets used as a crutch, an excuse sometimes, or as leverage for how to change the way things work. Sometimes that crutch or leverage can be successful, thinking how software is structured.

[00:11:36.985] Chris Hill: But to Steve's point earlier about the fidelity of the campaign, or the short-lived work construct, it can be so different from the next one that comes in that the architecture decisions you made on the previous one could not actually work out for you on the next one. You may think the future work is going to match what those interfaces were set up to build, and then find that none of that investment was worth it.

[00:12:10.905] Chris Hill: The ongoing desire is adaptive or dynamic capacity that you can fully compartmentalize. I do not know what the future looks like, but I know what I am building is planning for the unknown. Sometimes living in that abstract is difficult to get to, but there is no better place when designing architectures for highly integrated systems.

[00:12:41.715] Steve Pereira: One thing that came out of that for me was the idea of predicting the future. An interesting aspect of the Checkbox Project is that I validated this sense: this is not the first time they tried to do a checkbox project. That is largely true of large organizations trying to do complicated cross-cutting work.

[00:13:12.935] Steve Pereira: In a lot of cases, it does not exactly matter what the order of dependencies is or the type of initiative being pursued, especially if you can look backwards and say, we already tried this, where did we stumble? Maybe we can focus on making that a little bit better. Hoping for the best is a pattern I see in retrospectives where they do the retrospective and then do not line up action items to change anything or implement mitigations or countermeasures.

[00:13:53.005] Steve Pereira: This memory is very easily wiped in organizations where we get distracted by the next new thing, you have new people, and everything seems new, even though we repeat a lot of the same practices and challenges. Chris, go ahead.

[00:14:11.145] Chris Hill: We talk about team-level cognitive load and team-of-team cognitive load, but there really is an enterprise-level load on how many of these campaigns you can do. The more visibility you have, as Dean mentioned, the more you can understand your entire enterprise's capacity.

[00:14:43.705] Chris Hill: You might find that a team of 9,000 people plus another 500 to 1,000 people from the business and supporting roles, 10,000 people, are really only capable of doing three checkbox projects at a time. That is a really scary thought, because now you are having to pin down what could be commitments to Wall Street: that we can only work on three things at a time. Looking at the prospect of 10,000 people who can only finish three of these things at the same time can be intimidating.

[00:15:21.385] Steve Pereira: That is a great point. Dean, I want to give you a chance to jump in.

[00:15:30.105] Dean Leffingwell: Let me comment on the set of principles. One thing all our enterprises discover is that the platforms we build that abstract the functionality, the APIs we can use for things like checkbox projects, are generally not treated as products. If you think about the infrastructure to support things like checkboxes, we spin that stuff together and maybe there is a VP who cares about it, but it really has to be a product.

[00:15:57.045] Dean Leffingwell: It could be an internal product so you know how to impose requirements on it, know how to move work across the system, and provide a governance model. This move from project to product is not related strictly to things like the thing in your ear. It talks about internal infrastructure being treated as a product, with product management, with a backlog, and the ability to see it.

[00:16:19.605] Dean Leffingwell: The only way to get rid of more dependencies is to abstract the surfaces up into a set of APIs that could be used by anybody at any time without having to ping the team and say, I want to do something. Of the things we discovered, platform as product is one of the strongest statements.

[00:16:37.855] Steve Pereira: Gene, I am going to give you the floor.

Gene Kim

[00:16:41.685] We have about two minutes. I just want to congratulate you all. I put in the Slack channel that this was such an inspiration and one of those bracing revelations: how could 10,000 people only ship three checkboxes in a given year? A very bracing observation.

[00:17:01.175] The amount of intellectual horsepower and experience that went into the paper was one of the funnest projects I got to work on. My heartiest congratulations to you, and I recommend everyone read this. It is such a phenomenal piece of work.

Dean Leffingwell

[00:17:17.555] Thanks, Gene. It was a pleasure.

Gene Kim

[00:17:20.905] Thank you so much. By the way, I put this in a Slack channel: I post a slide I am quoting in every one of my presentations now, just to represent how bad life can be. Surely we are not as screwed up as that organization, right?

[00:17:38.465] If you did not pick it up, the Checkbox Project is kind of a reference to The Phoenix Project, but instead of a $20 million program, instead of a company-changing effort, it is just to ship one checkbox. Kudos. Thank you. Everyone should read it and pass it to their friends. Thank you all.