Host Intro (Gene Kim)

All right. Thank you. The next speaker is someone who should be familiar to all of us in this community. Dr. Mik Kersten wrote the awesome book Project to Product four years ago, which is being used by so many organizations to change how they think about their software efforts.

Mik taught me so much about architecture, which is so important to enabling fast flow, which I think this community understands better than almost any. In fact, it set the stage for the work that I've been doing with my mentor Dr. Steven Spear on how important architecture and the structure of the organization is to get the outcomes that we want.

We know that we can create a structure that fully unleashes the full creativity and problem-solving capability of the organization, or we can create one that suppresses or extinguishes it entirely. But one other aspect is important, which is signals can be amplified or signals can be suppressed. And these are the signals that leaders need to know whether the system is actually working, regardless of how good the architecture is.

I'm so excited that Dr. Mik Kersten will be presenting a real-life case study of a reorganization that he did at his company. He'll state the benefits that he expected, what actually happened, and what he did about it. So I think this is such an important talk because he models so explicitly how we need to think about organizational design and something I've really seen, but I think it is so important to get the outcomes that we want.

So here's Mik.

Mik Kersten

Hello everyone. I'm thrilled to be here at my favorite conference with all of you in person and building these new memories and relationships that I think will keep us going through the next year of Zoom calls.

It was, as Gene said, four years ago that Project to Product was launched here. Since that time, I've had a chance to learn a lot around what's worked, what's been easy, what's been challenging, about helping organizations shift and really elevate these principles of DevOps and agile to the business as a whole.

Over that time, also my own career has changed as well. I started as a developer, where I actually found it very easy to look at how we were able to transform our software architecture and open source projects, transform the team structure, and really change around our value streams as our projects evolved. But as I stopped doing actual work and just started helping other people do work and take burden out of their days, as I shifted from being a developer to being a CEO for 15 years, most recently with the acquisition of Tasktop by Planview, I've now become a Chief Technology Officer for Planview.

So my scope in terms of engineering has grown some more. I've realized that I needed the same kinds of tools I was using for software to better understand what was working about organizational structure. It's really interesting, actually, that in the past few talks we've heard a lot about abstraction, about modularity, and about organizational and functional design. I've been applying these in this management system that I've applied at Tasktop and now at Planview, looking at how we can better measure and understand the organizational structures to see: are they making work easier for our staff, or are we actually adding burden and toil into the systems?

I'll talk about that today, really from the perspective of how we measure the flow through the organizations that we create, how we measure these signals for the structures that we've set up, and without giving so much advice on what structures you want to end up on, but really how you take a disciplined and data-driven approach to better understanding flow.

The really good thing, I think, over the past four years is that I have had a chance to witness how this shift from project to product that so many in the community have been working on has helped catalyze change. We've got a whole new kind of alignment in terms of business outcomes that just wasn't there before when we were kind of throwing work onto agile teams. A lot of organizations have actually changed how they budget and plan and take a whole new approach to strategic portfolio management with their software, of funding value streams and having fixed capacity.

There's been this really interesting evolution of organizational design, because a lot of organizations have realized agile teams are not enough. Functional teams are kind of easy to deploy, but business agility is a little bit harder. And all of this, I've realized, centers around this notion of value streams, and actually defining and refining how we manage and measure our value streams.

The value streams, we know, need to be aligned to the customer, to delivering business value. They need to be autonomous, and as you'll see in the talk, when they become less autonomous, the flow of value slows down. They need to be cross-functional, because we can't squeeze all that cross-functioning into a single team, as you probably know from Christoph's previous talk on Google SRE. And of course, they need to be customer-centric. They need to know who their customer is.

Now the challenging thing is, and a lot of my work has been supporting organizations that work at very large scales, these value streams need to span functions. So span disciplines like engineering, architecture, design, business analysis, SRE, portfolio management. They need to span hierarchies, because a lot of organizations have created these functional hierarchies. They need to span geographies, and oftentimes they need to span sourcing partners as well when you've got that kind of global complexity.

This problem, I actually heard Charlie Betz, who's in the audience here today, calling it the messy matrix. So we're always looking at: how can we structure the value stream, the organizational design that we have today, in order to make them effective, given that we're going to cross organizational boundaries and silos and functions?

I think the challenge is, for a lot of organizations, this has just not been done effectively. Out of this is we've got well-defined strategies and all the best intentions of creating the best customer experiences, delivering value to the business. We've generally got well-formed teams because agile teams are effectively a solved problem. But this middle layer, value streams, where we need flow, where we need to track business outcomes, where we actually need to make sure that we're tracking to the happiness of our staff working on those value streams, tends not to be defined, tends not to be properly formalized, and is actually suppressing all sorts of signals from being fed back up to the strategy.

The signals around how much we should be investing in technical debt, when we should be replatforming, what's putting toil into the systems, which organizational structures are working, which ones aren't, tend not to percolate up when value streams are not effectively defined and when we don't have effective ownership of them. Then that means that we're lacking the alignment of actually making sure that the strategy is connected to delivery, that the teams are working on the right things, that we've actually properly staffed our platform teams, because typically they tend to be understaffed. A lot of the resources and teams get put on the business-facing products; not enough gets put on the platform teams that actually help accelerate the delivery of those products.

This disconnect I've seen traveling across, I think, the majority of enterprise organizations. I'll show you just some quick data of some of the effects that we've seen on this by collecting data from over 3,000 value streams across 40 large enterprise organizations, collected through the Planview Tasktop Viz tool. Some analysis, some data mining that we've done on this, and what we're seeing — and this will give you a sense of how problematic this is — is that across those value streams, 8% of what was planned by agile teams was delivered in the timeframe it was planned for.

So there's this complete disconnect in terms of how we're planning and the demand that's there to innovate for customers, and the actual capacity of teams working. The next one is even more bothersome to me: 20% of features are canceled after code has been written. You know how demotivating this is if you've ever written code when the business changes its mind or something's changed to the strategic plan, because again, these things are disconnected, the way that planning is being done and the way that teams are working.

Thirty-five percent of products have zero capacity for new work for the next 12 months because they're undergoing all this change. And again, we've got these disconnects. And then the disconnects in terms of the signals: 85% of products underinvest in security and debt because, of course, those priorities are not making it up when we don't have this properly formalized structured layer of the value stream.

I do think there's a better way. I think that the work that Gene Kim and Steve Spear are doing right now in terms of better understanding how dynamic learning organizations work and how to formalize this structure and dynamics points the path forward.

For me, the real aha moment came when I first met Gene in 2016 at a conference. I thought I had to meet him. I was just amazed by his work and showed him this visualization I had just made of what I thought good looked like. This was a visualization using the course tool Gource — Gene, you might have a little flashback here — the commit history, the source code history for the Eclipse Mylyn project I was working on over several years.

What's happening here is you're seeing all the commits and pull requests, and as this is happening, the modular structure of this codebase is forming. It's actually a large codebase. It was evolving quickly, had a lot of committers, and what we're seeing is that encapsulation, what we heard Christina talk about, where these modules are forming. They're encapsulating. They're hiding the right kind of information.

You'll actually see these orange rays darting out. Those orange rays are me constantly refactoring the codebase where, likely, encapsulation is wrong. I'm seeing signals that we're not hiding the information. These APIs are not quite right, so I'm constantly doing all these refactorings to make the work of other contributors and other teams using this codebase easier. That, of course, relies on me getting the right kinds of signals: is the encapsulation right or is it wrong? Where do I change? Where do we change it? Where do we make it better for people to contribute?

As this is going on, you actually get a sense of the encapsulation improving. This open source project, for its time, was actually very successful, and it gave me a sense of what good looks like in terms of making sure it's easy for people to contribute value and to maximize flow.

In terms of good modularity, one of the main things that we rely on is high cohesion. Every single module encapsulates common functionality, and that functionality can be reused as building blocks for software. It gives us that nice feeling of Lego as we're building software and being able to recompose it. Of course, the whole goal of software modularity is to manage the immense complexity that we're dealing with, so high cohesion is critical in doing that.

Then low coupling is the other key principle that we need because we need to decrease dependencies between modules in order to be able to evolve them independently, in order to hide information, so that we've got the right layers of abstraction, and then to avoid these cases where we get into these spaghetti code messes.

What I realized is the same kinds of things apply to value streams and organizations, where we need to encapsulate expertise and ownership, and we need to decrease dependencies between value streams. The more we decrease them, the more autonomous they are, the faster they can go and the faster they can move.

Really, the role of leadership — and this is where Gene and Steve have these concepts of configuring the organization and then running the organization — so when we're configuring the organization, we're designing it, and then we're assessing it. We're assessing: did the configuration work, or do we actually need to change the encapsulation?

The principles for that are structure — the way that we structure hierarchies, the way we structure dotted lines and matrices in the organization — and then the dynamics, the signals, how they flow as people work. Was it easy for them to work, or did they have to go to ten people to get permission to do work?

We've got these new toolkits that have come out of this community. Team Topologies and other approaches to organizational design give us a common vocabulary as well. And the Flow Framework — and this is really my main goal with Project to Product — makes it easy to measure these kinds of changes as we're making a shift.

Now I'll quickly take you through the Tasktop for-your story. Four years ago, as we were obviously evolving our shift, growing quickly, our shift from project to product, our hypothesis was that we should actually merge the product and engineering organizations into one organization, all hard-lined everywhere, one product organization, and that this would accelerate flow.

The expected outcomes were we'd have more customer focus, less us-versus-them, platforms — all platforms would be elevated as first-class products, just like our customer-facing products. And of course, we'd use the Flow Framework to see: did this work? Did it accelerate flow or did it not?

Now, in terms of the assessment, I started getting a feeling last summer that it wasn't working exactly as expected. What was happening in terms of the Flow Framework's business metrics, the value was there. So the Viz product's ARR had grown over 200% year on year. The team size had doubled. The hosting cost was also about 2x, which is okay.

But the challenge was that the flow velocity was trending to basically flat, even when we compensated for the onboarding time. So velocity, the flow, was not accelerating. Something was off. And then the happiness was showing a steady decline — the happiness of the staff working on the product, all of those various teams.

The diagnosis, as we dug in deeper, was the cohesion had been dramatically reduced and the teams were constantly reporting all this cognitive overload of having too much work put on them, too many complexity domains put on them, and the coupling had just gone through the roof.

I will never forget this moment where it was announced to our whole product-engineering organization that we have to 10x the amount of collaboration between value streams, because that's actually an anti-pattern if you've got that much dependencies between those value streams. So it was time for a refactoring.

We refactored the organization earlier this year in February, where we again split out product and engineering. So engineering now had engineering leaders. The organization was still very much matrixed in the sense that value streams are everyone's first team. SREs are more embedded.

Again, the point is not exactly where you end up in this matrix. The point is: is the structure meeting your business needs? Is it making work easier for people? In our case, it was sort of a similar thing to what Amazon noticed with two-pizza teams and single-threaded ownership. We kind of fell for the theoretical ideal of single-threaded ownership, of everything being one in one organization. What was better for us was actually splitting these things apart. Then we realized we were actually making much better progress on our goals of replatforming.

Really the whole goal here is to put this in place. You need to formalize these value streams. You need to measure: are we moving faster? Are we moving slower? You need to make sure that those signals are getting to you. My challenge was that not enough signals were getting to me other than through these metrics, because a lot of engineering signals were being suppressed at the time. And again, measure those flows and outcomes and connect delivery to strategy.

Very quickly to summarize: define your value streams. We've got the tool that's there: Team Topologies, single-threaded ownership, you'll hear Dean laughing while I'm talking here about the Scaled Agile Framework, SRE, those various models. Then make sure you're constantly assessing what you've defined via flow metrics, measuring business outcomes, and as well as happiness of the staff, which is a really important leading indicator.

The key thing is continually refactor that so you need low coupling and high cohesion for product value streams. Your organizational design and your software architecture — really think of those as one, and apply those principles of modularity.

You can learn more. Check out our platform at planview.com, which makes this easier. Check out the podcast where I talk to a lot of my mentors about this. The help I'm looking for is if you could document some of your own experiences measuring flow and organizational design with the Flow Framework community. Thank you.