Dr. Mik Kersten

Okay. So the next speaker is someone who should be familiar to most of us in this community. Dr. Kersten wrote the awesome book Project to Product five years ago, which is being used in so many organizations. And the language has showed up in so many of these presentations. Today he's currently the CTO of Planview.

Over the years, he's taught me so much about architecture, which dictates how organizations are wired. And so much of that shows up in the book that Steve and I wrote.

Mik and I were talking earlier this year, and when he told me about a conversation that he had with a mutual friend of ours on how generative AI will change coding forever, I immediately thought that this merited a talk here at the conference. So I asked him to share with us not only how coding will change, but the changes that he sees coming in how software is built. So here's Mik.

Thank you.

Hello, everyone. It's great to be here. And I just want to share with you that for my entire career, I've been looking for these 10x improvements in how software is built, and how our organizations can innovate and bring value to the market, and how to get that as broadly distributed as possible.

So to my surprise, about a year ago now, I'd always thought that these changes would actually come from programming languages. So I actually went back and saw some of the same messaging that's being used now around all these copilots that help you generate code. This is actually a presentation from JavaOne in 2008, so 15 years ago, when I thought these productivity benefits would come from open source tools, they'd come from new programming languages, and they're the things that would take out all that friction and toil from our day-to-day jobs as programmers, as developers.

And I spent over a decade doing that in building code. And what's been so fascinating with some of what you saw in the programming at DevOps Enterprise Summit is that we're now actually seeing just complete step-function change that's going well beyond anything that I was seeing as productivity gains through open source, through agile methods, through some of the automation practices that we all have and that we all apply.

And with that, we'll have the quick emergency broadcast. I'll take that as a round of applause, and we'll continue in just a moment.

So the point here is that we're actually now at the point where we're seeing it possible to multiply. I think we have line of sight to multiplying the productivity of an individual developer by 10x. And this was kind of always the dream. And again, all that friction and distraction was always a frustration.

And I'll now share with you some slides that I co-presented. It's with Oege de Moor, the head of GitHub Copilot, with the data that they presented in their own study of the application of GitHub Copilot. So we're seeing 55% faster coding. This next one is fascinating. The next two I think are fascinating. Forty-six percent of the code written in those repos was generated by GitHub Copilot.

And then the developers, and this is my favorite one, obviously, were 75% more fulfilled because they had to write less boilerplate and less of that annoying code and tests that actually deliver less value.

So I think this is just the start. And this is not about GitHub Copilot. This is about the fact that these large language models can take our thoughts, our intentions, and actually generate amazing code. And I actually believe we'll go from 55% to 5x to 10x over the scope of the next year or two. So this is just such a fundamental shift.

And if you think back to five years ago, when Project to Product was announced on this stage, one of the main themes of that book was looking at the last technological revolutions. The conversation I had with Carlota Perez most recently was in 2019 when we were brainstorming, well, what is the next technological revolution? We went through the age of software and digital. What's actually coming next? And we actually started talking back then about artificial general intelligence. Would that be it?

And she said, you never know when it hasn't started. But she says it'll have the same traits as the previous revolutions had. So some means of production will actually go from being very expensive to much cheaper and more scalable. Before that, it was electricity that really amplified things. My Apple Watch is a little delayed. And before that we had steam and transportation, mass production. All of those things really transformed how much we were able to deliver value to the market.

Now, all of a sudden, if you believe some of those previous numbers and you extrapolate them, and I'll show you how we can extrapolate them, all of a sudden writing code, the main limiting factor on digital innovation, is about to become 10 times cheaper.

So the question really is, how profound will this be? I think it might be as profound as the Internet and electricity. And of course, how will your organization, how will your teams leverage it? How will you get ahead of this curve? Because what we've seen every time we go through these technological revolutions is some organizations get very fast at wielding all this new power, whereas other organizations slow down and languish and then actually get disrupted and decline.

So really the point here is we are in this new age of artificial intelligence. And how will you and your teams apply this? And how will you accelerate some of the activities you're already doing around transformation, around digital, around DevOps, by harnessing what's now available extremely broadly through these large language models and AI?

So I see these three layers I think we all need to think about in terms of how to apply this to delivering value to our customers, to our organizations.

At first, there's this code level, so these code agents that can actually produce new code, produce value for us. So basically write the things that we want to deliver to our customers through software.

Now, I think that's just the start. I think we're actually going to have this next level of agents, and I'll show you a preview of that, that will operate in the value stream. So rather than just helping the individual developer, they'll help identify and resolve bottlenecks across the organization.

And then finally leveling it up to the theme of Gene and Steve's book, Wiring the Winning Organization. I actually think some of these concepts can then be applied, and some of the amazing heuristics and frameworks in the book can actually be applied at the organizational or the strategic level, which I think is a bit further ahead.

But I think each of these has the power to amplify the productivity at every level of the organization. And I think in each of these, what we'll actually see in the coming months, and what I hope we see in the coming years as well, is that this will actually amplify the effectiveness of what we're able to do today.

I think there's a lot of inefficiency in how organizations build digital experiences, build software. To get that 10x across organizations, to be able to build 10x the software, is, I think, an amazing thing that you'll be able to harness. And Sam Altman himself said that he does not think, even though he's got a very good perspective on just how much productivity can be gained through coding, it's not that we'll have a tenth of the programmers. We'll have 10 times the software that's being written.

But again, I don't think we should stop there because, for many organizations, we need to uplevel this as well.

But in terms of how I've been thinking about the code level and those kinds of agents and where LLMs help us there, we've been applying this at Planview. We're actually, of course, doing the code generation. There's things like code brushes that can clean up code. I think we're going to see a lot more of actually the prompts becoming the code. So we have a lot of developers now who do prompt engineering.

So I encourage you to try each of those things, whether it's through GitHub Copilot, Amazon CodeWhisperer, wherever it is. Just start learning how to use code generation, because this truly is the future of the coding profession.

Prompt engineering: you'll have to have developers in your organization learn how to do that. There's been a lot of great content at this conference around that, and I think we'll have exciting new things coming.

For example, I remember a few years back, it took our teams quite a bit of work to move one of our products from Angular to React. Those are the kinds of things that these LLMs are going to make completely automatic through Auto-GPT, where you can actually incrementally modernize an application, write its test suites, and validate that it's working with very minimal user input.

So all of a sudden, all of this tech debt and these legacy constraints organizations have, some of those things will actually start going away through the power of the growing power of these LLMs. So this truly will transform how software is built.

The challenge that I see here is that if you look at the data, and this is the State of the Project to Product industry report that we had at Planview, we ran earlier this year, we're actually seeing that only 8% of the end-to-end value delivery time, the end-to-end flow time, is spent in development.

So 10xing that won't help in these organizations. These are 34 different enterprise organizations, 3,600 value streams. And this is all systems data. This is data that's been extracted from Jira, GitHub, GitLab, the actual systems, and work is getting stuck upstream and downstream of development teams through all these dependencies, these wait states.

So it's not enough just to do that 10x unless your organization has already removed those big red boxes, which again, all these organizations haven't. These are some very large organizations who are in the midst of these transformations. And we have to really get past this.

So I'll dig into this data a little bit more. Right now, to show you how big a problem this is, leaders keep completely overloading teams. And by the way, the data that we're seeing, we augmented that system data with some survey data. Technology and business leaders think that their value streams, so their teams of teams, have 10 times the capacity that we actually measure in those value streams.

So planning is being done without a view on capacity. Productivity is not being properly measured. And of course this is causing these tremendous wastes. Forty percent of team efforts are wasted due to overload and bottlenecks. So putting more overload and bottlenecks on teams, it won't help if they're 10 times more productive, because we're actually just making the problem worse and not addressing the constraint.

Eighty percent of value streams don't proactively invest in technical debt. So again, if this is not happening, we're having the teams do the wrong kind of work because we're not prioritizing the kind of work because all of these things are disconnected. And by the way, the power of these models doesn't work unless they actually have that data connected the way that they can reason about, as I'll show you in a moment.

So I think it's time. The main message here is we can't stop at those code-level agents. We actually need to uplevel this to the level of value streams and actually help us solve these problems that continue to plague our development teams, that really are effects of organizational dysfunctions.

So we've now got a whole library of flow optimizations. In the Scaled Agile Framework, these are called the flow accelerators. We know how to measure flow through the Flow Framework and the flow metrics. So we can actually apply these things automatically.

And we've been experimenting with this. We've actually got flow optimizations that will be applied automatically to value streams. If the value stream is overloaded, why not automatically reject the work or maybe prioritize some very critical work? All of these things can be done once you've got the whole organization modeled in a way that the large language model can access.

Managing roadmaps: again, prioritize the key work but create capacity for reducing tech debt. If tech debt investment goes too low, of course the organization should be warned about that. We need to make this much more visible to our leaders.

And now through the magic of the approach of Auto-GPT, we can actually set things like objectives and key results for these agents to track to. So if we see that our modernization, our dependency reduction efforts are going too slowly, we're underinvesting in architecture or in modularity, these kinds of things can be automatically implemented, where the large language model, basically the agents, are in the loop and actually help us do this.

So the fascinating thing is this is now possible today. So for the last, I'll just give you a short preview of what I've been doing with our research and development teams, where we're actually able to point an agent at the data in an organization.

Of course, the data has to be connected. So imagine the team-level data, the agile plan data, the OKRs, the investment data, and so on. And I can actually ask an agent this question. So this is a copilot. In this case, it's the Planview Copilot. And I can say, "What should I be worried about?"

And it's able to take all this data and tell me that the team is overloaded because the flow load exceeds 1.5 times the flow velocity. So it's actually able to apply from all the body of knowledge it has access to, from the RAG that we added by giving it Project to Product and other relevant literature, and from the data that it's seeing here. It actually does all of the computations.

GPT-4, and to some degree Claude 2, but nowhere near GPT-4, can actually do all of these computations and all the data it's seeing and tell me that we've overloaded the team and we're going to miss our goals.

It's also noticing that the rate of defect fixing is taking more and more capacity within the value stream, so they have less ability to deliver features and innovation. And this is now possible to do at the scale of an entire organization, because once you connect the data, once you feed the right data, it can actually highlight these antipatterns and these problems for you, and of course present it at the right level to the right stakeholder.

So I can actually ask it. I've asked it many countless things at this stage. I've asked it to explain flow metrics to me as if I was a five-year-old child. And it will pick cookies and tell me how many cookies are being baked concurrently. Of course, that's the flow load on me and my friends.

Also, with the prompt engineering, I've realized that whenever I'm demoing something like this to an executive, I actually always will append, "And explain this to me as if I was a five-year-old child," because then it uses very natural, normal language. And yeah, don't read too deeply into that.

And that is, by the way, a trick that you can do with GPT-4. If you ask it to explain things to you as a child, it will go through these very thoughtful processes.

So the neat thing is, if the data is curated the right way, we can now interact with it. And I think this will empower leaders at every level of their organization to help support their teams, remove those bottlenecks, and remove the constraints.

And then I think we can go to the next level. I think we're actually going to get to the point where if we've got large technology replatforming, things we need to do, it's going to be possible to have those happen in a very automatic way.

If we have organizational structures we need to look at, we now have from Gene and Steve a whole new and extremely simple and effective vocabulary, applying things like simplification, amplification. If the agents are not seeing that in your organization, where should you look at improving that and provide leadership with guidance of what those organizational dysfunctions are?

And finally, investment allocation. If the investment data's there and you're not getting an ROI for a particular initiative, if your latest launch or replatforming modernization effort is falling behind, of course this can now be much more visible, purely because once you send the right kind of connected data, organizational data, basically this organizational work graph, the LLMs have this incredibly powerful reasoning ability that I actually know exceeds my own as a product manager and as an organization leader.

So I think the key thing is now leveraging this, making sure all your data's connected, and just getting started.

So I do think we're going to have this cascade now where there's a 10x. We're not there yet, right? The numbers you saw earlier from the code-level agents were 50%, but a potential 10x capacity gain at that code level of individual developers. Then another 10x at the value stream level. I mean, we know that's possible from the data we're seeing in these value streams, the data that's flowing through them. So it's possible to unlock that, and even a 2x will have a massive impact on your organization, on your customers.

And of course I think it's much less clear what will happen at that strategy level. But the fact that you can use all that reasoning and, again, apply the framework that we have, such as that from Wiring the Winning Organization, will allow the LLMs to apply those automatically and look for where we improve.

So quickly, in summary, I think in all of this, how we leverage AI is all about people times machines, not people versus machines. We want to amplify the work of our teams, of our leaders, all the way up the organization.

To do that, you actually do need to make sure all of that work is connected. Because as soon as, for example, we're not feeding the LLM a dependency, it can't help the same way that once you're digging in manually through this yourself, you won't be able to make that same conclusion if you're missing data. So all work has to be connected for these LLMs to leverage it for you.

And I would say just start adopting now. If you're not adopting code agents right now, do it. If you're not trying out these value stream agents, do it.

And the other key thing, of course, getting back to flow, is you need to actually make the economic case to the organization. So what we've done, of course, is looked at how did flow improve for teams that used GitHub Copilot. Because it's not enough to look at how much code was generated, but how quickly that value got to market and how the flow velocity improved. So of course these things are very constructive, and we now have these LLMs in the loop.

So I'll wrap up now. But in terms of help, I'm looking for any learnings that you have applying it at these various levels, especially that code level, the use cases that you found really working, as well as the value stream level. If you start experimenting with that, I would love it if you could share it with me and the team.

So it's just mik@planview.com for more. So with that, thank you very much.