Carmen DeArdo and Paul Shepheard

Carmen DeArdo: Thank you for joining us. We're going to talk about tracking business impact: what's the point? Paul, you want to?

Paul Shepheard: Yeah, sure. Hi, everyone. My name's Paul Shepheard. I work for Deutsche Bank. For the last three years, I've been running a function called Delivery Platforms and Services, where we try to create the environment in which people can build good products by making sure that processes and tooling and ways of working are aligned to good practices along the lines of what we've been hearing about over the last three days.

Carmen DeArdo: And I'm Carmen DeArdo. I'm a VSM strategist at Tasktop, and basically, I just try to keep up with Mik and Dominica.

We try to start with: what's the problem? I think we've heard some of this. We've implemented Agile. We have all kinds of new tools and technologies. We've done a lot of work internally from an IT perspective, but yet, does the business care? Is the business happier? Are we seeing any impact to the business of what we're doing?

When I take a step back, and I spent some time at Fortune 100 companies, what I saw is you need a place to start. It's a journey. It's really not a transformation. I think Jon Smart talked somewhat about that today. We will transform. Agile's a great place to start, and you need to do code to cloud. We know that the right-hand side of that value stream's important. However, there's this connection back to the business, and we have to look at the entire value stream and look at increasing flow across the entire value stream if we're actually going to achieve our goal of being more responsive to the business.

Mik has talked about some of these in Project to Product. I'm not going to go through all of them, but I think a couple of the key ones are the disconnects: IT speaking a different language. Mik tells a story of going in. The CFO doesn't want to hear about story points. It's, what are you doing for me? What's the business value? This black box idea where you're solving your own problems: why do you want Docker? I don't get it. How's this going to help us?

So how do we bridge that gap, and how do we show there's value in the work that we're doing from an IT perspective? I get to talk to a lot of different customers and clients, and we have this discussion and I say, okay, what are you doing? Well, we're doing Agile. We're doing DevOps. And it gets around to the point of: okay, that's great, but that's not a goal. Doing Agile and doing DevOps is a means to something. What is it a means to?

I think as you start to get through it, where you try to end up is it's a means on accelerating the delivery, and I think Dominica's added, and protection of business value. You're really trying to reduce that flow time from the concept of something, some idea, some hypothesis, as Jon Smart talked about, to actually getting delivery and feedback. Having that true goal then allows you to start to focus more on how are we going to start to achieve that?

Here's a quote from Gene that I thought was very good. It talks about business and technologies create value together and to win in the marketplace. How do we turn that mindset around? What are the things that we have to do? There's a lot of things that are necessary, but there's still more things that we need to do to get to this goal.

You've seen this slide a few times, but I'm going to put a little different spin on it. My view is if I had one or two questions to ask a company about where I think they are or the likelihood of them making it through the turning point, one question I would ask is: are you still seen as a cost center? The companies that are making it through the turning point have turned that around to where they're seen as a profit center.

The last thing I'm going to say, I promised a Churchill saying about DevOps, was I think there's a lesson from Churchill that we're not at the end of the journey. We may not even be at the beginning of the end, but maybe we're at the end of the beginning. Deployment is not the end of the journey. It's just the beginning of what value we hope that we get from that deployment. With that, I'll turn it over to Paul.

Paul Shepheard: Thank you very much, Carmen. I think that's framed the problem nicely. When I meet teams, and I'm not just talking about internally at our organization, but at other organizations and events like this, at the Lean Coffee that I was at earlier this week, when you talk to people about value, value is somewhat absent from the conversation. People talk a lot about techniques and practices to improve ways of working, but what is the point of all of that?

If you look at the statement that has rung true for me for a while now, many teams, products, and companies really do struggle to correlate the investments that they make with business outcomes in an objective, measurable way. That's an important thing to do. If we want to bring the businesses and the technology teams together, we need to find the language to be able to demonstrate the blood, sweat, tears that we all go through to take these organizations into the future, into better delivery shops, that they're having tangible benefits. We need that for us. That's feedback for us. If we're not creating more value over time, then that's important for us to know so that we can adapt and we can change.

So what is the point? It's not an existential crisis question. What is the point of what we do? It's value, and it's right there, first principle of the Agile Manifesto: our highest priority is to satisfy the customer through early and continuous, that's important, delivery of valuable software. Not just software. Valuable software is the point.

Let's look at where we're up to as an industry. We've been improving software delivery for 30, 40, 50 years, and actually, we've achieved a huge amount. If we look at what good teams look like, you're starting to see a lot of commoditization in the kind of languages and the practices and the techniques through these events, which I think is a really promising thing because it shows where we are on the maturity curve. Good teams tend to focus on similar things. Good teams tend to focus on three dimensions of trajectory, but towards better: flow, so how quickly can we turn ideas into feedback; quality, so how often do our users get surprised, either positively or negatively; and value, the ideas that we put to production, were they viable? Did they make sense? Did they have the intended consequence or impact or not?

If we look at teams' measurement systems, we're getting better at some of these as well. Measurement's important. If you're a professional athlete, you measure all manner of things. I'm a very amateur athlete. I run, but not particularly well. My watch tells me how slow I'm going, but I don't measure to the same extent that a professional athlete would. We're professionals at what we do. Measurement is really important.

If you look at measurement against those dimensions of flow, quality, and value, you see some interesting things. In flow, you see a lot of commoditization of the language. Dr. Kirsten yesterday talked about things like lead time and cycle time and work in process. Hands up, who's read Accelerate? I'm expecting it's pretty much most people have read Accelerate. Nicole Forsgren's work, along with Gene and Jez, gave us a lot of this language that we're now using collectively, and there's power in that. If we look at quality, there's a lot of commodity in the language that we're using there as well: mean time to recover, mean time between failure, service level objectives, and the like. But if we look at value, if you go and talk to teams, there's still a high degree of inconsistency in how teams are considering value and how they're measuring value.

As we just described, it's a fundamental, important part of that triangle. Why is that? Value is harder to quantify than flow or quality. Flow and quality are consistent across delivery systems. If you use Jira or something else to track a widget into production, Jira doesn't care what that thing is, but it'll tell you when it was born, it will tell you when you started work on it, and it will tell you when it hit production. But value is harder to quantify. It's harder to measure. Value is domain-specific. It's not always about dollars. Value could be risk reduction. Value could be regulatory compliance. Value could be improved security. We all want to work on the most valuable things in the right order, but we have to form some kind of basis to trade these things off against one another. Teams try to take all of these different domain concepts and distill them down into dollars, and that hasn't really hit the mark. People talk about value points, but that's somehow less tangible, less consistent.

Value often is difficult to correlate with any given change. If I put a new feature into my product and my revenue goes up, how much of that is attributable to the change that I made? Is that some other event? Is that a competitor action? Is that a marketing campaign? Is that some kind of seasonal disruption? Value is genuinely harder to quantify and measure, make empirical.

It turns out that part of the problem is with data, and we work in IT. Ninety-nine percent of our problems are data and data modeling questions. Let's look at how we consider value today. When I make an investment, when anyone makes an investment in a system, you really have a hypothesis. People talk about hypothesis-driven development a lot. You say, I think that if I deliver this thing, then I'll be able to get some value. If I deliver this new feature, my revenue goes up. If I fix this bug, my stability improves. Whether you're talking about the level of a feature that might take 15 minutes to code and deploy or a 12-monthly investment governance cycle where you're pitching for a complete rearchitecture of something, the language is pretty much the same: if I do this, I will get that.

Then we measure, and what do we measure? We measure the inputs into the system. We measure the cost. That might be the capacity of a team. It might be the amount of budget that I need to get to do that change. Big organizations spend a lot of time measuring that, and they measure that pretty tightly, and that's important. Then we measure the outputs that are produced from that system. We have the inputs, the capacity, the people, the licenses, the hardware. We then produce outputs. Those outputs are changes. They don't have to be software changes. They might be process changes or something like that. They turn into milestones in plans, and people commit to them, and you track them, and you RAG-status them. I love the watermelon analogy I think it was Mik gave yesterday about things that are green on the outside and red in the middle. So we track those outputs as well. Then you have the outcomes at the end, and that's where it gets a bit squishy. At the beginning, we're really clear: if we get this money to spend, we'll definitely get this thing. As you get to the end, it's, well, things change, market moved, stuff outside of our... And that's real life where that happens. There's quite a chasm between outputs and outcomes.

One of the things that good technology product companies do, and one of the things that I'm seeing teams use increasingly, is to bridge the gap between outputs and outcomes with something that we refer to as impact. Take that single hypothesis that says, if I do this, I will get that, and split that into two separable hypotheses. The first hypothesis is: if I do something, if I deliver something, then that thing will have an impact in the world in some way, and that impact will be measurable. The second hypothesis is: when that impact occurs, then I will be able to derive some value from that impact. It turns out that then gives you an ability to ask some different questions and to expose some different things in the system that enables people to become active in how you drive better value.

That sounds a bit theoretical and hand-wavy, so let's try and make it a bit more specific. This is our hypothetical situation. If anyone works in a data processing environment, they'll recognize this kind of pattern. The box on the left there says nightly reconciliation process. Let's assume that's a box. The job of that box is to look at a bunch of files our client sends us every night. Those files say, I think you're holding this much inventory on our behalf. Then let's look at a file from our systems. Our system says, we think we're holding this much inventory on your behalf. The reconciliation process says if our client says we've got 100 widgets and we look in our inventory and we've got 100 widgets, we're good. Tick, nothing to do. But if our client says we've got anything other than 100 widgets, and we say we've got 100 widgets, that's a mismatch. That needs to be output, and then a team needs to look at those mismatches.

Those mismatches might be genuine, but a lot of the times they're not. It might be because the client file comes in in a weird state. They change the format, they move something about. Actually, the inventory's in the right place, but the reconciliation process isn't smart enough to understand that. And so it produces these false positives, and we have a team of superheroes there that need to investigate those false positives. Pretty traditional situation that you'll see, and we want to make that more efficient.

A traditional investment case for this might be if I invest an amount of money, an amount of capacity to build a more intelligent reconciliation process to better understand. We'll probably use AI and machine learning, and we'll probably put blockchain in there because then there's a better chance we'll get the budget. I'm glad you laughed. If we build a more intelligent reconciliation process, then we'll reduce the cost of that investigative process. We want process efficiency, and that stuff gets funded, but where's the impact? What are we going to produce from that? A more intelligent reconciliation process.

What we want is a statement that says, if I invest to produce a more intelligent reconciliation process that reduces false positives from 700 to 200 per night, then I can reduce the cost of the investigative process by 50 percent. They're going to have half the stuff to look at, and we're going to need half the number of people to do that. Fifty percent of those people can now go and do other more valuable, value-adding activities with our customers figuring out what we do next.

Let's have a look at how that might play out in terms of work. These are months going left to right. The red line is where we are with our false positive indicator, and the green line is where we want to get to. This first thing is a waterfall way of working where we're going to do the analysis up front, then design, build, test, deploy. The question we're asking here is what is progress really? When we talk about making progress, what is it?

Let's play this forward. We go through our plan. We've done our analysis. That's on time. We sign that off. All the right people send their emails. Good. We do our designs. All the architects look at it, and they scratch their chins and nod and say, yes, design's good, on time. That's progress. We go through the build. Build completes on the day that we said that we thought it would. That's progress. Fantastic. Test the same. In the meantime, the red line hasn't moved. We haven't learned anything. The conversation that was had at the closing speech yesterday, when Steve was talking about our job is to go from ignorance to knowledge, I love that. From ignorance to knowledge. Intelligent people are looking at these specs and building code and what they think is right, but it's not validated. There's no feedback there. But this is green. This program is green. It's hit every milestone on time. What happens? You get the joke: we deploy it, and the breaks go down a bit, but we don't get to where we wanted to, and we don't find out until a very long time after we started.

Let's look at agile teams. They delivered on time and on budget. I hear that a lot. These guys delivered on time and on budget, but so what? The business didn't win. We didn't solve the problem. Let's look at an agile team because we're all super smart agile people. We know better than that. We're going to deliver in iterations. We're going to deliver, let's say, every month. Let's not get too far out there. After month one, we say we're going to deliver. Our progress will be software in production. After month one, we deliver software to production, and the breaks went up. Oops. But we know. We did something. We put it in production, and the algorithm that people put in there, for whatever reason, it got through their test. Maybe their test data was inaccurate. It went into production. It got quickly backed out because this went in the wrong direction. But we learn. We learn six times faster than we would've done in the old way.

So what is progress? Progress can be considered one of two things. Software delivered to production, you can say that's progress because I learned something. But it wasn't directional progress towards the goal. It was learning that enabled me to then get better directional progress towards the goal. Maybe the goal wasn't meetable. Maybe the goal was never meetable. But what you get here is transparency. Real progress is measured impact from which value can be derived. When you get to the conversation about how much of the team can now be redirected away from this process and towards other value-adding activities, you've got an empirical basis for that conversation. It's no longer this war between IT saying, well, I delivered the system, you just must have specced it wrong, and the business saying, well, it didn't do the job. We've still got just as many breaks, and now we've committed to this other stuff over here and we're in a hole. That pulls those people apart, and what we're trying to do is bring those people together.

I hope that has made it a bit more tangible and a bit less hand-wavy. The benefits of using impact metrics in this way, there are a few. Progress can be described in terms that the stakeholders care about. If I run that operations team, that reconciliation team, I don't really care about MVPs or PSPIs or what a release train engineer is. I care about the level of the breaks. I care about the customer experience of not having all these phone calls chasing stuff that isn't really real. I care about the things that I need to drive my business. But in a lot of agile systems, we say we want to bring the business closer, and then the first thing we do is give them this new language to learn, which generates friction. Progress is described in this way in terms that the stakeholders care about.

The second thing is alignment on the intent. The intent isn't to build a new reconciliations platform. If you align everyone in the system to the intended outcome of the work, all of a sudden, those people can be active in better ways to solve that problem. Someone that understands the inner workings of that system can say, hey, if we just fix some of the reference data, we can get halfway there in a week. Let's go and do that. Let's not write any code. One of Dan North's favorite sayings that I always refer back to is that if you can solve any problem with zero lines of code, you're winning at software, because software is a liability and has a cost of carriage. That's a different talk.

Transparency. This gives you real transparency. It was said yesterday, feedback is a gift. You can see objectively if you're making progress towards a goal or not, and you can have a grown-up conversation about that. That's a really powerful thing. You get, as I say, this improved opportunity for everyone to be able to contribute to the better outcomes.

Last few remarks on this before I hand back to Carmen. As we've started to deploy this, and not just at our organization, other organizations started using this. A lot of technology product companies are leading the way on this. When you think about it, this is basically similar to your online retailer looking at conversion rates for checkout. If I improve my UX, then more people who put stuff in the basket will eventually check out. Let's make a change. Let's see if that happened or not. This isn't new. This isn't untested. It's a growing trend, but still not common.

You get a lot of the same or similar benefits from using impact metrics as you do from using test-driven development, if you use them well. In test-driven development, you need to understand the pass criteria before you do something. Before you start, you need to understand what outcome you're looking to achieve. If you do impact-centric delivery, it's the same thing. You need to ask the question: how will we know that we're winning? How will we know that we're adding value? How will we know that we're done? You need to understand your success criteria.

In TDD, you need to make that success criteria fully transparent and explicit and measurable. In impact-centric delivery, you need to do the same. These measures are the things that you talk about as your primary measure of progress, not with things like requirements signed off. In TDD, it requires you to create a system that is testable. You can't do TDD without a system that's testable. In impact-centric delivery, you are required to create a product or a project or a program that is testable. How many projects or programs or products do you see today that are testable and that create dashboards that say whether those operational metrics as indicators are being hit or not, whether essentially the product is broken in the same way as you would look at a build and say the build is broken?

In TDD, you have to have transparency so that everyone can see when the tests are passing or failing. The feedback is immediate and people can act. For impact-centric delivery done well, you do the same thing. Your steering committee decks become data, talking about the state of your business process or your product and how that's improving over time. You can see whether you're passing or you're failing, and you can respond to that.

In impact-centric delivery, you also get to carry these indicators forwards. One of the great benefits of test-driven development is that the tests live for a long time. They transcend the transactional piece of work, the feature or the project that you're working on. They become a living record of the intended behavior of that part of the system, or the intended outcomes of that part of the system. If in six months' time, a brand-new developer comes along and he or she changes something that implemented their feature perfectly but broke something that you were depending on six months ago, then that becomes apparent, that becomes transparent, and before that makes it through to production, that can be discussed. Was that an intended consequence? Is that test no longer relevant? Or do I need to go and do something to make it compatible with some other expected behavior that's in there?

With impact-centric delivery, these become operational indicators. Those reconciliations, those breaks that got down to a level that was acceptable for a reduced team, that might be great at the end of that piece of work, and then half the people get reappointed at something else. Then six months later, you onboard a bunch of new clients, or someone screws up some reference data somewhere, or something else gets broken in another system upstream of you, or the inventory system starts sending incorrect data through to that reconciliation process. That line creeps up, and it goes from the 200 mark to the 300 mark to the 400 mark. This allows data to inform people that that is happening and for alarm bells to go off.

There's been a lot of talk about macro-level topics over the last few days. This topic is a bit more micro-level, but it's something that we're seeing is absent, but we're seeing a growing conversation in it. Like I say, there's no silver bullets. We're still learning. At the end, we'll talk about how you can get involved and reach out to us. I'm keen to hear stories from other people of what they're doing and then build up the progress for this together. With that, I'm going to hand it back to Carmen to close. Thank you, Carmen.

Carmen DeArdo: Thanks, Shep. I think it's an important point that Shep just talked about. I talk a lot to customers about running experiments. An experiment has to have three parts. You have to have a hypothesis: what is it that you're trying to achieve? Obviously, you have a set of activities. But then the third part, which I think doesn't necessarily get thought about much, is: how are you going to measure this? I've seen teams run experiments and then debate whether it was successful or not. It should not be a debate. At the end, you have to think about how would you measure this before you even undertake that experiment.

If we go back to some of the things that you've heard about this Flow Framework, there's the flow metrics, and we've heard about that, but then there's also those business results. Sometimes an experiment may be about flow, and it may be how can we go faster? I think Mik gave an example that was from an organization I was part of. We kept doubling down on the create-release part of our pipeline, thinking if we added more capacity there, if we added more tools there, our flow time would go down. We didn't really treat it like an experiment. We had no way to measure it. When we finally measured it, we realized that wasn't our bottleneck. We're only spending 2.5 percent of our time in create-release. It was the other parts of our value stream, the ideate part, the operate part, that were taking up where work was waiting. Because we really didn't set this up as an experiment, and because we didn't have metrics, we continued to apply the practices in The DevOps Handbook, but how did we know if they were really helping us or not?

It gets back to what's the point. Those flow metrics have to be tied to a business value. A value stream is going to have different customers. A lot of your products are not just your products for your end customers. They're your internal products. One thing I see time and time again is we do not treat our pipeline as a product. I was having a discussion at the bar last night with one of our salespeople, and they were going over the fact that every time there's a different tool, there's a different CIO, there's a different CFO that has to sign off on a tool. That's one sign we're not treating all these tools as a product. The tools that we use to develop all the business value on is our most important product. Mik talked about the BMW example. The plant is their most important product, not the cars. We are very poor at that, and I would go as far to say as if we don't turn that around, if you're in a company that doesn't turn that around, I think you will have a hard time making it through the turning point.

You should decide, I have a product. The business value's lying on top of that. We have all these tools as part of that value stream. Invest in that. Let the product owner make the decision. Don't escalate every decision on every tool to a different CFO to sign off on. Architect it as a product, measure it, run experiments to get faster, and then have a way to measure that.

We're going to end with some advice. For business leaders, you have to have this concept of a product portfolio. I realize it's hard, so you have to start small. Start with one or two products. Don't go off and say, in two months, we're going to take this large company and come up with 300 products. That is not the way to do this. Start with a few areas, a few products, because you're going to learn, as Shep said, from doing it, not from talking about it. Make sure that you have metrics around not just the flow, but also the value, the things that are in the top of the Flow Framework, the value, the cost, the quality, and importantly, the happiness of the team. Because we know that if a team measured across that product value stream, not all the architects, all the developers, all the ops people, that doesn't really tell you what you need. This team that's working on this value stream, how do they feel? That's going to tell you if you're getting it right. That's going to tell you if they're frustrated, like the Nokia example that Mik talked about.

Those teams then can make the decisions about how to choose the work and how to prioritize, because again, it's not always about features. Sometimes it's defects. Sometimes your impact is risk. Those things a lot of times don't get considered, but we know the Nike talk about minimal viable compliance. We know that those are big items that also have to be considered, and they have to match your strategy. Sometimes understand, as the Microsoft example, where you have to take a global strategy and just say, if we're going to get to where we want to go, we really have to invest in debt reduction because we're not going to achieve the business value results that we need.

For technologists, again, I can't overemphasize this: we have to treat your tool network as a product. You have to architect it for speed of delivery of business value, and you have to have that value stream architecture and map set to a product model. Because again, that technology layer is really orthogonal to aligning things across a value stream and how to go fast across a value stream. If I have a system with a million lines of code, I just don't want to start to develop APIs and microservices. I want to develop microservices where I know it's going to provide the products that are applying business value, get them faster, get them speed.

Finally, here's some further reading. What are we looking for? I think this is kind of a new area, so we're looking for help. How are you tracking that? What type of metrics are you producing? What successes have you had, and what challenges still exist? We'd love to hear that. Thank you very much.

Paul Shepheard: Thank you.