Host Intro (Gene Kim)

Every year since 2015, we've had a three-day event called the DevOps Enterprise Forum. It's where we gather about 50 people to write guidance papers based on problems elicited by this community.

And I mentioned this DevOps Enterprise Forum as evidence and inspiration that we can do amazing things virtually, because that's what we had to do in 2020 and 2021. It was something that many of us were skeptical and worried that we couldn't make successful, and yet it was just as fun as any of the ones that we had done in the past.

So one of the things I'm excited about is to have some of these Forum teams share their amazing works, because so many of them have actually influenced our profession. Like the Project to Product work, the book that Dr. Mik Kersten did, actually came from a series of these DevOps Enterprise Forum teams. So did automated governance and so forth.

So next up is the team for measuring value, and here's Jeff Gallimore, Betty Junod, Matt Ring, and Andrew Davis.

Matt Ring

Awesome. Thank you, Gene. I am going to go ahead and get this screen sharing here.

Okay. Just thumbs-up here if you can see the title screen. Excellent. Awesome. All right. Thank you.

So this is most of the co-authors here for this paper to talk about measuring value. Before I introduce the paper, I just wanted to run through real quick the list of co-authors here and give us a chance to introduce. Many of you met me yesterday: Matt Ring, Senior Product and Engineering Coach at John Deere. And we'll just run down the list. Jeff?

Jeff Gallimore

Yep. I'm Jeff Gallimore. I'm old hat at this point. CTIO and co-founder at Excella.

Betty Junod

Betty Junod. I'm Vice President of Product Marketing at VMware Tanzu.

Andrew Davis

And I'm Andrew Davis, the Chief Product Officer at AutoRABIT.

Matt Ring

Awesome. Thank you. And then our fifth co-author, Dwayne Holmes, is unable to join us for today.

Okay, so for the guidance paper, the problem that we set out to solve with this one: we know that in the technology industry, in our community, we've worked to systematize measurement of application and organizational systems. We've done so with flow metrics, quality metrics, ways to measure employee engagement, organizational culture. But value has continued to be this more elusive thing. So what's valuable at my company may not be valuable at yours, and what's valuable to my customers isn't necessarily valuable to yours.

So this paper, what we want to do is really both acknowledge that this ambiguity exists, as well as lean into some of the findings from the product community and some of the ways that they're trying to systematize discovering, creating, and measuring value in their digital products and solutions.

We're not going to go through the entire paper here, but we wanted to tease out - the paper itself starts with the opening vignettes that I think kind of capture some of the essence here. And so we wanted to kind of recreate that for you here.

So with that, the opening starts out with this presentation. Jane is a director of technology for a Fortune 500 financial services company which recently celebrated its 97th anniversary. And on this warm and partly sunny Thursday, Jane is presenting to her company's senior leadership team the results of their multi-year digital transformation journey in adopting Agile and DevOps practices, promoting a culture of learning and experimentation, and shifting from dynamic project teams to stable product teams.

So Jane walks through the outcome-based metrics that her organization has been capturing throughout this transformation. And she notes improvements that the organization has achieved in better metrics: MTTR, availability; sooner metrics: improved flow metrics and lead time; safer metrics, like attack closure rates, security vulnerabilities and production rates going down; and of course, happier metrics, happier employees, like through eNPS.

She gets through all of this thing. And then at the end of the presentation, the CEO asks...

Jeff, you hitting that one, or am I hitting that one?

Jeff Gallimore

So all of this sounds great. It seems like we're delivering more and faster, and our teams do seem more engaged. But really, what value are we getting from this investment?

Matt Ring

Jane's kind of caught off guard here. She just reviewed how her organization has met or exceeded their target goals of this transformation, and how system performance and security posture is all improved, and they're getting changes out the door much sooner. Did they miss that whole section? Was she speaking too fast?

So Jane starts to click back through her presentation slides and then begins reciting the DORA and flow metrics improvements again. The CFO stops her. It was like...

Jeff Gallimore

Jane, this looks fantastic. Congratulations. I'm really grateful that you're able to make these efficiency improvements, because we're really going to be needing to tighten our belt a lot due to the market downturn now. And so it'll be great to cut significant amounts of the current spend from the engineering team. So well done.

Matt Ring

Thanks. Jane struggles to form a response there. In all her preparation, she wasn't prepared to have to defend this position. She was certain that the results were going to speak for themselves. And then, reading the room, the last thing that they were going to want to hear is about more technical jargon being thrown around, like modernizing the mainframe or building a new Kubernetes platform.

So the senior leadership team thanks Jane for presenting, congratulates her on her achievements, and also asks her to come back and report where they can make further budget decisions. So she leaves the meeting feeling dejected, wondering how such a positive transformation story resulted in the threat of decreased funding. And then she looks out the window at the partly cloudy skies and was like, do more with less, I guess.

So that's how our paper opens up here. The question that our paper intends to address, what the leaders of businesses at our company are all wanting to know, is: what value are you providing? What business value and customer value are you providing, and how are you measuring and communicating that out?

Andrew, I'll turn it over to you.

Andrew Davis

Sure. Yeah. In fact, I already saw somebody in the chat thread, Marshall, saying that value is like ice cream. It comes in many flavors. We didn't actually encounter the ice cream analogy when we were working on the paper, but we did recognize early on in our discussions that value is subjective. What's valuable for me is not valuable to Matt and vice versa.

But also value is volatile. It's subject to just constant change, fluctuation, depending on how you're feeling, and it's context specific. So what's valuable in one context is not valuable in another context.

So to be able to measure value - and really, our paper ended up putting a lot of energy into communicating value more than measuring it - first, it's important to understand the organization's direction and context. That's sort of at the highest level, the biggest aggregate understanding that you need to gain to set the context for what is valuable for the organization.

Second, analyze and explain how you contributed - you being the broader you of the team or whoever you're advocating for there - how you contributed to value today. Being able to keep that clarity of the relationship between the current activities and the overarching value in the highest context for the organization.

Third, structuring your next product or initiative outcomes in such a way that you can measure. So basically setting the outcomes and measuring and tracking progress along the way is a big topic of the paper. And then the paper ends with a long section on avoiding common traps. We'll get into that just a little bit here, but definitely point you to the paper for more details.

Betty Junod

So there are two models that the guidance paper leans on, and Matt's going to explain these all in just a minute. The first is the difference between output and outcomes and impact. That's a model that came from Christophe Achouiantz, I think is how we say it. The second is the Product Kata model by Melissa Perri, if anybody's familiar with that framework that she's put forward.

Although I'm familiar with this in words, I do find myself falling into this output-focused mode more rather than focusing on outcomes. So Matt, can you illuminate these topics a bit more for us?

Matt Ring

Yeah, sure. I'll just do a real brief flyover of these. The paper definitely goes into it a little bit more.

The first one, some people may have seen this. Jeff Patton has talked about it. Christophe Achouiantz certainly illustrated it here. But a lot of times, in our technology space, we focus a lot on the outputs. We focus on the backlogs, the software changes, the delivery, the products, the solutions that we build. Those are all really the outputs there.

And then we're asked to be like, well, how are you impacting our company's top and bottom line? And really what this article is talking about, or what this illustration shows, is that those are really more the impacts. Those are the business impacts we're trying to do here. We're trying to generate revenue, improve cost efficiencies, improve our risk posture, those kinds of things.

What this illustration really helps with is connecting the dots between the outputs - the software changes that we do - and those downstream impacts that we're really trying to effect with our organization. It's really around those outcomes, and it's really user outcomes, the customer outcomes.

So we produce software changes. We deploy an API. But what behaviors are we trying to change from that? Are we expecting people to use the API? Are people using the software changes that we did? Are they happy about using them? Is it changing their behaviors? Those are the outcomes that we're really trying to get to.

So it's helping people shift their thinking beyond the software release towards what behaviors or outcomes we were trying to achieve by releasing, by shipping those software changes.

The other one is this Product Kata that was introduced by Melissa Perri. It lends itself, or it leans on top of, the Improvement Kata, the Toyota Kata that many of us are familiar with, but really applies that to the product space.

So digital product is definitely, in terms of the Cynefin framework, in that complex space where there's a lot of emergent practices happening. We have to sense and respond to what is the right thing to build. What are the digital products that solve our customer's problems?

So she applies the same idea here, same thing with the Improvement Kata. First, you need to understand your overall direction: where you're trying to go, what's your company's vision and your organization's vision. And secondly, where are you today? And then from there, what are the incremental steps that you take along the way? And then how do you conduct experiments, or experiment your way towards that next milestone?

So this one really, again, just leans into what a lot of the product folks are doing in terms of iteratively and incrementally and experimentally figuring out what's the right thing to build.

So those are the two models that we lean into in the paper. And then lastly, just to wrap this up, I wanted to turn it over to Jeff to tease a little bit more about some of the traps that we want to avoid here.

Jeff Gallimore

Yeah. So far we've talked about the concept of value, concepts around value. We've talked about the recipe for how to align on it and measure it. But even with doing all that, it's still possible to fall into value traps along the way.

We've described seven of what we think are the most common value traps, so you can be on the lookout for them in your own organization. Each trap that's described in the paper has a description. It has some examples of what that trap looks like in practice. And it also has what you can do to either avoid it or get yourself out of it if you do find yourself having fallen into that particular trap.

We've got seven of them in the paper listed here. We can't go into all the details for all of them right now. Some of them might resonate just from the short title that we've provided.

I do want to highlight one of those value traps, which is value theater. This is probably my favorite. It's characterized by lots of documentation, like business cases, lots of process, and maybe, as Paul Gaffney might say, lots of pageantry. If you want to know more about that, look at his talk from the summit in Vegas a couple months ago.

But all of this activity around value is not necessarily correlated with the delivery of actual value. So that may sound familiar to some of you. If you want to learn more about these traps, of course, check them out in the paper.

And I think we're ready for some Q&A, maybe.

Q&A

Matt Ring: Yep. I think I'll lean on one of my other co-authors to help if I haven't been following the Slack.

Betty Junod: We've got a lot of comments, but not exactly questions yet. So if there's any questions, folks, please pop them in. I did ask folks to share if anyone's ever felt that opening vignette. Do you see yourself in the vignette? Have you experienced this? Share some good zinger responses or one-liners you may have heard while you're presenting the outputs of a great project or what have you.

Oh, where's the paper available?

Matt Ring: On IT Revolution?

Betty Junod: First and foremost, I know I posted right before the session. There's a post right before we started our session, somewhere around 9:43 or 9:42, I did put a link to it directly on the IT Revolution website. So feel free to check that out.

Yeah, Gene popped a question. Let's go. Why are technology leaders so prone to falling into the efficiency value trap? Who's got an opinion on that one?

Jeff Gallimore: I'll start off. I'll share one opinion. I'd like to think of myself as a technology leader, and this is something that I've struggled with, had problems with. I think there are probably a couple of reasons.

One is, at least for me anyway, my primary affinity has been to the technology, what Andre Martin would call the craft. I relate very well, and there's a lot of intrinsic understanding about the benefits and value of the craft and the technology and such, that not everybody else is wired the same way.

And so I have to be cognizant, one, that other people are wired differently, and number two, have the ability to actually do that translation. I didn't learn that skill until much, much later in my career, to be able to actually put things in dollars-and-cents terms in many cases, rather than the ones and the zeros that resonated and I related to so well.

Andrew Davis: I'll chime in another little tidbit on this. One of the most universally recognized forms of value is money, dollars and cents. As Jeff put forward, it's not the only value that's widely recognized across the company. But if your job is sales, where your success in your job involves bringing more dollars and cents into the organization, it's very easy for people across the organization to understand and appreciate the benefit of what you're doing.

But if you're in a more esoteric job - you're maintaining servers and these kinds of things - the value of what you're doing is not as easy to translate across the organization, not as readily appreciated. So you actually have more work to do, because the value that you're bringing actually requires a translation. You have to explain to others and explain up the food chain the benefit that you're bringing in a way that is not required if you're distributing dollars or distributing ice cream around the organization, where everybody appreciates it naturally.

Betty Junod: And in the paper, we have a great example that ties that sales and business organizational goal value to the organization down to translating that into the technology team outcomes that they need. So I think that's a great example of what both Jeff and Andrew are talking about, of translating and connecting those components. Ultimately, the organization requires all the functions to operate together. So another plug to check out the paper.

There is a question here. Any tips for...

Matt Ring: Hey, Betty, I'm sorry. I'm going to put my hat on of trying to keep the trains running on time.

Betty Junod: Yeah, I was going to go to one more question, but we do on Slack.

Matt Ring: I'm looking at all of them. I think we can promise to get into Slack and answer them through Slack.

Betty Junod: Yeah, because some of these are really good.

Matt Ring: Because now we're like two minutes, right?

Betty Junod: I know. Yeah.

Matt Ring: That's right. Go from running early to running over.

Betty Junod: Well, that's okay. Fantastic. That's par for the course for us. You want to wrap this up, Jeff?

Jeff Gallimore: Yeah, let me wrap up. As we wrap up in this last 90 seconds or so, we wanted to leave you with three key takeaways about this concept of measuring value.

The first, we've hit on a lot: value is subjective and volatile and context dependent. It means different things to different people. And what makes matters even harder sometimes is that meaning changes over time. So we have to keep that in mind.

Second, output should lead to outcome, should lead to impact. Make sure that we have a line of sight through all of these things for the work that we're doing. So if we're working on something, make sure that we understand how it connects to an outcome and how that connects to an impact.

And then third is to iterate. We want to revisit value early and often with the others that we're working with or that we're delivering to along the way. And we also want to revisit how we're creating it so we get better at defining it, get better at identifying it, and ultimately get better at delivering it.

And then one final thought that we'll leave you with. There was a reason that Matt, Andrew, Betty, and I wanted to work on this paper. It's because we saw an opportunity to improve what we saw as a vital skill for technologists and technology leaders, to help them have more impact on their organizations and advance in their careers.

So all of us, and all of you - just speaking for myself - I've not always done this well. And I've fallen into many of the traps that we have in the paper. So improving your ability to communicate in value-oriented terms can unblock your career and unlock opportunities for you and your organization to win. And if we can help just one person do that, this will have all been worth it.

Matt Ring: Yeah. Fantastic. That's it. Fantastic. Back to you.

Gene Kim: Thank you, teams. And I look forward to having all those questions answered in the replies. Thank you very much.