Host Intro (Gene Kim)

[00:00:04] I've been an admirer of the work of Shannon [00:00:07] Lietz for nearly a decade because she has helped pioneer and [00:00:10] codify so much of the work has become known as DevSecOps. [00:00:14] What I've always loved about her work is how she's [00:00:17] been continually seeking how to better bridge the world of information security [00:00:20] and what actually needs to be secured and who [00:00:23] needs to be doing the securing. [00:00:26] So what is some evidence that people who matter actually value her work [00:00:29] in 2014? She won the coveted Scott [00:00:32] Cook award for innovation for the work that [00:00:35] she did building the cloud security program at Intuit, which is [00:00:38] only one of the many amazing and pioneering things [00:00:41] she did there. [00:00:42] So earlier this year I was so delighted when I learned [00:00:45] that she became VP of security at Adobe, which is [00:00:48] the fifth largest software company by market capitalization. [00:00:51] So with the term DevSecOps being used [00:00:54] so widely in our industry already. You think that her work is [00:00:57] almost done, right? [00:00:59] But after she shared with me what her aspirations truly are and [00:01:02] what she's been working on right now. It may be realized just [00:01:05] how far we need to go to truly create the ideal interaction between [00:01:08] information security and the rest of the organization. So [00:01:11] I was so happy that she was willing to share the [00:01:14] work that she is doing right now with this community and I trust that [00:01:17] you will find it as startling and as exciting as I [00:01:20] did. Here's Shannon.

Shannon Lietz

[00:01:23] Thanks Gene. I'm really excited to be [00:01:26] here at DevOps Enterprise and been [00:01:29] a lifelong dream a super [00:01:32] super excited. I am here to [00:01:35] share a little bit about my recent research and current [00:01:38] research as you all may [00:01:41] know I have been in this industry for a long [00:01:44] long time and I'm gonna share with you today a [00:01:47] little bit about RAVE metrics. So let's arrive and [00:01:50] we're gonna measure software value and trust and talk a little bit about the [00:01:53] overview of some of the work that I've been doing to [00:01:56] pull it together and then I have an ask for you [00:01:59] at the end. So stay tuned. [00:02:02] So a little bit about me, like I said, I've been [00:02:05] in this industry for a really really long time and that number on that. I [00:02:08] really does mean a lot. You can see a little gray hairs here [00:02:11] excited to be at a virtual discussion because [00:02:14] I really find these to be rewarding and love [00:02:17] to get connected to folks in this community. I've been [00:02:20] in the industry so long that honestly I started out [00:02:23] as a Dev so as much as I'm a quote security [00:02:26] professional I would say I've just [00:02:29] really become a DevSecOps over the course of my career. [00:02:32] I am very involved in [00:02:35] STEM and I have a couple of little girls [00:02:38] so they like to do ballet. I create [00:02:41] comic strips. I'm into kickboxing. I started [00:02:44] the DevSecOps stuff that's out there [00:02:47] and have been working in pioneering a [00:02:50] lot of different things. I work at Adobe. I'm a [00:02:53] VP of security is Gene had mentioned and I'm [00:02:56] just gonna share a little bit about what's going on in Adobe. [00:03:00] So I recently joined I've been here about [00:03:03] I don't know a little over a year and we have [00:03:07] some amazing products. I'm really really [00:03:10] excited about what we do as a software [00:03:13] company over 100 products. We have 4500 developers [00:03:16] Plus. [00:03:18] And we have about 15 billion in [00:03:21] annual revenue. So pretty big large company [00:03:24] essentially investing [00:03:27] in the software of all of our future. I remember [00:03:30] when I first got to Adobe again [00:03:33] another lifelong dream. I started my career [00:03:36] using Adobe products. I remember like some of the first stuff [00:03:39] I did was create a logo for a company that I created [00:03:42] back in the day. [00:03:44] And I really am a [00:03:47] big fan and so I'm so excited to really be here [00:03:50] and start this journey. [00:03:52] So what is this really mean? I do at [00:03:55] Adobe. So I am the VP [00:03:58] of Adobe Security I run [00:04:01] a group that essentially is [00:04:04] focused on two things product security and [00:04:07] adversary emulation and [00:04:10] understanding how you build resilient [00:04:13] products. So this notion of shifting left [00:04:16] as well as how do you really learn [00:04:19] from adversaries for investment purposes to [00:04:22] make it so that we can learn from [00:04:25] those adversaries and pull together that total story of [00:04:28] security being added into [00:04:31] software. [00:04:33] So maybe you've heard about DevSecOps. I [00:04:36] always ask people have you heard of it? And what I [00:04:39] find over the years is it went from a conversation [00:04:42] at a nice soccer presentation? You [00:04:45] may not know what that is. But basically it's a group out [00:04:48] there that does information security and it's a [00:04:51] practitioners group and I remember my first slides [00:04:54] I actually had like a tortoise and a hair and [00:04:57] I was explaining how the security industry [00:05:00] needed to evolve and so as much as you know [00:05:03] folks here are really involved in DevOps. I believe that [00:05:06] a lot of us are really trying to bring security into the [00:05:09] forefront of software and make it much easier as well. [00:05:12] So I'm sorry about DevSecOps because [00:05:15] the name does have a whole bunch of folks in DevOps [00:05:18] kind of worried about that. But at the same time, I'm also not [00:05:21] sorry because I am finding that over the [00:05:24] course of the last 10 years software has been increasingly [00:05:27] getting better at doing security before it [00:05:30] gets released. And that means that we're checking our software [00:05:33] and making sure that it's as resilient as [00:05:36] possible. So that's my plug for security. I [00:05:39] actually have an interesting take on [00:05:42] what I think has been happening in the security industry [00:05:45] and DevOps and all these things. And so this talk [00:05:48] is really gonna be focused on more about measurement and [00:05:51] some of the things I've learned but I'm gonna start you off [00:05:54] with sort of the journey towards how did I get to this point with this software [00:05:57] metrics conversation and really if [00:06:00] you look back in 2014, the essence [00:06:03] of all of this came from sitting down and [00:06:06] realizing like we were security folks and the developers [00:06:09] were basically moving around some of those [00:06:12] Clunky to you know clipboards that we [00:06:15] had and so we finally sat down and wrote down [00:06:18] really what we needed to do to change. It was like we [00:06:21] need to lean in instead of say no, I think at one [00:06:24] point I got a a t-shirt sent to me that said the [00:06:27] word no on it with a period it was like, you know, [00:06:30] that was the brand that was being sent around was security [00:06:33] was really all about it had to be our way [00:06:36] or the highway and in software development that [00:06:39] that can be really a pain if you will being a [00:06:42] developer myself, I would say I know what [00:06:45] it's like to be sitting there trying to make something work and then being told [00:06:48] oh, by the way, you have to lay on all these different things to be [00:06:51] able to make your software secure capable available [00:06:54] at the same time that you're just trying to [00:06:57] figure out how to make it work for somebody and make it useful. So in [00:07:00] 2014, I would say, you know, we kind [00:07:03] of put this down and it was a little [00:07:06] bit of an experiment and at the same time it was pretty good experience. [00:07:09] I think now what was also really interesting. [00:07:12] 2014 we knew we needed to get data and security science [00:07:15] was actually pretty expensive back then to try and [00:07:18] do what people are doing these days with security capabilities [00:07:21] and security science. If you will big data [00:07:24] was pretty expensive. It's actually [00:07:27] gotten way cheaper. It's become more commoditized over [00:07:30] the years and then this notion of business-driven security [00:07:33] score is really getting to the point where we weren't just saying here's [00:07:36] a list of things you have to do thousands of things you have to do but really [00:07:39] getting to the point where how can we summarize it make it much more [00:07:42] simple? And so I would say, you know [00:07:45] what came out of that 2014 realization has been [00:07:48] many many years, but one of the big [00:07:51] most critical things is we've been putting all of the [00:07:54] security capabilities into the CI/CD program. We've been actually, you [00:07:57] know, building things into the software pipeline and [00:08:00] at the end of the day, I think it's been measurement that's [00:08:03] been the most critical and most necessary element [00:08:06] of pulling people together. [00:08:09] So why do I say that? And by [00:08:12] the way, I find it to be extremely challenging to [00:08:15] talk about security measurement and how you're [00:08:18] going to actually be part of the conversation of building software. So, [00:08:21] you know when it comes down to [00:08:24] it, the reason behind this is [00:08:27] metrics really create a little [00:08:30] bit of that guiding light, that North Star measure what [00:08:34] matters the John Doerr book. How do we [00:08:37] actually start to determine what we're going to do and [00:08:40] how we're going to solve problems? And so at the forefront [00:08:43] for a security professional we're all trying to figure out how do we actually make [00:08:46] this thing? That's really important getting rid of adversaries in [00:08:49] our software taking away their advantage [00:08:52] taking away the attack surface. How do we really make that come [00:08:55] to the forefront? The idea behind this is like [00:08:58] we could go focus on what we can measure, [00:09:01] you know, the idea [00:09:04] behind metrics is really interesting and unique [00:09:07] and what we can measure is often. [00:09:09] What we should measure it's not necessarily what [00:09:12] matters most but it's got numbers associated with [00:09:15] it and they're actually something that's relevant so it can help the conversation. [00:09:18] And what I find is that commonly those [00:09:21] metrics are actually around value creation. [00:09:24] So there's been a lot of amazing work out there. [00:09:27] So don't take any of this as being pejorative towards the [00:09:30] work that's been done out there because it actually isn't. In fact, I'm a big fan of [00:09:33] the work that's been done out there. I just have a different take on how [00:09:36] I think it needs to come together to truly take us to that [00:09:39] next level of how do we actually all work together to inspire [00:09:42] a community of software development that I [00:09:45] think is at the forefront of what people thought they were putting together [00:09:48] when we we built out things like, you know [00:09:51] software as an industry. [00:09:54] So we have a ton of stuff out there the DORA metrics [00:09:57] the QSM metrics this SRE metrics [00:10:00] really all centered around this notion of developer productivity [00:10:03] and some would say as developers. The last [00:10:06] thing I want is for somebody to measure out [00:10:09] how I'm doing in terms of productivity and then others [00:10:12] are really keen on it. Like we really need to get to [00:10:15] the point where we're measuring out our Wellness how [00:10:18] we're actually doing what we're doing. Are we getting the [00:10:21] help we need to be able to build software more complete [00:10:24] if you if you will and over the years. I've [00:10:27] also heard that a developer has to do everything they [00:10:30] have to be able to make business decisions. They have to be able to make security decisions. [00:10:33] They have to be able to make all these decisions and they have to be able to [00:10:36] write the software at the same time and and maybe that context [00:10:39] is something we need to change is that it's not [00:10:42] necessarily that they have to do all these things. It's that [00:10:45] as a community, we all have to [00:10:48] do enough software development ourselves. So this [00:10:51] notion of maybe being you know security [00:10:54] Um part of the security Community but creating simplified [00:10:57] capabilities that could be included in software is [00:11:00] something that is part of the paved road path of where a developer [00:11:03] can be productive and solve [00:11:06] for customer problems. So, you know [00:11:09] long while back I started out trying to [00:11:12] map out. What did I mean by DevSecOps? And and what [00:11:15] did we mean as a community about what it was going to do to [00:11:18] fit into the process and at the time there's a little bit of work [00:11:21] done out there around software development life cycles and [00:11:24] security folks were showing up and trying to leverage their [00:11:27] tools during whatever part of the process. So [00:11:30] hey, I need access to GitHub to be able to look at your code and [00:11:33] and then come talk to you about how that might be working and ultimately [00:11:36] value and [00:11:39] trust was something that I think came to [00:11:43] the forefront for me is it wasn't that we just needed to do [00:11:46] this notion of measuring value creation, [00:11:49] but trust and so, you know, I love [00:11:52] this picture because it's really about [00:11:54] if you have value and trust being measured this notion [00:11:57] of software being delightful is something [00:12:00] that I think comes to the forefront. [00:12:02] So I I took this picture to [00:12:05] heart. I actually am a big fan of some of [00:12:08] the products we have and one of the ones that [00:12:11] I spent some time on is Stock. I I do [00:12:14] tend to love to try and visualize in their [00:12:17] rate how I'm thinking and feeling about the journey [00:12:20] that I'm funding with my research and my [00:12:23] time and so I did learn a [00:12:26] lot at Intuit previously where we talked [00:12:29] about things like design for delight and [00:12:32] having the principles of designing for delight and really that [00:12:35] customer understanding of software that Delight [00:12:38] that's being created is I think the driving [00:12:41] force that we're all trying to measure but by [00:12:44] the way measuring that is hard because measuring trust [00:12:47] is hard and measuring security is hard measuring all [00:12:50] these things that are a little bit more I think [00:12:53] not necessarily part of just taking code and [00:12:56] putting it down and putting it on to a system and so, you know [00:12:59] today I'm going to talk to you a little bit about how we actually bring those all together. [00:13:02] And another one had that has recently inspired me has been [00:13:05] Scott Belsky who works at Adobe. And [00:13:08] so if you take Scott Cook and Scott Belsky, I [00:13:11] have definitely been inspired by both of these gentlemen [00:13:14] to really think past the notion of [00:13:17] what we do day to day and more about how do we actually take [00:13:20] this as a community a journey together and [00:13:23] how do we really bring this complexity [00:13:26] of software to the forefront where we could solve human [00:13:29] problems and simplify that for our customers? [00:13:33] So, you know what? What does it all result into? Well, [00:13:36] if you hunt down both of the inspirational [00:13:39] folks and you look at the basis of [00:13:42] their work, some of it is really about what makes someone a [00:13:45] Net Promoter. I mean ultimately if you think about it, [00:13:48] what products do you leverage which ones make you happy? What [00:13:51] are you buying out there? Like what products do [00:13:54] you have sitting in front of you? Those are things that if [00:13:57] they were to go away, I think somebody mentioned to [00:14:00] me that superhuman said that if their stuff was to go away they [00:14:03] actually measure the notion of would you be disappointed and [00:14:06] I think it's a really good measure because it helps [00:14:09] to understand a little bit about that trust algorithm [00:14:12] that we're talking about here. [00:14:15] So, you know what makes someone rave about your [00:14:18] products to their friends meaning what makes them [00:14:21] put their credibility on the line and really talk [00:14:24] about your software in terms of if [00:14:27] the most amazing thing it does this it does that and [00:14:30] oh by the way, it's always up. It's always online. That's how [00:14:33] we got to the cloud in the first places. We were just unhappy with [00:14:36] like hey, this system is in scaling. There's error rates [00:14:39] associated with it. And as we all research really [00:14:42] what the expectation is of the customer, we invent all [00:14:45] these amazing capabilities that make it so we can get to [00:14:48] that potential of what people dream about in terms [00:14:51] of using software products. [00:14:53] I think that's the most amazing clearest Vision I've ever [00:14:56] had about why software is something that speaks to my heart. [00:14:59] And so to me I wanted to sit [00:15:02] down and really say, all right. What is that algorithm that makes me rave [00:15:05] and all of a sudden it was like I put two and two together. I'm like, well, maybe [00:15:08] there's something here. That'll make me remember all [00:15:11] these metrics and have a harness for being able to do so, [00:15:14] so I'm going to talk to you a little bit about the fourth [00:15:17] things that I think are important for this trust algorithm and [00:15:20] one of them is, you know, as a customer I can rely [00:15:23] on your product because it's resilient and so resilience [00:15:26] is an interesting part of this journey. I [00:15:29] picked out that word very specifically I [00:15:32] didn't talk about availability. I didn't talk about some of [00:15:35] the reliability secureability as being that forefront [00:15:38] but for the for the customer really having resilient software that [00:15:41] rugged software if you will is something that [00:15:44] I think they deeply care about because it is part [00:15:47] of that algorithm of why they come back for more [00:15:50] Another one is you know, why do I want to [00:15:53] do things with your software? So it having the ability to say I [00:15:56] want to adopt your product because it helps me I think [00:15:59] software has to help you for you to want to use [00:16:02] it and to Value it so that utilization of [00:16:05] software is a really critical element. And [00:16:08] then another one is your product has high-velocity feature [00:16:11] evolution meaning it's changing and evolving [00:16:14] and it's taking my feedback and I'm getting more [00:16:17] out of it over time. So you're earning my [00:16:20] trust continuously from this part of [00:16:23] the algorithm. And then the last one is like your product has [00:16:26] a low error rate. I'm not disturbed by constant interrupts. [00:16:29] I I when I click on something it's [00:16:32] not going to an error page all the time. I'm actually [00:16:35] seeing that you put some time and effort into the experience and [00:16:38] that your error rate is actually pretty low. So [00:16:41] I call it RAVE I bring [00:16:44] all these measures together. I like to build upon [00:16:47] what other people are doing not just reinvent the wheel. [00:16:50] Coined this phrase because I think that branding [00:16:53] brings something together and really focusing [00:16:56] on the most important elements of what we're all trying [00:16:59] to do is how we're going to get to that point where [00:17:02] we're all talking about it in a way that makes it forefront and [00:17:05] we're gonna solve that big problem, which is humans have [00:17:08] better software to solve problems that we all need to solve. [00:17:12] So what does it all mean? So RAVE is really resilience, [00:17:15] adoption, velocity, and errors. Now, I'm not going to predict every [00:17:18] single metric that's going to be at the [00:17:21] forefront but I am gonna try and get us to start talking about and [00:17:24] debating which metrics matter most in the community. So [00:17:27] using the RAVE framework what I of what it really [00:17:30] trying to get out there is that we're bringing not [00:17:33] only developer productivity to the forefront but [00:17:36] we're also adding in this notion of customer delight [00:17:39] and so let me talk to [00:17:42] you well quick about the resilience that's out [00:17:45] there, you know software's design gracefully and it helps [00:17:49] us to establish an operation with thresholds. [00:17:52] If you think about it resilience is really [00:17:55] about saying what you would like to achieve and then [00:17:58] building your stuff so that it eventually achieves that [00:18:01] outcome. I believe thresholds are [00:18:04] critical in establishing good resilience. And the way you [00:18:07] do that is really to look at the metrics associated with it that [00:18:10] are out there. So are you willing [00:18:12] to put down what your service-level objectives are. Do you [00:18:15] have availability with five nines? I think that that's a threshold. [00:18:18] It's like what is your threshold if your software's down [00:18:21] 1% of the time, did you meet your threshold? Is [00:18:24] that the risk you'd like to take for your company? I've got this [00:18:27] notion of secureability and many people have heard me talk about it throughout the [00:18:30] years and if you don't know what it is definitely look up secureability. [00:18:33] Look at my name and and go take a look at what I'm driving and [00:18:36] in terms of trying to make security measurement come [00:18:39] to the forefront because I do think it's critical and there I [00:18:42] think five nines would be probably pretty drastic because [00:18:45] there are things we have to do to be able to balance security against [00:18:48] all the other resilience metrics. So I [00:18:51] do think that you could say hey is it five nines or one in many [00:18:54] cases? If you look across the industry adversaries are becoming [00:18:57] much more aggressive and holding them back [00:19:00] and creating great software is about making sure the [00:19:03] risk is is dealt with in the right way. These These are [00:19:06] really important if you look DORA is mentioned in [00:19:09] here, so I'm giving you an idea of [00:19:12] Tricks that you could put into your resilience area. I'm not [00:19:15] telling you which one to go chase down. But I do know that [00:19:18] when you focus on the most critical things of the resilience problem, [00:19:21] where are you having the biggest challenges then you [00:19:24] tend to put Priority where it matters most and [00:19:27] so I would say the balancing of the abilities. We've all talked about that [00:19:30] in the past as well. The balancing of the -ilities is really [00:19:33] about these thresholds and and how you actually focus on [00:19:36] the most important parts of what you're developing and software. [00:19:40] Adoption, you know software is able to be actively adopted [00:19:43] and fully utilized. It's [00:19:46] one of the biggest things we all want in the industry [00:19:49] is not only am I building it but you're using it and you're [00:19:52] getting a lot of value out of it is the way I like to talk about it. So [00:19:55] some of this is coming through with things like product-led growth and [00:19:58] I think monthly active users that now that's [00:20:01] out there that's coming to the forefront top 99 customer [00:20:04] set, you know [00:20:07] product-led growth is at the forefront of really helping us [00:20:10] to think about software is only useful when it's [00:20:13] being used and when it's being adopted and when people [00:20:17] are delighted they really do talk about and rave [00:20:20] about your software. So I do think that this is part of that algorithm as well [00:20:23] velocity, you know software achieves [00:20:26] the rate of change that matches the expectations of [00:20:29] its Community. I think when we talk about developer productivity, [00:20:32] there's a really big sweet spot [00:20:35] here in velocity that's helping us to understand that [00:20:38] customers truly depend [00:20:40] on us to evolve and take their feedback and change and [00:20:43] do what's necessary to continue to create value. It's [00:20:46] not just that they adopted your software. It's that they actually will [00:20:49] stay because you continually change and so here [00:20:52] you can see the SPACE metrics the product-led growth metrics and [00:20:55] DORA metrics are pretty forefront. The one that you choose or [00:20:58] the two that you choose or the 10 that you choose. It's really up to [00:21:01] you how you want to manage the velocity part of [00:21:04] this algorithm for trust, but I do think that it's [00:21:07] a critical element and that's why like I said, I'm a big fan so it's [00:21:10] not a it's not an OR in my mind. It's an AND in a [00:21:13] lot of ways. I actually believe that there's so many great metrics [00:21:16] out there that can really help all of us to have the [00:21:19] right debate and learn from each other in this space and then [00:21:22] finally errors, you know software is successful and achieving [00:21:25] quality targets when it has a low number of introduced errors. [00:21:28] No one wants to use buggy software. [00:21:31] It's one of the reasons why we all have things like, you know, [00:21:34] bug tracking software out there because errors actually [00:21:37] erode the experience and so some of the great [00:21:40] metric [00:21:40] Are out there like error rates. I love the one error budgets. [00:21:43] In fact, there was somebody talking about security budgets at one point following the [00:21:46] SRE metrics that are out there, you know defect leakage [00:21:49] is important and test-case pass rates. [00:21:52] And so you're going to see I think if you were to [00:21:55] look out there that there's some amazing metrics in each one of these categories that [00:21:58] are part of RAVE. Okay. So what does [00:22:02] this really all look like in reality? And [00:22:05] why does it matter to me as a security [00:22:08] professional and the rest of us as a security community? [00:22:12] Well, the truth is is that security needs to [00:22:15] be part of the conversation and part of these critical metrics. And [00:22:18] as you saw I said something about secureability [00:22:21] in resilience. Well, it was through all [00:22:24] this inspiration of understanding what was truly out there [00:22:27] in terms of the total population of metrics that changed [00:22:30] my mind about it just being secureability. That was my [00:22:33] driver and in fact [00:22:35] Security has to be part of this conversation in [00:22:38] a meaningful way. So if you [00:22:41] think about it, that means secureability part of resilience here means [00:22:44] that we have to have our software be malware-resistant. [00:22:47] It means we have to have it adversary-resistant. We [00:22:50] need to be able to focus on things like adoption and [00:22:53] realize that in some cases the adoption [00:22:56] may also show adversaries that are [00:22:59] trying to adopt products that we don't necessarily want to be [00:23:02] attracting velocity. We really do need [00:23:05] to make it so that developers have the tools they need [00:23:08] the capabilities. They need built into the CI/CD [00:23:11] and part of that conversation is we [00:23:14] should be focusing the security professionals on the velocity that [00:23:17] developer velocity as part of the journey as well. [00:23:20] And finally with error rate. I know [00:23:23] a lot of security professionals spend their time on errors [00:23:26] and defects and we call them vulnerabilities and [00:23:29] whatnot. But truly we all care about the same things [00:23:32] as a community whether we're security or [00:23:35] DevOps and this is a unifying force in the [00:23:38] absence of utilizing this type of framework for building [00:23:41] trust. I think we're really going to be missing out [00:23:44] for many many decades to come and I do think this [00:23:47] does solve a couple of things which is bringing security to the conversation [00:23:50] without it. Essentially, we'll still be [00:23:53] silos will still be having challenges and I [00:23:56] think that's where we have to change and like I said, I'm inspired [00:23:59] to get to this journey as a security professional because [00:24:02] really I learned about creating customer [00:24:05] delight and I think security is part of that conversation of [00:24:08] creating customer Delight. So for me, [00:24:11] it's all about creating the right [00:24:14] experience. Now, Gene asked me [00:24:17] if this slide in and I'm really excited about it. So the help I'm [00:24:20] looking for in this is I really want to have you [00:24:23] help me take this RAVE metric survey. There's [00:24:26] gonna be some questions in there about Dev and [00:24:29] security and I do think that that data would be at the forefront of [00:24:32] helping us to bring this RAVE plan. [00:24:35] together and create trust in software [00:24:38] Also, if you could reach out to me if you're interested in debating [00:24:41] with me about metrics the good ones the bad ones [00:24:44] of why and want to do a podcast with me. Please reach [00:24:47] out. I'm totally interested. I'm gonna be doing this in 2023 and [00:24:50] something new to look forward to with [00:24:53] that all said thank you so much for your time. And [00:24:56] let's get back to DevOps Enterprise. Let's [00:24:59] rave as a community and bring measurement to [00:25:02] software value and trust.