Denali Lumma

Hello everyone. Good morning.

Thank you so much for joining me today. I'm excited to be here with all of you like-minded folks.

My name is Denali Lumma, and I am doing a couple of different things. I have a consultancy where I advise various companies and investors. I also work at a company called Modular, which is focused on AI/ML infrastructure, and I lead the software development lifecycle group there. I'm also part of the DORA community, DevOps Guide.

I've had some amazing opportunities over the years to work at different technology companies, early stage all the way to IPO and large publicly traded firms. I've done a lot of work in many things related to software development lifecycle: test automation, build automation, release automation, CI/CD, DevOps, infrastructure as code, and all of that great stuff.

Today I want to share some findings of some research I've been doing recently that I think is really compelling and really interesting. I have some initial early findings that I'd love to tell you about.

Basically, what I'm seeing as I look at open source projects and do comparisons and analytics is that it is actually possible to forecast the future value of a codebase, or the company that produces that codebase, with cycle time variance. As part of the DORA metrics, the DORA community, we're very familiar with cycle time, lead time. There's another aspect to that number that I found to be highly correlated with positive business outcomes.

Here's the agenda for today. I'd like to go over the concept of technical debt, the term, what we mean when we say it. Then I'd like to cover my definition of the inverse of technical debt, which I'm calling technical capital. I'd like to explore some of the ways that value forecasting is done in the world of finance, and look at some of the basic constructs that we use for forecasting time series data, some mathematical concepts. Then I'd like to actually go through some open source projects and show you some of the results.

The tech debt metaphor. We've heard the term, I think, many times today, yesterday. It's used frequently. When we say tech debt, we are describing, essentially, what we think is the future value of the codebase. A lot of people think of it roughly as like credit card debt that is compounding. We're basically forecasting how much, you know, we have a codebase today, but how much will it be worth next week, or next month, or next year, with our current practices?

The general idea is this common-sense thing: an ounce of prevention is worth a pound of cure. But tech debt isn't really like credit card debt. I think the metaphor erodes pretty quickly when we're dealing with software. The actual problem is really about complexity and understanding. That's kind of the primary issue.

From my perspective, when we say tech debt, what we actually mean is: how hard or easy is it to comprehend the system? How understandable is the system? So actually, tech debt is more like losing your mind, as opposed to defaulting on your credit card.

Tech debt is working with a system that cannot be understood, or is very hard to understand. From my experience, and I've done a lot of work with lots of different companies doing sort of digital transformations, corny phrase for what I do, it's pretty easy to measure tech debt, actually. It's really just: how easily or hard is it to understand the code? That can be measured with cycle time. Even change failure percentage, MTTR, all of that actually feeds back into cycle time.

So if there's one metric to understand tech debt, I think cycle time is the metric.

But I actually think more interesting than tech debt is this idea of: how much future value do we think this codebase will have? I like to refer to this as technical capital.

In the world of finance, there's lots of metrics and lots of frameworks and ways that future value can be forecasted or understood for a company. I think some of those can be applied also to technology.

Forecasting. What do we do when we forecast? We oftentimes work with time series data. Here's an example of time series data from the world of finance. Basically, there are two attributes. There's a date attribute and then there's a value attribute, so in this case, the closing price for a given stock.

Here's an example of some time series data from the software development lifecycle that we probably are more familiar with. We have, again, the date attribute, and then we have the cycle time. Cycle time can be generalized pretty well with something like the 85th percentile or the 90th percentile as a value for that day.

A few other mathematical concepts, just to refresh everyone's memory, or to introduce these concepts if they're new for you.

When we talk about data and understanding data and datasets, we have a few things that we look at. One is the mean, and the mean is value-based. It's also the average. Basically, this is something intuitively you'll probably remember. You take the sum of all of the values, and then you divide that by the number of values to get the average. That's the mean.

There's also something called the median, which is different. It's position-based, as opposed to value-based. The median represents the middle value in a dataset that is ordered in ascending order. It's basically the value that separates the lower half from the higher half.

Percentile. What does percentile mean? Percentile is kind of like a more generic term for the median. The median is the 50th percentile, but we can also talk about other percentiles like the 25th or the 75th.

As we're looking at time series data, as we're looking at data, we're trying to understand the shape of the data and what it means. We often consider this idea of the Gaussian distribution, which the common term is the bell curve. It's basically a way for us to understand how the data relates to itself, how common or uncommon something is in a group.

Some key properties with a Gaussian distribution: we have the mean, which is again that average value-based, the center of the bell curve. Then we have something called the standard deviation, which is describing the spread or the distribution of the data.

Sometimes it's helpful to look at an example dataset. So we take a simple example dataset: 1, 1, 1, 1, 2, 10, 17.

What is the average of this? Well, we sum it up, we divide by the number, and we get approximately the average, or the mean, of this dataset is five.

What is the median? In this case, we go into the middle of the ascending ordered list. We take that value, which here is one.

What is the 75th percentile? Calculate the 75th position in the list. We go to that position, and we see that it's number six. So the value for the 75th percentile is 10.

All of the previous math concepts, hopefully, were probably just refreshers. But what is the standard deviation? Well, the standard deviation is not really something that we can just calculate in our heads. Unlike the other stuff, this is actually a little bit more complicated. There are steps that you follow to do this, but you can't really do this without a piece of paper or a calculator.

Essentially, the standard deviation is: find the mean of the dataset, subtract the mean from each data point to find the deviation, square each of those deviations, find the mean of the squared deviation, and then take the square root of the mean of the squared deviations. It's a little bit complicated, but those are the steps that you follow to get the standard deviation.

Step one, step two. I'm not going to spend a lot of time on this, but you can review it if you want. Step three, step four, step five.

In summary, for this dataset, we have these different attributes and different ways of describing the data and how it relates to itself. The mean is five, the median is one, the 75th percentile is 10, and the standard deviation is six.

With this data, we can do a lot of things, especially, as I said, with finance and other applications to forecast into the future.

There's something called a T-score, which helps us understand how a particular data point relates to a distribution. Here's the formula for that. There's something called a Z-score, which is almost exactly the same, but it assumes access to the entire dataset.

With all of these data, we can look at things that help with forecasting. We can look at rolling averages. We can look at a trend line. We can look at the relationship between the T-score and the Z-score, which is typically how anomaly detection is done, for example.

There's a whole field of math associated with predicting data, especially data that has some random variation, called Ito calculus. If you're interested, I recommend diving into that.

There's also someone within the financial field who is considered to be a notable thinker. His name is Harry Markowitz. He really had this idea around how to invest money effectively and how to forecast so that you have the highest return on investment.

His first big idea was to invest in the things that had the highest mean and the lowest variance. So he landed upon this important combination of high mean, low variance. He actually built upon that work, and it became what's known today as modern portfolio theory. He actually went even one step further and said reducing risk and increasing return should be done with further diversification. Again, the focus is on the reduction of variation. That was his big idea.

Reduction of variation is a common theme across fields. It shows up all over the place: manufacturing, quality control, healthcare, finance, obviously, construction, energy efficiency. It's a very, very important concept that has been developed in these different distinct domains, and it generally confers value.

A lot of people have actually been thinking about this problem for a long time. There are some original thinkers historically that spent a lot of time studying this and developing these insights, which I feel like, as modern people, we often reinvent or rediscover wisdom that has already been known for many, many years. Frederick Taylor, Walter Shewhart, of course, W. Edwards Deming, and more.

Essentially, from my perspective, as I study all of these domains and as I do work with people in the field looking at how to create value with technology and with codebase, there is this universal theme that has become very clear to me, which is that probably the most important aspect of creating value is actually reducing variation.

From my experience, it's actually quite easy to measure both technical debt and technical capital. I think technical debt is represented very well with cycle time, and technical capital is represented very well with cycle time variance.

I want to show you a few examples. This is early research, so if you have any counterexamples or feedback, critiques, I'd love to hear it.

Let's take a look at some machine learning frameworks, TensorFlow and PyTorch. This is a simple chart of the Google Trends for PyTorch and TensorFlow since 2004. I would make the assertion that developer interest is a very good proxy for value of an open source project. So I would make the assertion that this graph actually shows us very clearly the value of the codebase or the project.

As you can see, TensorFlow is in the lead to start with, very much in the lead, and at some point PyTorch is behind, but at some point they actually cross and PyTorch now becomes the leader. TensorFlow now loses value.

Here's a graph of PyTorch cycle time variance and just cycle time. The variance is in blue, and you can see here that it's actually trending down, which is good. That's what we want. When I look at this graph, like that famous quote from the character from The Matrix, I don't see code anymore. I just see blondes and brunettes and redheads. When I look, I just see money here. This is worth money. This is value. It's going down. That's good. You want the variance to go down.

It also happens to be the case that cycle time is going down, which is also good. By the way, they're usually related, but not always.

I look at this graph. This is TensorFlow. This is not good. Variance is going up. Cycle time is going up as well. Here they are, standard deviation cycle time compared, so you can see the same trend.

From that data, my conclusion is the interest, which is a proxy for value, shift away from TensorFlow toward PyTorch, I think, occurred at about the same time the standard deviation cycle time of the two projects crossed.

Let's look at another example of databases: MySQL, Postgres. Developer interest very high for MySQL for some time, but clearly trending down. Postgres is low, but slowly, subtly growing. They're almost converging now.

Postgres standard deviation for cycle time, very consistent. Also, the cycle time is very consistent. It's trending up a little bit at about a factor of five, but it's quite strong.

MySQL: not good. Standard deviation is going up at a factor of 111. Here the two standard deviations are compared, and that's pretty clear: Postgres versus MySQL.

It's almost the case that MySQL's drop in developer interest, which is again a proxy for value, appears to be the inverse of their rise in cycle time variance.

Browsers: Firefox, Chrome. Firefox is in the lead. Eventually, Chrome surpasses. Chrome cycle time variance trending up at a factor of 90. Firefox cycle time variance trending up at a factor of 99.

Here's how they look compared. The rate of increase of Chrome is less than Firefox, but they're pretty close. However, if you review this, Chrome is actually significantly below Firefox in terms of cycle time and variance as well. So trending in the same direction, but Chrome is better.

Frontend frameworks: Angular, React. React has won. Basically, after a tough battle, variance is trending up at 148. Angular, this is interesting, Angular is trending up only at 85. So what does this mean?

Actually, when I looked at this, I was like, okay, there's a disturbance in the matrix here because this isn't really making sense anymore. I actually think that both Angular and React are moving in the wrong direction, with little distinction. I think that, to me, what this means is there's actually room for a third player. So if anyone would like to invent a new frontend framework, I think you could do really well with your business.

Code editors, last one: Emacs, Vim, and VS Code. This actually pains me because I happen to be an Emacs person.

Yep, Emacs was in the lead. Vim was sort of middle. Of course, VS Code, I don't even think, existed for some time. VS Code is at a factor of 58. Vim is at a factor of essentially 61, and Emacs, unfortunately, is at 280.

Here they are in comparison: Emacs, Vim, VS Code. As you would expect, cycle time variance follows developer interest, which is again a proxy for value. The leader in the market, VS Code, has the lowest variance at 58. Vim has the second lowest at 61, and the least popular, Emacs, is at 280.

In conclusion, I think there are lots of applications where this could be quite useful. I think that as investors, VCs face a lot of challenges valuing technology companies, and they often make wild guesses. They could use cycle time variance as a way to understand future value pretty consistently.

I think as operators, leaders of businesses, technology businesses, CEOs, COOs, et cetera, CTOs, it's really hard to know what to do. Things are extremely complicated. There's all sorts of stuff going on with supply chain and vendors and communication prioritization, et cetera. It's hard to know if what you're doing is correct.

You could also use cycle time variance to understand that better. Are you creating future value for your company, future value for your codebase, with the changes and the initiatives that you're introducing?

In conclusion, I think the application and measurement and reduction of variance offers a way for leaders and investors to make better decisions and manage better.

Thank you so much. Questions.