Dr. Stephen Magill

Hi, I'm Stephen Magill, VP of Product Innovation at Sonatype, and I'm super excited to be here today at the DevOps Enterprise Summit, talking about the latest State of the Software Supply Chain Report.

This is a report that Sonatype puts out every year and has been publishing for the last seven years. I've been personally involved in the analysis for the last three years. A lot of that work was a collaboration with Gene Kim, those more recent analyses that we did. It's always really exciting to go in, redo analyses from previous years, updated with an additional year's worth of data, but also think about additional questions we can ask, additional analyses we can do to try and gain more insight into how people manage their software supply chains. How do enterprise software development teams, open source projects, and individuals stay on top of the constantly evolving mass of software that they're using and incorporating into their applications? And how do they do that securely? That's what we're going to look at today.

I'm going to hit the high points of some of the more interesting findings from this year. If you want full details, you can access the full report; I have a link at the end of the presentation.

I want to start with an overview of the space, and in particular by looking at just how much open source is out there. If we look at total projects that we see in the various ecosystems -- Java, JavaScript, Python, and NuGet -- here is data on the total number of individual projects that exist. JavaScript is absolutely massive, at 1.9 million projects out there. Java has a lot, 430,000.

What's interesting, if you think about this, say you're a Python developer and you think about all the libraries that you use, that you pull in when you start a new project, things maybe you haven't used but you've heard about, guaranteed it's not up at that 336,000 level. We see that when we look at utilization. If you go and look, for a particular ecosystem, at how many of those projects actually show up in other projects' dependencies or are utilized in applications, it's a much smaller percentage of that full set of software. Java, for example, is at the high end, with 15% of those projects in Maven Central being utilized, and JavaScript is down at the low end. If we look at npm, only 2% of the projects that are published to npm are actually utilized by other projects.

The next thing I want to focus on is growth. Yes, it's a small portion of these projects that are used, but still, it's a large number and it's growing. We see 71% growth in the Java ecosystem, still a very vibrant community growing very quickly. JavaScript has 50% more projects year over year: 50% more projects in the JavaScript ecosystem than there were last year. You can see numbers for all of these. They're all growing at a very rapid pace. Open source obviously is a huge success, continues to be, and continues to grow both in supply and in utilization as well.

The next thing is the vulnerability landscape. If you look at where the vulnerabilities lie and how they break down in terms of usage, you see a really interesting situation where it's the most popular projects that have most of the known vulnerabilities. That sort of makes sense, because the security community, both white hat and black hat, focuses its efforts on those most popular projects. You have way more impact, positive or negative, good or bad, if you find a vulnerability in a highly utilized project. Those projects get the attention. Those projects are, by and large, where we see CVEs being published.

It doesn't mean there aren't vulnerabilities in less-used projects. It doesn't mean you should go out and pick some obscure dependency to base your application on. That might not be good from a technical risk perspective, and there are probably vulnerabilities there too. They're just not known yet. But what we see, to take Java as an example, is 23% of the 10% most popular project releases are vulnerable. If you look at the 90% least popular projects, only 4% of those releases are vulnerable.

Here's a graphical representation of this. You can see this is broken down by decile, a 10% slice. The top 10%, like I said, 26% of those are vulnerable. The next 10%, only 7% are vulnerable; then 7%, 3%. You can see it drops off very quickly. The height of these bars is the popularity, so popularity is very skewed. Those top 10% of the projects are much, much more utilized, much more frequently utilized than the rest, and you can see the security community's attention basically mirrors that.

The next thing to mention is the huge increase we've seen in what we're calling the next-generation software supply chain attacks. This is things like dependency confusion, typosquatting, malicious code injection, sneaking code into a repository upstream. That has just exploded over the last couple of years. In 2020 it was down below 2,000 instances. In 2021, it was up at 12,000: a 650% year-over-year increase, which is huge. It's been good to see there is awareness of this. People are noticing. There are technology solutions out there that are starting to address these, and the tools are catching up. You can stay safe with respect to these dependency confusion attacks, but it's important to know that they are happening, and they're happening more and more.

For the rest of the analysis, I want to talk first about the data set that it's based on. We looked at four million dependency upgrades, cases where a dependency was upgraded from one version to another. There were 234,000 dependency versions represented across that data set. This is a Java data set, so it is Java-centric, although I think a lot of the findings extend to other ecosystems as well.

Forty percent of the open source projects that are out there occurred in this upgrade data. If you go back and look at the roughly 434,000 in the Maven ecosystem, that's about 10%. So about 10% of that ecosystem was utilized in this data set. What's interesting is that the four million dependency upgrades, those upgrades and those 234,000 dependency versions, the 40,000 projects, represent that 25% of the dependencies were actively managed. What I mean is that when you look at all the dependencies that show up in this data set, only 10,000 of them, only 25%, were actually updated at some point. This data set covers the last year.

That's the general data. Now what did we do with that? First we looked at vulnerability density. I was talking about how in the top 10% most popular projects, 23% or 26% were vulnerable and then it drops off from there. If we look across all of the versions that were utilized in this data set, 8% of them were vulnerable. That's not small. It's not 26%, but it's not small.

What's interesting is that while 8% in general were vulnerable, if you go and look at a per-project level -- you're using some library as a dependency, and the version that you're on maybe is vulnerable -- you might reasonably ask, can I fix that? Is there a version I can move to now that's not vulnerable and remediate this security issue that I'm pulling in? The answer is a resounding yes. Only 784 of these projects that were being upgraded had no remediation path. There was no version that you could move to to fix the vulnerability. That's only 0.3%, and these are very different numbers. It helps to see it visually to get a sense of this scale.

This is what 8% looks like compared to 0.3%. If you think about this talk's title, "The Minefield of Open Source," if you think about the graphic at the left as this minefield you have to navigate, and you want to avoid stepping on a red square because these are the vulnerable dependencies and you don't want to pull in those vulnerabilities, it looks kind of dire. There's a lot of red out there. But the figure at the right is really the thing to be concerned about. These are the places where you could get stuck. Generally, I should mention, you're stuck for a narrow window of time. I'll say more later about how to choose good dependencies, high-quality dependencies that release frequently, so that you can minimize any window of vulnerability. If you have good tooling, if you're paying attention to what's vulnerable, what's not, and what a safe upgrade path looks like, you can be in a space that's much more like the figure at the right.

That's the landscape. How are industry participants and companies approaching management of their own dependencies? This is some survey data. We've done the survey two years now, asking industry software developers, team leads, and others: how do you approach managing your software supply chain? What do you have in terms of controls around build and release, around consumption of open source? How do you manage risk at your organization?

This graphic represents how mature the participants in the survey as a whole are with respect to these various dimensions of control. It goes from completely unmanaged at the left, where we're just not paying attention to that at all, to the far right, a monitor-and-measure approach where you have controls in place and you're actively monitoring to ensure those controls are working, changing things as needed to make sure you're on top of whatever risks or threats might live in that area of the software development process.

Right is better; right is more mature. These show the distribution across the data set of maturity for each of these. We can see remediation is actually quite good. As a whole, the industry is doing a pretty good job making sure that as vulnerabilities arise, as those squares in the space I was showing earlier turn red, they are moving on fixing those vulnerabilities and keeping things secure.

What's interesting is there's a lot less maturity around how you choose suppliers. The questions in this category were all around what process or standards you have in place when you go to add a new open source dependency. What sort of evaluation do you do of that project? How do you evaluate what that will mean for your technology development going forward to now be depending on this project? Is it a high-quality project that it's fine to depend on, that will benefit you, stay up to date, and stay secure? Or is it a project where you're going to run into problems? Maybe it's breaking your build a lot, changing the API, or not responding to security incidents.

What's interesting there is that's how you get proactive about security. Remediation is responding to things that come in, being reactive. As CVEs get released, you're fixing them. Supplier choice is getting proactive: saying we're going to start bringing in things that we know will cause fewer problems down the line, we're going to do some planning, and we're going to get ahead of this issue. I want to talk next about how you get proactive. If you were to go back and say, I'm going to take my score here in the supplier category and improve it, what are some things you can do?

One is to pay attention to some quality metrics. These are various quality metrics that have been published over the last few years. There's Mean Time To Update, MTTU, which is something that I introduced with Gene and Sonatype a couple of years ago. It measures how quickly projects update their own dependencies. There's OpenSSF criticality score, which is a popularity-based metric that asks how critical a package is to the ecosystem: how many people depend on it, how many people contribute to it, and things like that.

There's also a scorecard metric from OpenSSF, which is more of a checklist of various best practices that you should be doing. Are you using CI? Are you scanning code with static analysis tools, linters, and things like that? They don't have a method for distilling that down to a metric, so there's no numerical scorecard result. We didn't include that in the analysis because it was not quantitative, so we couldn't do the sorts of data analysis that we were doing with the other metrics. But I'd love to see a quantitative version of that; I think that would be great.

Then there's a libraries.io source rank metric, which is also something you'll see if you go to libraries.io and search for a package. It's a measure of various things, with a description there about what it's measuring. Over at the right, I have a summary of the types of things these various metrics are measuring. Is it mostly popularity-based? Popularity is a big component of some of the libraries.io source rank analysis. Is it measuring maturity? Is it measuring, at its heart, more development practices, or is it looking at dependencies? MTTU is very much looking at the dependency structure, so it's high on that axis. You can see a snapshot and overview of what these metrics focus on, and what's interesting is that they focus on different things.

That makes it interesting to analyze them and ask which one is most associated with various outcomes that we care about. That gives you a sense of which project attributes are more associated with particular good outcomes. That's exactly what we did. We asked: suppose developer A chooses a high-quality component and developer B chooses a low-quality component for one of these quality scores, and we looked at each one in turn individually. Is A's project less likely to have vulnerabilities? Is A less likely to experience breaking changes as they keep their dependencies up to date? Are they going to have to do a lot of work to do that?

What we see is yes. For certain quality metrics, high quality with respect to the metric is associated with good outcomes. For example, low MTTU projects that update their dependencies quickly are less likely to be vulnerable. Projects that are slow to update their dependencies according to MTTU are more likely to be vulnerable. This was the only metric showing statistical significance for this category.

High-quality projects in general are less likely to have breaking changes. This was true with respect to all the metrics. OpenSSF's criticality showed the greatest effect. This is a summary of the different effects. MTTU: 1.8 times less likely to be vulnerable if you have faster MTTU, and 3.2 times less likely to have breaking changes. The high scorers according to criticality were eight times less likely to have breaking changes, and you can see the other results.

One interesting thing is libraries.io and popularity. We included the question, what if you just choose based on popularity? A lot of people do that. Those were not good predictors when it comes to vulnerability, or they were not associated with security. More popular projects were actually more likely to contain vulnerabilities. That exactly matches and mirrors the findings I talked about earlier, the population-level observation that the most popular projects are where the vulnerabilities live. That's where the security community is spending its time. Libraries.io includes a lot of popularity-type evaluations in its metrics, so it makes sense that that mirrors popularity's results.

MTTU is positively associated with good outcomes, which is great. Another great thing is that MTTU is improving over time. If we look at the Java ecosystem, Maven Central, we can see each year the dot or circle where the average MTTU is for that year, and we can see that decreasing over time, which is great. The community as a whole is getting better at keeping up to date and keeping secure when it comes to these vulnerabilities that come in through dependencies.

That's an overview of how you might choose high-quality components. Now, how do you keep those up to date once you've chosen them? First, when you think about where a developer is spending their time when it comes to updating dependencies, it's pretty narrowly focused. Only 25% of the dependencies occurring in this data set were actively being updated. This is a graphic of all of Maven Central: only some of those are being used, and only some of the utilized projects are being kept up to date.

When we look at how they're being kept up to date, there's a whole lot of what we're calling suboptimal update decisions. Sixty-four percent of those updates were classified as imperfect, not optimal. What does that mean? We had a number of rules for what an optimal update decision looks like. Some are objective guidance: don't use alpha, beta, or release candidate versions unless you want to be a beta tester. Don't do that in a production app. Don't upgrade to a known vulnerable version. If there are only vulnerable versions available, at least choose a low-severity vulnerability and try to be as secure as possible. Try to choose the latest when there's a tie. Then there are subjective criteria, like choosing common update paths, staying with more popular versions because that's better from a technical support perspective, choosing newer versions, minimizing breaking changes, and things like that.

Most updates do not qualify as optimal according to this. What does that non-optimality look like? It looks like a lot of wasted work. Imagine an imperfect upgrade. These rows are updates that a project made. It had some dependency, updated it from 1.1 to version 1.5, and then from 1.5 to 1.8, but that whole time 1.9 was actually the latest version. Then it did another update, and finally landed at the latest in update decision four. What it could have done -- the blue lines are the optimal path -- is go from the out-of-date version directly to the latest version. When a new version comes out, go directly to that version. You can see there were four updates when you could have gotten by with two. If we assume each update takes some fixed amount of work, that's wasted work.

How much wasted work? A lot. For a medium-sized enterprise that has 20 application development teams, we estimated you would save 160 developer days per year, which works out to about $192,000 depending on development costs, benefits, and things like that. There are definitely cost savings to be had by being more efficient about dependency update.

It's also important from a planning-work perspective: being able to make the right decisions, having the right context, and having the space to make the right decisions. This is a view of all the updates for Spring Core, a very commonly used enterprise framework. Time goes from top to bottom. Versions are on the horizontal axis. There's a lot packed into this figure. One thing to note is that how dark the squares are shows how many people are updating to that version. The most recent versions, the ones at the right, are the darkest. Most people are staying up to date or at least close to the edge. Because there are two dark edges, this shows there are two minor versions being officially supported by the Spring project. The vertical gaps are gaps between minor release versions.

That's great; those projects are doing well. What's happening at the other side of the figure? First, there's a whole lot of red. Red means these people have updated to a vulnerable version. You see a lot of versions that are known to be vulnerable still being utilized, probably for a variety of reasons. We haven't gotten into exactly what might feed into those decisions, but certainly these projects over at the left are having to be in a very reactive mode.

When I was talking earlier about proactive versus reactive: if you're over at the right, you can be proactive, you can plan, you can say, okay, there's a new version of this project out, we need to update in the next 60 days, that's our standard, let's plan that work and get it done. If you're over at the left, you're responding to vulnerabilities as they come in. You can imagine a wave of red moving from left to right. As additional security research happens, new vulnerabilities are discovered and disclosed. People over at the left using these older versions are much more likely to have a high-severity CVE come out that forces them to do an update in the moment, unplanned. That's a really important reason to just stay up to date in general. It gives you that space to plan.

Those are the findings in general. What guidance comes out of that? What should you do day to day as part of your software development process? First, have a process to choose high-quality dependencies. I went through some of the quality metrics that are out there and what we see in terms of association between those and good outcomes. MTTU was a good one to pay attention to. But pick some process, have some quality standard, and apply that when you pull in new dependencies.

Have a process or tooling that lets you chart a safe course through that minefield of vulnerabilities. When you are making a decision about updating because there's a vulnerability, make sure you have access to the data to tell you which one you should update to.

Live near the edge so that you can be proactive, but not right at the edge. When we did the analysis, on average, the optimal version was 2.7 versions from the latest. That's because latest versions are more likely to be release candidates or still in beta. They're more likely to be subject to these supply chain attacks that I mentioned. They're more likely to cause breakage or bugs. Staying back a little bit lets you see how a version is landing, see how the community is reacting to it, and give it some time to have issues discovered before you adopt it into production. You can be aggressive for non-production apps, but for production apps, you generally want to be close to the edge, but not quite there.

Those are some of the guidance that came out of this year's report. There's a lot more data in there, a lot more analysis. I'd encourage you, if you're interested, to check some of that out. The website is on this slide here. There's the report, the full PDF, many pages. There's also a great summary in webpage form. You can click through and see some more of the graphs, some more of the analysis that came out of this year's report. Thank you.