Kim Weins

We're excited to be here today to talk to you about saving the software supply chain: critical trends and five must-dos for DevOps. I'm excited to have here with me today Daniel Nurmi, who's our CTO and co-founder at Anchore, and I'm Kim Weins, the SVP of Marketing at Anchore.

All right. So that'll be the gap. Let me go ahead and present, and then I'm going to go over here, share the screen. Okay. Put this down in maybe the lower corner so it won't be obvious. All right, so we'll go ahead and continue.

Well, do you want to just kick it off, Dan? I don't need to intro you. Why don't I just go to the next slide, and then you just start right here?

Daniel Nurmi

Okay, that sounds good. I'll start right now.

All right. Thank you very much, everyone, for being here. We're going to start this presentation by talking about the question: why focus on software supply chain? And it's a little bit about a definition of what we're going to be presenting here.

What we're looking at here really is a depiction of a common CI/CD or development infrastructure that you might have in place. This might look familiar, where on one side of this picture we start with our application source code, and then in the middle we're going through a number of stages of taking that source code, turning it into executables, doing build pipelines and things like that, where at the very end, the output of this kind of machine is we end up with something that we're publishing, software we're publishing and offering out to our consumers and customers, or we're taking those applications and executing them in production.

Now, underneath that, we've also got our platform. This is all of the infrastructure that facilitates this whole process: our source code management systems, our CI/CD systems, automated QA and testing, and finally, things like our publication systems or our orchestration and execution environments.

Now, when we look at it, usually as software producers, we just look at this as a machine. It's functional. It's doing a lot of work for us. When a malicious actor looks at this, they see a lot of interesting areas where they might be able to come in and infiltrate this environment in order to influence your software, injecting malicious code or putting in a bad password or escalating privileges, that kind of stuff.

And so when we look at this picture from a security perspective, we also have to think about, for all of these stages, these are the kind of areas that are potentially weaknesses that could expose us to a security attack. And generally speaking, when you're talking about supply chain security attacks, these are the kind of areas where hackers will come in and make some attack happen in order to get their code introduced into your code or your infrastructure, and then your software is compromised so that people downstream consuming your software or software is executing has that malicious actor's software already in it.

So this is kind of the general construct that we're going to be talking about today. And next, I'm going to pass it back over to Kim, who's going to talk about a survey that we performed, asking folks various questions about the topic.

Kim Weins

Thanks, Dan. So one of the things we wanted to do today is really bring some data to bear to look at what are organizations doing today, and how are they concerned about the software supply chain, and how are different practices in their organization either helping or hindering in software supply chain security?

So the survey we're going to be referencing today was of 425 technology leaders at large organizations. So that means organizations with at least 1,000 employees. Many of them were at much larger organizations, over 5,000. And this was CIOs, CTOs, CISOs, VPs, directors, managers, and architects that spanned across DevOps, development, IT, security, and cloud, across a variety of industries. And we're focused in two regions, about three quarters in North America, and then about a quarter within the European regions. So this is really going to give you a picture of what's happening in those larger organizations in terms of software supply chain.

So one of the interesting trends that we saw is, not surprisingly, growing container adoption, but also that growing container adoption is creating new security challenges. So this data shows us that a quarter of the users that we surveyed fell into a category of advanced container adoption, meaning that most of their apps were in containers. And then another 40% fell into the intermediate category, where a significant number of their apps are in containers. So together, that really represents about two-thirds of the respondents were really in those middle to later stages of container adoption.

Now, as we ask those different users at different levels of maturity how they saw supply chain risk, the interesting thing here is that the more advanced they were in container adoption, the higher their perception of supply chain risk. That's really interesting because normally you might think this would be in reverse. In the cloud space, for example, we saw that as users used cloud more and they got more comfortable with the tools and the technologies, what the cloud providers were doing, they started to worry a little bit less about security. They knew that they could solve and address the security issues. Here, we're really seeing the inverse, that as people start to use containers more, they start to recognize that there are potential opportunities for threats to come into their supply chain, into their software factory, so to speak, into their DevOps process.

So Dan, maybe talk a little bit about what you think that means for DevOps teams.

Daniel Nurmi

Yeah, absolutely. These are very interesting responses to Kim's point, and this leads us to what we consider a must-do, which is really around the idea of we, as practitioners, as software providers, we really need to create specialized processes that actually understand containers.

And the reason for that is that, well, first, containers represent a new type of software bundle that some of the more traditional security tooling isn't aware of. Containers can have, yes, the application that you're building, and the container usually has a name. It is an application or something like this. But it also can typically have almost an entire operating system environment. That's additional software, additional configuration, other stuff that comes in in support of that executable application. Containers can also have data artifacts. They can have secrets and keys. Anything that's required to support the execution of that application in an encapsulated, container-like way are also elements that are part of that executable.

So development processes built around containers, in addition, are typically built with the idea of a high degree of automation in mind. And this leads to, in your process, oftentimes a lot of containers being generated, whereas previously you might have had far fewer executable artifacts. Now with containers, we might see a lot of containers, a lot of things that can actually execute or be distributed to your clients and customers. That's another interesting new thing about containers.

And all of this really leads to the fact that when choosing tools or looking at how to change your processes to deal with the fact that you've got containers with these new characteristics, you really need to make sure that both our processes and our tooling understand what a container is, what are the security implications of having them, and how to get that information and then secure them properly.

Kim Weins

All right. Great. Let's go on to some additional trends.

So the other thing we want to highlight is that cloud native applications, often built with containers, use code from multiple sources. And this shouldn't be a surprise to anybody in the DevOps world. We asked people what they estimated to be the source of software in their containers, and they estimated that about a quarter of that code was open source. Now, in fact, we've also seen data from scans of software applications that this may, in fact, be much higher, that in many industries, it can be from 60% to 80% of source code is represented by open source. But that's something that organizations are really learning how to grapple with, how to manage, and how to secure.

Now, given that open source is such a big part of source code in modern cloud native applications, we asked people about their top container security challenges, and you can see here at the top of the list is the security of open source containers. So 23% of the respondents rank that as their number one security challenge, followed by security of the code that we write. So looking at the fact that open source is representing more and more of your code and that people are challenged by open source security, that really leads us to our next DevOps to-do.

Daniel Nurmi

Yeah. This was another interesting finding, not necessarily as closely related to containers, but definitely related to the concepts of supply chain security. And we think of this as boiling down to the statement, you really need to pay very close attention to software that is not developed in-house.

So the critical, I think, concept to understand is that while we as a software development organization or any organization that's got some software development in it, we understand our own applications very well, and we think of what we're producing as our application. But in reality, most applications today really do depend on a quite large and rapidly updating collection of external software coming in from a variety of external sources, all of which have their own security levels and processes involved.

And so the real takeaway here is, from our perspective, it's really important to differentiate between the kind of security controls you're applying to your own application source code and processes, from the security implications, tooling, reports, policies that you're generating around all of the external software that you're bringing into your environment in support of your application.

Kim Weins

All right. Let's move on to some further trend data, and this was really a finding that diverse DevOps toolchains are definitely going to require security attention. So we asked people about the CI and CD tools they were using in their organization. We wanted to understand whether there was diversity of tools, and we found that there absolutely is.

Only in 20% of organizations was there really a single CI/CD tool that had been selected and was being used by all teams. In 38%, there might be a preferred CI/CD tool with exceptions, and then in the remainder of the organizations, there was either multiple tools or people were just selecting their own tools, meaning that it was a free-for-all.

And I think that's because development organizations want to be as efficient as possible, and so they want to pick the tools that they're comfortable with. Also, as organizations grow and acquire, often new tools are brought in to add to existing tools.

Now, it wasn't really just about the CI/CD tools, but beyond that, when we started to ask about things like registries and other repositories as well as CI/CD tools, et cetera, we found that on average, organizations were using a median of six different DevOps tools. So that means that there's a lot of different tools that exist in the organization that all need to be secured as part of your software supply chain security initiative.

Daniel Nurmi

Yeah, and this was another interesting one because it definitely reflects what a lot of organizations already know, which is that there's a lot of variety in your choice of infrastructure tools to facilitate that SDLC or that development life cycle. Now, from a security perspective, something interesting has really come out of this observation at this moment in time, which is that we're seeing more and more of these types of infrastructure platforms are building in more security controls into their own platforms, which is definitely a good thing.

However, oftentimes those security capabilities are really matched to the platform itself, and so it's good that there's more security controls built into a lot of these platforms, but it's also complex and creates some problems operationally in that if you have five or six different CI/CD systems, each with their own security tooling, it's sometimes hard to gather all that back and get a consistent policy across all of those tools that are providing slightly different reports and capabilities and functionality.

So from our perspective, it's okay to have multiple tools all providing security scanning, but it's really important to have at least one type of tooling that can sit on top of your entire environment and be able to aggregate all that information, do its own scans, even if they're overlapping, because there's certainly nothing wrong with having multiple security tools checking for the same thing.

And in addition, that platform-agnostic approach can actually get you a benefit that sometimes we don't think about as much, which is that it provides you a little bit of insurance against your platform being infiltrated. That can also lead to your security tooling being infiltrated if you're using the same security capabilities of the platform itself. So by externalizing that, you give yourself a little bit of risk reduction for getting your security value performed over all of those environments.

Kim Weins

All right, great. Let's move on to the fourth trend, that there is room for improvement in securing the software supply chain.

So we asked organizations whether they have been affected by a software supply chain attack in the last six months, and you can see that about 60%, actually 64%, were impacted. In some cases, that was a very significant impact, some cases moderate, and in some cases, a minor impact. But really, this is starting to hit most organizations out there today, and we expect that to increase or continue as time goes forward.

So we also then wanted to understand, okay, if you're being impacted by attacks, what type of supply chain practices have you implemented already? And specifically, we were asking for containers. And you can see just about half are scanning third-party containers and open source containers. And then from there, some of the practices really go down where very few people or a very small minority are implementing them.

And I want to especially identify the things that are at the bottom of the list, which are about SBOM. So SBOM stands for Software Bill of Materials. This is starting to become a way that people are recognizing as an important aspect of being able to help to secure the software supply chain. And very few organizations are either requiring an SBOM from their software suppliers who are sending them software, and also very few are producing SBOMs for any software that they are creating. So that leaves an opportunity for some improvement.

Daniel Nurmi

Yeah, and this leads us to our next must-do, which is, from what we're seeing in the security space when it comes to the conversation around supply chain security and a number of other things that I'll discuss in a minute, we feel that it's time now to really get on a path to start producing and asking for software bill of materials from your software providers, and as a software provider, to provide that same kind of metadata.

And we're seeing this more and more. Organizations are starting now, as we saw in the response. It's a small percentage relative to some of the other responses, but it's not zero. Organizations that are actually asking and requiring that their incoming software include, alongside the software itself, this rich set of information about all of the other software, including open source dependencies and vendor dependencies and configurations and proof of security practices, licenses, rich metadata that comes along with a downloadable, essentially, that practice is only going to continue to increase.

And this is something that's been manifested pretty concretely in the U.S. executive order that came out earlier this year, which is really moving forward at a fast pace. And we look at that as a very good example of this practice starting to become a reality, where that order actually stipulates, in order to provide some software to certain government agencies, it must come with an SBOM and a number of other materials alongside the software.

So we predict we're going to start seeing that happening more and more, even between enterprise organizations and other organizations and projects, between open source projects and organizations. And so we think that now it's a really good time, and it's actually pretty straightforward to start including in your software development practices the generation of these software bill of materials documents anytime you're building new application software.

Kim Weins

All right, great. A final trend: what challenges are organizations facing with container scanning?

So we asked respondents what were the top container scanning challenges, and you can see here we've differentiated between a significant challenge versus somewhat of a challenge. The number one is probably not surprisingly, identifying vulnerabilities in containers. But then it was interesting what came up as number two, three, and four, which was about getting developers to remediate, wasting developer time on false positives, or it taking too long to remediate. So these were really about the issues that developers are facing or organizations are facing as they try to not just identify the vulnerabilities but fix them.

We also asked about false positives specifically, because this is something that we've heard about being a significant problem, I think we're all aware of that have been working in securing software. But we wanted our respondents to estimate what percentage of vulnerabilities that are found do they feel are false positives. And really almost half, 44%, they feel are false positives. And so that really raises some questions about really how do you manage and deal with that when we know that in the CVE data and also in the nature of the scanning, that there can be a lot of false positives.

Daniel Nurmi

Yep. And this work leads to our final must-do recommendation, which is actually oriented a little bit around the notion of shift left. And so before I talk about that, just briefly about the topic of false positives.

Going all the way back to some of that information I was going over earlier about containers and how they're composed, there's a lot of software inside of a container, and as we know, there's lots of open source dependencies. The vulnerability data being published out there is, there's a lot of it, it's not all in the same form. And so it's sort of a reality that with scanning tool that's looking specifically for software vulnerabilities, we need to live with the notion that there will be false positives, or almost equivalent to an actual false positive is a vulnerability that exists and is accurate, but when our security team does an assessment, determines that it's not going to impact the application, right? Which is just as expensive as evaluating a false positive, maybe even a little bit more.

But we're going to need to sort of live with that reality that there will be a constant flow of software and vulnerabilities reported against that software. And so the must-do here really is about the process for dealing with it. And in our opinion, shift-left security approaches are really an important way to try to address and handle this reality a bit more efficiently.

The reason for that is, well, first of all, shift left is the notion of making sure that through all of the stages of your software delivery life cycle, you've got some security scanning. You can look for as many things as you can at every stage as new software is being introduced towards that stuff being executable. Do that early and often throughout the entire software delivery life cycle. We kind of consider that a shift-left security approach.

And by doing that, there's some real benefits you get. The obvious one, in a sense, is that the earlier we're able to find issues, the more valuable it is to be able to detect that, because the later you go in that stage, the closer to exposure you're getting. So at the extreme, we've got an application that's actually running in production. Finding a security problem at that side is far more expensive than if you were able to find it just prior to execution, for example.

But the other value that may be a little bit more subtle and really maps more to this idea of making the process of dealing with false positives more efficient is that the sooner you can find a security violation or finding, the closer you are typically to the team or the person or the process that has the ability to remediate it. Rather than looking at a running system and having to go all the way back to a developer, if you're able to find that security violation right after the code has been committed, you're very close to where that developer, there is a person who can fix the problem lives in terms of information flow and the ability to get that done quickly, far more efficient to be able to find these things as soon as possible. And so that really leads us to that must-do of implementing shift-left processes. It's far more efficient and valuable when it comes to securing your chain.

Kim Weins

All right. Thanks, Dan, and thanks everybody for joining us for the session today. We'll be available on the Slack channel for Q&A.

And we shared some results from a survey. If you want to get the full results of that survey, feel free to go to anchore.com/survey, and you'll be able to get all the trend data to help with your DevOps process and to help make that more secure to protect the software supply chain. Thanks, everybody.

Daniel Nurmi

Thank you very much.