Rani Osnat, Daniel Nurmi, and Rory McCune

Rani Osnat: Hello. Hi, everyone. This is Rani Osnat, VP of Strategy and Product Marketing at Aqua Security, and I will be the moderator today for our session. Very pleased to have with us Dan Nurmi, CTO at Anchore, and Rory McCune, Cloud Native Security Advocate at Aqua Security. Hi, guys. Why don't you go ahead and introduce yourselves. Daniel, you go first.

Daniel Nurmi: Thank you so much for having us to talk about this important topic today. My name is Daniel Nurmi. I am the CTO and co-founder of Anchore, and we are a company that was started around the time when containers were really starting to pick up steam. What we do is produce technology, open source, and products that really aim to help organizations who are adopting modern development techniques introduce security and compliance best practices directly into their DevOps toolchains. Very pleased to be here and talking to you today.

Rani Osnat: Thanks for joining us. Rory?

Rory McCune: My name is Rory McCune. I am a Cloud Native Security Advocate at Aqua. We are focused heavily on the cloud native and container ecosystems, and really helping customers as they go through their lifecycle, moving more into cloud native and into containerization, and securing that entire process. I am really glad to be here to talk about this, as it is an issue that is attracting a lot of attention at the moment.

Rani Osnat: Today our topic of discussion is software supply chain attacks and why they matter and what DevOps needs to do. This is something that has obviously received a lot of attention recently. Why is there a focus on the supply chain right now? Daniel, you start.

Daniel Nurmi: This is, as Rory was mentioning, a really topical discussion that has been going on, and there has been a lot more focus, especially over the last six months to a year. Part of the reason is that there have been some high-profile security events: things around Codecov and SolarWinds and other types of very high profile, large impact, large blast radius security events. The discussion after the events started to ask the pertinent questions: what happened here, why is this something that is starting to affect so many organizations when the actual event happened in a very focused location? Some of that discussion started to formulate around supply chain security.

Daniel Nurmi: In addition, a lot of this has to do with the adoption of modern development infrastructure techniques, where there is a lot more automation and a lot more software being combined to produce applications that ultimately run in production. The industry has started to focus on all of the processes used for automating development and delivery of applications. With that automation, there has been an increase in the amount of software, the dynamism of these systems, and how it all comes together. We need to focus on the process of building applications because that is part of a more general global supply chain, and that is where these events have really been focused.

Rani Osnat: Rory, anything to add?

Rory McCune: It is a good point that we are seeing a lot of this now, but this has been building for a while. I did a talk on this kind of topic six years ago at an OWASP AppSec conference, looking at things like npm and RubyGems and how those package supply chains were secured. Security people probably were thinking this was going to happen sooner or later, and warning about some of these topics. I went back to a quote from 1984, when Ken Thompson said, "You can't trust code you did not totally create yourself," in "Reflections on Trusting Trust." Like a lot of security, this has been going on a long time. What we are seeing now is, as Daniel was saying, huge adoption. Attackers follow adoption. Attackers follow the process of, this is where people are, therefore this is where I want to be, and then we get incidents.

Regulation and Open Source Response

Rani Osnat: We have seen very recently there was an executive order in the United States. Given this, how do we see the open source community responding? What is coming down the line, and how will this impact DevOps as we stand today?

Daniel Nurmi: Building on what Rory was talking about, this is not necessarily a fundamentally new topic. One of the events that came out of a lot of security incidents over the last year and a half or so was this US executive order. It is essentially a laundry list of the same type of security and compliance mandates that the security industry has been talking about for a long time, but it started to focus it and codify it, recommending requirements for US federal projects to ask for information and proof of security process from software projects coming into their projects. It is almost defining a more formal way to describe that, as software consumers, these are the kinds of things we are going to start requiring from software producers in order to add to the assurances that can be used to assert some level of trust. It is a good flag in the ground for a supply chain security discussion because it starts to formalize those requirements.

Rani Osnat: My perception has been that they are catching up to some of the things that have been happening rather than imposing something entirely new. They are taking best practices that are out there and implementing them as a best practice at the federal level. Rory, what do you think? There are several initiatives happening in the open source world specifically. How do you think it will impact them?

Rory McCune: It has been driving conversations that were already happening. The CNCF had a white paper already in draft around supply chain security before the executive order came out, but this ties that in and gives greater impetus. Another thing we have seen off the back of the executive order is big tech companies like Microsoft and Google stepping up with millions of dollars and saying, we are going to start putting this money directly into initiatives around this problem. Would they have done that without the executive order? Possibly, but it is fair to say they did this after meetings with the US government. This is essentially saying, companies are making a lot of money out of this; they need to give back to security.

Rory McCune: The risk with regulators stepping into this world is that it is hard to marry regulation, which moves quite slowly, with the very fast-paced cloud native development landscape. There is concern that they will come up with things that are complicated and hard to implement. Hopefully, if they take advice from people like OpenSSF and CNCF, that will not happen. People with more experience on the ground can drive the detail while regulators handle the overarching piece.

Rani Osnat: If you have questions during the session, put them in the Slack channel, ask the speaker track four, and we will engage with those questions as we go along. Do either of you think there is a risk that when the federal government gets involved, things can get a little crazy and not necessarily move in the right direction? Is there a risk of people throwing money at things that are not effective, trying to satisfy checkboxes instead of addressing the issues more deeply?

Daniel Nurmi: That is a fair general comment. In this case, having looked at that executive order and tried to understand the implications, it is dense, but what it really boils down to is a formalization around this: as a software consumer, we are going to be asking for software bills of materials. We are going to be asking for some proof that the producer of software is doing security practices. They do not go so far as to dictate exactly what those things look like at a fine level. Anybody who has been a software producer, even selling software to a large organization, has seen requests for the same kind of information. A big enterprise will ask to see a spreadsheet with open source dependencies, a report of development security practices, and what compliance standards the company conforms with. So I do not see a huge risk in this case.

Rory McCune: I agree. My only concern is when high-level standards come in, there tends to be a process where someone says, I need an audit guide, something I can send with a group of people to a company to assess. At that level, when people translate from high level down to detail, there is a temptation to try to make security perfect, which it never is and never will be. If someone writes a standard that says, do this and you have perfect security, it will end up 1,500 pages long and almost impossible to comply with. I hope they resist that temptation. I do not see anything yet, but history tells me it is possible.

Audience Question: Media Attention and Threat Models

Rani Osnat: There is a question from the audience about why Codecov did not get more larger media coverage.

Rory McCune: Security incidents come faster and faster these days, and the news cycle turns so quickly. Codecov was really serious and got that spark of attention, but the next week something else came along and moved it out. When breaches first became a big thing, people said companies would take things more seriously, but now breaches are every week and part of doing business. No one pays attention unless it is truly vast. I hope supply chain will not go that way, but Codecov is a good example of a conversation I thought would last longer that dropped out quickly.

Daniel Nurmi: There is probably some fatigue after a while. There is only so much "another supply chain attack" that people can ingest in a short period of time.

Rani Osnat: Speaking of that, what do you think of educating people about the risk? Before best practices and what can be done, what are we protecting against? What is the worst case scenario or likeliest scenario in supply chain breaches?

Rory McCune: That comes down to threat model. What is the threat model of your organization? Who do you think you will be targeted by? That will drive the kind of supply chain attack you will see. Everyone will see the automated attacks spraying every system on the internet. But then there is the targeted piece. The really tricky one for modern software is: who are your customers? If you have one high-profile customer, attackers might go for you because they know you are in the supply chain of someone high profile. We have seen hosting companies compromised because one of their customers was a target. It is not just my threat model; it is my ecosystem's threat model and my customer's threat model.

Daniel Nurmi: I agree. When we talk about supply chain and what we can do about it, we need to understand whether we are playing the role of a software producer, which means you might not consider yourself a target, but if your customers are, you are absolutely a target. Or you may be a consumer at the end of a supply chain, needing to know what to ask for and how to understand all of the software coming into your environment. Some of these concepts are recursive, but it is important to position ourselves in one area or another.

Rani Osnat: I once worked at a company that sold software to the automotive industry. Automotive has real supply chains: manufacturers, tier one suppliers, tier two suppliers, tier three suppliers, and tier four suppliers that create screws and bolts. Every link takes responsibility and their brand equity is hinged on quality delivered to the next one. In software, everything flows through. If I build commercial software using an open source package, and that open source project uses a Java library with a severe vulnerability that infects multiple supply chains, there is nobody stopping this earlier. Nobody is quality controlling the screw. People do not understand they are not just responsible for what they are using, but for whatever that component is using, and so on.

Open Source, Transparency, and Dependency Trust

Rani Osnat: Why is open source so important in this discussion? What is unique about it?

Daniel Nurmi: If we are a software producer, we have expertise over our own application source code. But it is extraordinarily rare that all of the source code involved in an application is source code that your development teams produced. Pretty much every application running in production today has a fair number, if not a very large number, of open source dependencies brought in to support the application. This is a challenge because there are many software elements, but it is also an opportunity. Open source by nature is highly transparent. There are opportunities to understand what open source projects we bring in, inspect the source code, make assurances around provenance, and understand security processes. If we were building only with closed source dependencies, it would be harder to understand those dependencies. We have to understand the total sum set of software used to build our application so we can pass that along to consumers, and open source makes that possible.

Rory McCune: I always talk about open source as potentially more secure than closed source, because you can look at it. The challenge is getting people to do the looking. There is a big challenge because of the sheer weight of dependencies. Take Left-Pad: one tiny JavaScript library got removed from the supply chain and a whole load of applications in the Node.js world had serious problems. It only left-padded text, but it was a critical part of a supply chain. No one is going to code review all their dependencies. Kubernetes, including its dependency tree, has about two and a half thousand dependencies. No one is checking all of those. The challenge is how the ecosystem identifies the key points and makes sure those get the review we want. OpenSSF and the money Google announced for security activities in open source projects are good because breaches, while bad, are spawning useful activity.

Rani Osnat: In cloud native, surveys talk about 70 or 80 percent of the code base being open source and only 20 percent being customer-written code. Also, open source is open, so more known vulnerabilities are discovered there than in custom code. Where do we go from here? How do we solve dependency trustworthiness? What controls can we have?

Rory McCune: There are high-end things, but there is also what is practical and what people can get started with. There are good projects for checking dependencies and versions. OpenSSF has Scorecard, which assesses an open source repository on GitHub against things like whether it is active, whether it is getting patches, and whether it has a security policy. These are basic things, but a good starting point. If the industry starts saying every project should have a security policy and check vulnerabilities to maintain dependencies, that is a good place to start rather than getting put off by the massiveness of the problem.

Daniel Nurmi: There are also interesting architectures for considering what supply chain security looks like and what practices generally need to happen: the SLSA project, the Alpha-Omega project, Linux Foundation and OpenSSF work, and Microsoft supply chain infrastructure designs. A key foundation of almost all of these projects is inspection or insight: transparency into what is actually there. Security teams need information. It is not necessarily first about the security tooling itself; it starts with what software is there. Across all the software coming into the organization, what happens in the middle as we build, and what we deploy, what are all the software components that make up our production environment? That question alone is an important foundational layer.

DevOps, Dependencies, and Toolchain Risk

Rani Osnat: A question here asks to what degree people are not upgrading dependencies because they fear breaking things, enabling vulnerable components to persist. That brings us to a broader topic: what can DevOps do about this? Is this a security issue or a DevOps issue?

Rory McCune: Avoiding old dependencies is tricky because underlying components do not have enterprise support lifecycles of five or ten years. Kubernetes was at nine months support until recently, then moved to 12 months. DevOps has a huge part to play because in an environment where you deploy rapidly and are used to fast change cycles, it becomes easier to make smaller changes. Smaller changes have less risk. If you deploy every day or week on a short cadence, making small changes each time, you are less likely to have 100 patches and need to work out which caused a problem. Fast, regular deployments can help software supply chain security because you can embed it with less risk.

Daniel Nurmi: I agree. The complicated part is deciding whether to always keep to the latest dependencies. That can work, but as your dependency list grows there are functional considerations. What we see organizations employ is deciding whether to update a dependency based on insight about what is happening with that dependency. If we inspect our dependency list, know what the dependencies are, and consult vulnerability sources for new vulnerabilities, that information can guide us to update targeted dependencies because something has happened, test them, and move on. Insight early in the build process helps here.

Rani Osnat: DevOps should be better equipped than traditional waterfall SDLC because it has canary deployments and blue-green deployments. You can test dependencies before deployment or do it gradually. Outside DevOps, things are harder: databases running three years with no upgrades because people are afraid to shut down the server in case it will not reboot.

Rory McCune: I agree. Industries used to multi-year change cycles are now going to cloud native stacks. They need to walk the walk of DevOps along with the move, because they cannot take the traditional lifecycle where you put something in place and it sits forever. Very old software, an unwillingness to reboot, and pet servers, pet databases, or even pet Kubernetes clusters are where you start running into problems.

Daniel Nurmi: Being able to update incrementally and rapidly in almost an automated fashion goes beyond security value. It gets you competitive advantage because you can make changes and add features faster. The combination of that process gives good security value and other business value.

Rani Osnat: What about the security of the DevOps toolchain itself: CI/CD pipelines, GitHub repos, Docker registries, and so forth? Do you see those as risks?

Daniel Nurmi: Absolutely. Those are places where malicious actors could get in. Any place that has the ability to modify the software ultimately being produced or deployed is an area of risk. DevOps teams are adopting techniques where change control is managed with infrastructure as code and other programmatic and reviewable techniques, which is important. But developer infrastructure components are risky and need to be secured.

Rory McCune: All of those services are part of your attack surface. They are elements you need to consider when thinking about controls. The challenge is that it is all on the internet now. These services are publicly available, and you are one access control mistake away from being exposed. Developers may clone a repository and fork it into public GitHub because they want to work on something and are not ready to commit it back. If that were on a private system, people would not know, but it is on GitHub and searchable. CI/CD systems like GitHub Actions or Jenkins are essentially free compute resources, and attackers like free compute resources. Anything that lets me run code will be a target, even if not for super serious purposes.

Rani Osnat: We have seen free versions of online CI tools abused for crypto mining. If people can abuse that tool for crypto mining, they can probably do it for other things too. What steps are organizations taking in the field to mitigate supply chain attacks?

Daniel Nurmi: It is early days and there is variance. Some organizations have invested for years, with serious policies around credentials, deep change control, and some source analysis of open source dependencies. For organizations not at that level, we are seeing focus on supply chain security itself. It codifies controls related to developer infrastructure, CI/CD tools, deeper inspection of dependencies, and consideration on the output side: realizing we are part of a supply chain, what do we need to produce at the end of our internal development process so we can provide more information to the next consumer in the chain?

Rory McCune: People should not feel too worried if there are many new terms coming at them, like SBOM and signing. A lot is still developing. But there are things with good take-up: automated scanning tools, vulnerability scanning of container images, and infrastructure as code scanning. In an automated cloud-native world where infrastructure is described as code, that opens it to static analysis. Even basic checks, like whether containers want to be privileged, can stop a dangerous thing getting into the environment. Getting comfortable with point activities opens you to the next step, rather than trying to fix supply chain security with one big thing.

Deployment Models, Signing, and Vendor Tooling

Rani Osnat: If you are a service organization, like a SaaS provider, how do you balance frequent deployments with clients' expectations of not having frequent changes that may risk service delivery?

Rory McCune: That takes us to partial deployments. You can put a new version out, but not give it to everyone. You can have early adopter and mainstream channels, like big cloud providers do. If you can segregate customers and give different offerings depending on what they want, that can help. It is a hard choice because stability is all in SaaS, but if you leave it too long, you will get an outage.

Daniel Nurmi: A SaaS provider can help by being transparent about how the organization does this. Describe to customers: we have beta, alpha, new features here, stable features there, or we support an old feature for this amount of time before switching over. If that boundary is clearly communicated through major version or release number, transparency helps get customers on board and keeps everybody moving along with you.

Rani Osnat: What are the chances of things like signing and attestation being the way out of this?

Rory McCune: I would not say they are the way out, but they are definitely a component. Signing comes up every couple of years. Some over-hype it, others say it does not do much. There is a middle ground. The key is ease of use, and that is where signing has failed before. I am excited about Sigstore and Cosign. The basic signing process there is simple. You could start signing your container images and binaries today using Cosign. The on-ramp is nice and small, so I am hopeful that initiatives like that will help adoption this time.

Daniel Nurmi: Signing and attestations are not the thing that will solve all of supply chain. They are another element of assurance. Trustworthiness comes from enough assurances of certain types, all adding up. Signing and attestation tell you something about provenance and whether something has been modified or tampered with. That solves vectors around tampering, but it is not the end-all be-all. The Sigstore project is compelling because of how simple it is to start signing things and because it flexibly supports different ways of signing and doing attestations.

Rani Osnat: What do each of your companies do to help deal with this supply chain risk?

Daniel Nurmi: Anchore has been in a position for quite a while where our fundamental technology takes inputs like container images, does deep inspection, and categorizes everything into known elements: packages for different language ecosystems, Linux distro packages, Windows knowledge base article updates. We categorize container images and provide that information as an accessible software bill of materials. Getting information about software coming into your environment and being produced by your development pipeline is something Anchore technology can do. We have open source tooling as well. One is called Syft, an SBOM generator. You can point it at a code checkout or container image and it will generate a rich software bill of materials in standard output formats. We also have another tool that can take that SBOM and produce a vulnerability scan. These tools fit into developer infrastructure on the left-hand side of organizations that are pulling in software, building their own, and producing high quality metadata next to artifacts at the output.

Rory McCune: Aqua sees many of the same problems: container scanning and image scanning, combining open source and commercial approaches. We also focus on going beyond scanning to dynamic threat analysis. When building a container image, run it in a sandbox and look at its behavior: does it do things that appear malicious? Static analysis is great and necessary, but adding dynamic analysis before production is important. We have open source projects like Tracee for dynamic analysis and Trivy for scanning. Trivy is useful for supply chain work because it covers container images and infrastructure as code. Another important element is when pushing into production: having a gate, an admission control that asks, has everyone done the work they were supposed to do? Aqua focuses on applying policies and enforcing them as things go into production. It is about getting the whole lifecycle picture because no one approach in security gets you all the way; you have to combine approaches.

What To Do Tomorrow

Rani Osnat: We are getting close to the end of our session. I want each of you to give one to three points that organizations should do now. Tomorrow morning, what should they embark on to protect themselves against supply chain risks?

Rory McCune: The first thing I would do is try to work out what you have. You might find it is harder than you think. Look at the libraries you depend on and the SaaS services you depend on. Get a picture of those. Then start with point activities. Do not think you have to solve this today; you will not. Get an open source vulnerability scanner. Start with open source projects, because they are easy to get started with and free. Scan your container images. Start scanning your infrastructure as code templates and look at high risk issues. Do not worry about getting everything done today. Start at high risk and work your way down. Job zero: work out what you have.

Daniel Nurmi: I agree. Step back and look at your software chain, your build process, and start thinking of that chain or process as a link in a global supply chain. I am a software producer: I have inputs, processing in the middle in CI/CD, and output. That is a link in a more general supply chain. Think about what you can do on the input side, what processing you can do, and what you should produce on the output side. Reference process initiatives from CNCF and OpenSSF for general models and map tooling in to generate information that, if everybody generated, would put us in a better spot. A lot of us are designing these systems, so consider where on the input side you need to generate information, what processing you need to generate assurances, and how to store them so you can include them in outputs.

Rani Osnat: Daniel, should companies be more restrictive on their inputs, meaning stop the free-for-all of grabbing open source packages from wherever and using them to build things quickly, but not necessarily safely?

Daniel Nurmi: I cannot say that generally, because it matters what you are doing with your software and what the consumer of your software is expecting. But one theme throughout the discussion is that understanding your inputs is probably the most important thing. You cannot make decisions about whether you need to be more restrictive or what policies to assert over your inputs without understanding the functional requirements that made you have those inputs in the first place, and without understanding what they are.

Rory McCune: I would add that there is an element of treating any dependency you have as a cost. Every dependency I add is another thing I have to scan. Every SaaS service is another link in the chain. The longer my chain is, the more points there are for someone to attack. You cannot generally say just cut down, but you can recognize there is a cost to what you are doing, and you may only recognize it when one of those links breaks. With containers, when people ask how to make vulnerability scanning easy, my answer is: see if you can install fewer packages. A lot of container images bundle things they do not need. If you can reduce that, your job is easier because you have fewer things to focus on.

Rory McCune: There is a lot of activity around SBOM and signing. Do not feel that you have to be the master of all of that now. SBOM specifications are still moving their scope and adding security features. Keep a watching brief. Get started where you can, but realize things will change rapidly over the months and years to come.

Daniel Nurmi: We are not going to overnight achieve all the security things that we might add to an environment. But adding assurances and more checks is extremely helpful. The more layers of security we put in place, the less likely we are to be a target. Malicious actors look for environments with obvious weak spots. You become a target by not having these things. By transparently and publicly having security controls and threat models in place, you put yourself into the category of projects paying attention to security, and that makes you less of a target than others. Make it as explicit as you are comfortable with that you are doing these things.

Rani Osnat: At this point we will wrap this up. Thank you to everyone who joined us and asked questions, and to our VendorDome participants, Daniel Nurmi, CTO of Anchore, and Rory McCune from Aqua, Cloud Native Security Advocate. I am Rani Osnat, VP of Strategy at Aqua. It has been a pleasure having you. Thanks.

Daniel Nurmi: Thank you very much.

Rory McCune: Thanks, all.