Presenter (Mike, Sonatype)

This is billed as DJ's talk, but I have given this a whole bunch. I was not supposed to be here, but now I am here, so I am going to do the first part. When we come out of the demo, we will bring DJ on as we open it up to more discussion. He is much smarter than I am; I am just really good at this particular presentation.

I want to start with a reminder: this has been on the OWASP Top 10 for over six years. When I first joined Sonatype, we used to talk about this a lot. Before that I was at a couple of Fortune 100 insurance companies as a practitioner doing build and release management and starting DevOps initiatives, so I have walked this path.

How many people in the room identify as security professionals? How many identify as developers? The rest are QA, management, or another function. In our most recent State of the DevSecOps Community report, with input from over 5,000 people, 62% of organizations still do not have an open-source governance policy. This year I am asking everyone I meet: do you have one, and if not, why not? In 2019, 24% self-identified that they thought they had been breached through a known open-source component, up 70% from 2014, when about 14% said the same thing.

Your applications are made up of 80% of these things. Security people may focus on code developers wrote, static code analysis, and dynamic testing, but the apps developers produce are mostly made of borrowed components. We are not obviously taking the time to identify what is in the apps and keep the information at our fingertips. OWASP A9 says: if it is known vulnerable, stop using it.

The DevOps movement has helped: 47% of respondents said they deploy multiple times a week. If you can deploy multiple times a week, you have a chance at the problem I am about to present. If you cannot, you need to speed up just to begin to address it properly.

Back to open-source governance. We asked people to self-identify whether they had mature practices. Only 62% of the self-identified mature practices had an open-source governance policy. My retort is: you are not as mature as you think if you do not have that one basic concept. They are probably doing 95 of the 100 things they need to do, so they think of themselves as mature.

The attack vector is going up, and it is no surprise when over half of organizations are not even looking at the information. Components flow into applications and then sit in production facing the web. A9 also encompasses several other OWASP Top 10 attack vectors: you can inherit cross-site scripting, remote code execution, and other vulnerabilities through that one subcategory.

The bad guys are doing DevOps better than we are. When there is a responsible disclosure, they can turn around and exploit within three days. Many of you cannot release every three days; they can. The NSA has reported seeing weaponization within 24 hours. We have to automate at least as fast as attackers can weaponize, because there may be three days from announcement to effective attack.

The volume is overwhelming. In 2018 there were 15,000 new open-source releases per day; in 2019 that went up to 21,000. The average organization consumes over 300,000 open-source components a year. Sonatype sees Maven Central download reports and talks to npm, NuGet, and other package ecosystems; in the State of the Software Supply Chain report, the average was 313,000. Most companies are using about 8,200 component versions, or 2,778 individual components, roughly three versions per component. These averages are lies, but they show the order of magnitude.

If you had an open-source governance policy, in theory you would need to vet 8,200 components a year. Who has an AppSec team of 15? Nobody. The largest team in the room is 10 people supporting 200 developers, which is an unusually good ratio. Usually I hear one AppSec person for every 1,000 developers. If you do not have a team of 15, you cannot manually process the flow of open-source components into your organization. It is physically impossible. This is why we turn to automation.

Susan Moore wrote a 2017 post saying that by 2020, 99% of vulnerable exploits would be known by security at the time: we would know the attack vector and still watch it happen because we could not stop it or act fast enough. I think that is hyperbolic, but interesting. Given the volume and variety of components, the rise in attacks, and the lack of formal policy even with a manual process, unless we change, that pessimistic future is plausible.

Now I am going to dive into the hands-on part. This demo is not totally contrived. Shortly after the Equifax breach in March 2017, we found the REST showcase project, forked it, and put that sample project into a Tomcat container. That is real world in the sense that developers often take something like Struts, a reference example, or a prior project, use it as a base, build around it, and deploy it through Tomcat. The Struts 2 RCE and Sonatype Nexus community project is on GitHub.

This is Sonatype IQ Server integrated into an IDE. We do this to put information directly into the hands of the people best positioned to solve the problem: developers at the time of coding. When we talk about shift left, this is the moment when we can get it right the first time and not send a defect downstream. That is the lean concept: just do it right. Jez Humble and Gene Kim have talked about this, and it is an Edward Deming concept.

Our theory at Sonatype was that people continue to use known vulnerable components because they are simply unaware. Developers have components, but nobody gives them the information they need. If the organization has a manual ticketing process, developers quickly stop using it because approval can take weeks and does not fit a sprint. They grab the component anyway, decide forgiveness is easier than permission, and move on. If we put information in their hands at that moment, we have an opportunity to deal with it.

The app has a vulnerable Struts Core component. The IDE shows other available versions, with colored bars representing whether there is a policy violation. We codify policies as rules, and Sonatype provides the data. In developer training, we teach people to find a green bar with no red under it. A big green bar means it is probably in an API-compatible version range. The goal of risk is not zero; it is to get risk down and to quantify it.

The demo builds the Maven project, with one governance-related switch skipped because this talk is not about licensing. Then it builds a container image called HackMe and runs it. The application comes up with three orders. The attack vector uses a specific order URL. The exploit is a Python package in the GitHub repository, and it lets us execute a command in the container. If it were not a container, it would be your VM or bare metal server. The command shows we are in the Tomcat folder, proving remote code execution.

This does not take a nation state. People imagine hacking as hard and requiring exceptional skill, but a teenager who finds the right dark-web attack kit can weaponize it. I do not know how to do this, and I am doing it. From here the attacker enumerates: who am I, what can I do, what access have I been given? The byproduct of doing nothing to the Tomcat container is that it runs as root. Runtime tools such as Twistlock or Aqua Security can identify that, but without them, it is not obvious. Now someone like DJ would be salivating: enumerate, poke around the system, find what it talks to, find data, or download a cryptocurrency miner.

A component vulnerability can be chained into other vulnerabilities. If this is running inside an old Kubernetes cluster, root in the container may let the attacker escape into the underlying host and pivot into the rest of the network. The demo cats the password file, and the room suggests looking for keys, Amazon keys, or shadow hashes. With remote code execution, the attacker is limited only by imagination.

After killing the container, we look at the data. The app was on Struts 2.5.10, and the goal is to get to at least 2.5.17 or higher. In Maven, if a Struts project is set up properly, you change the version in one place and it affects every Struts component through the parent dependency. If teams update some Struts components and not others, they create another set of problems. Changing the version to 2.5.20 updates dependencies, loads them on the classpath, rescans, and reevaluates.

The local IQ Server has to hit Sonatype cloud data services. Sonatype is not a giant on-prem database to maintain; the data is constantly updated in the cloud. There is an automatic vulnerability detection system in front of a dedicated 70-person security research team.

For the offending Struts Core issue, the write-up shows the data behind the visual red or orange square. At the front line, the information is visual and actionable: choose a new version. Deeper in the report, there is the National Vulnerability Database write-up and then Sonatype's explanation. The research team writes so both developers and AppSec professionals can understand and participate. Those worlds do not always overlap, so writing for both is a challenge.

The report includes recommendations, workarounds, research sources, the exploit package link, scoring, and attack vectors. Sonatype's researchers rescore issues using the vectors themselves. Often they score higher than the National Vulnerability Database; sometimes lower. The point is to provide detail and precision that lets automation work. If you automate against data that is mostly right, you get garbage in, garbage out and alert fatigue.

AppSec professionals are used to tradeoffs from static analysis: how many rules to run, how deep to go, how many false positives and how long the scan takes, and whether fewer rules might miss problems. But open-source component analysis does not have to follow that pattern. Components are hashed. Sonatype does class-level analysis and advanced fingerprinting. If the component matches, the system can confidently alert and feed the information to the developer and the deployment pipeline.

With one Struts version change, the demo removes a large amount of risk and eliminates the exploitable vulnerability. The app still has a few issues: for example, there may be no good version of one REST API, or there may be a workaround. Jackson Databind had recently put out its first non-vulnerable release after a long back-and-forth. The key is that the original remote code execution issue is gone.

The fixed image is rebuilt and re-containerized. Open-source scanning is fast. Static analysis should not be used to scan all open-source components; that jams the wrong tool into the wrong job and is not moving at DevOps speed. Rerunning the attack against the fixed app returns HTML instead of popping a shell. The demo proves the vulnerability was real and easy to exploit, and proves that changing a version number and rebuilding removes it.

An audience member asks whether patching can be automated. The answer is: we are working on it. A zero-touch upgrade is scary, but so is pushing production code with a zero-day vulnerability. If you deploy properly, you can roll back. Upgrade now versus upgrade later is an organizational risk decision.

At Sonatype, we describe this as a software supply chain, no different from the physical world. Deming's work in Japan, Toyota, Mitsubishi, and others emphasized building quality in. The irony is that we think writing tests and doing extra work will slow us down because we are doing more things, but in reality it increases productivity by removing rework. If the developer fixes the component before it goes downstream, security finds no issue, no ticket is created, and the developer keeps writing features, tests, or experiments.

Components come through the front door. A product like Firewall can inspect them as they arrive and stop bad ones immediately. That matters because if a developer builds a proof of concept or minimum viable product around an unacceptable component and only learns at the end, the team may have to change frameworks or do substantial rework. For security, knowing is half the battle. Stopping it at the gate and knowing what is coming in can save major manual effort. The math here was 15 person-years of work, based on four hours of research per component issue. Sonatype's 70 researchers aim for two issues per researcher per day and improve continuously toward that goal.

Even if the milk was good when you bought it, it ages. Software ages like milk. You have to keep asking whether those 8,200 components are still good. Sonatype Lifecycle is baked into CI and delivery, so as software moves it keeps being scanned against data of the moment. Since the talk began, the hosted data service probably gained new vulnerabilities. The system can automatically break a deployment process. Sonatype does not recommend breaking builds because that makes people angry; stopping something before it gets beyond QA, as an exit or entrance criterion, is a more appropriate place.

The hard part is vulnerability data. This kind of data does not readily exist. Sonatype had to build a team and become a data feed; the company could not buy a good enough feed. The National Vulnerability Database is inadequate for this use case. Sonatype scrapes places where open-source developers discuss development: Git commits, issue trackers, dev forums, and other places. Machine learning and AI processing help discover issues. The missing step is that open-source developers often do not self-report to the National Vulnerability Database.

Because of that, Sonatype knows about about 1.3 million vulnerabilities not in the National Vulnerability Database. The NVD only recently added its 100,000th CVE, and not all of those are open source. Benchmarking OWASP Dependency-Check, which relies on the NVD, found that using only the NVD might get one in six valid findings. The NVD maps CVEs to CPEs, which roll up to names like Apache POI or Struts, but not to the specific component, class, or file that contains the offending code. That creates false positives and false negatives.

Q&A

An audience member asks whether scanning Log4j repeatedly across many teams creates overhead. The answer is that Sonatype scans are fast, so scan time is not the concern. The bigger concern is incident response: when there is a zero-day, can the organization immediately say which applications use which versions of Log4j? Everyone may use Log4j, but no two teams necessarily use the same version. That inventory is mean time to detection. Then the organization can work with teams to fix in the next release, ideally in less than three days for the nastiest issues.

Another point is that developers pin component versions. CocoaPods, Gemfile.lock, and similar files lock a version, so it does not automatically become latest. Scans belong on the developer machine while people type, in the build server when automation runs, and in repositories over time. If code is checked in with pinned versions and later deployed while bypassing controls, it can carry old vulnerabilities. Long release cycles make this worse: if an app releases once a quarter or once a year, an upfront scan may be months old by deployment. Scanning repository code at least weekly helps teams get ahead of that information.

An audience member asks whether "repository" means Nexus. The answer is source control, such as GitHub, because people check code in and forget it. There is also a team-lifecycle problem: a project gets created by a team with a fixed budget, then the team is removed while the product remains deployed. Nobody is there to fix it. Knowing the vulnerable component exists helps prioritize and prevents security from simply throwing the problem back over the wall.

Another audience member asks whether Nexus can show what is in it. The answer is yes. The demo did not show the full platform, but Nexus Firewall and even the free repository health check can scan repository contents. In one setup, a Docker proxy recorded anything that came down; redirects sent hub.docker.io traffic to the repository. Repository health check provides a cursory public-data scan; Firewall gives a more detailed report. A repository report can look like an application report, and it exposes the SBOM, the software bill of materials: versions, components, and subcomponents.

A question asks what scanners Sonatype uses and whether it scans manifests or containers. The answer is that Sonatype prefers to scan binaries and scan as-is. A manifest is a shopping list; it may say romaine lettuce, but you do not know which lettuce is in the cart until you inspect the cart. Even if the manifest names a version, that does not prove the artifact is genuine. Sonatype wants to identify the actual artifact in the app. But scanning technology varies by ecosystem: JavaScript may require parsing JSON files; Python may require a requirements file or pip freeze output because binaries are not the same.

Another question asks about the OS and base image. The answer is that it is both. Application libraries, container base images, OS packages, and runtime configuration need different tool types. The speaker names Anchore or Clair for OS-level package interrogation, Sonatype for the application stack, and Twistlock or Aqua Security for Docker benchmark/configuration and runtime analysis. These tools can see whether something runs as root, who it talks to, and support network segmentation. For containers, those tool types together answer the full range of questions.

DJ adds that when looking at containers, teams also have to look at the host, runtime, and tools. In a previous security-architect role, vendor images entered production pipelines and had more vulnerabilities than internally created code. That means teams must scan not just code they make, but code vendors make and code that assembles pipelines. Otherwise, they are bringing in a Trojan horse. Tools such as Twistlock, Anchore, or Falco/Sysdig can help with host detection. The pattern is to ingest, verify, validate, and then publish a blessed artifact for use.

A final question asks about Docker images. The answer is that Docker itself is becoming a generic term, like Kleenex. Runtimes and clients are changing: kpack, Podman, and others. The hope is that if everyone stays within the OCI standard, scanning should still work, but Sonatype is actively testing that and does not want to promise support for every case yet.

The talk closes with a mention of All Day DevOps, an online virtual event on the 6th. The point is that after the conference, the recordings will be on YouTube and useful for sharing inside organizations. The talks are practitioner-led, with no vendor pitches: people living the work, sharing successes and failures.