Ax Sharma

01Opening and protestware

All right, I shall get started our master class on software supply chain integrity and security. I'm Ax Sharma, a security researcher at Sonatype who loves to blog incessantly and talk about supply chain security. So let me just dive deep into the latest news topics, right? Lately, what's been hot in supply chain security? Protestware has been a hot trend, right? Devs who are not afraid to make a point by using their open source software. So let's see what protestware is. Open source libraries, as you know, they're being used by hundreds of thousands of projects, right?

And what we are seeing lately, especially this year, developers behind popular open source packages, they self-sabotage these packages. To give you an example, colors and faker packages, right? It's used by millions. Thousands of projects use it. It's downloaded 21 million times weekly on npm alone, Colors is. I know for faker, it's more like three million weekly downloads. So this happened earlier this year. The developer behind these libraries, Marak Squires, he introduced an infinite loop in these libraries printing, "Liberty, liberty, liberty," right? And this was a protest against corporations who, according to the developer, corporations have been using open source for the longest time, but they haven't been funding it or giving back.

So, this is how it started, and the trend hasn't stopped here. Yeah. So once again, this developer chose to sabotage his own package and the intention was to introduce these infinite loops. And nobody noticed it right until people's applications started failing, and then this thread started on GitHub. "Hey, what's wrong? I'm using this library." So yeah, it didn't stop there. Now we see node-ipc package, right, which is again, very popular library. It's used by Vue.js. So many Vue developers saw their pipelines breaking, and this particular malware took this protestware saga a step further.

Not only did it print anti-war messages on the screen of developers or your applications, if it believes that you were a Belarusian or Russian user, it actually deleted your data. So it would replace all files on your system with a heart emoji. It would overwrite it. So, okay, colors and faker, it was protestware, but it didn't really damage data. It just froze your applications in protest, right? But this one actively deleted it, and for this little shenanigan, the developer was heavily criticized by the open source community, saying that moves like this could seriously impact the credibility of open source and the integrity.

And the trend hasn't stopped. It's almost like this motivated other developers to start protesting the things they believe in. So as of now, we've seen even event-source-polyfill, es5-ext, styled-components, all these, again, popular libraries. Many applications rely on these. They have anti-war messages in them, but this is more of a peaceful protest, I would say, because if you look at the code inside these, if it sees you're in a Russian city, it prints anti-war messages that put a stop to this war. It encourages users who are based in Russia to sign petitions, go to more reliable news sources.

It uses the URL shown as BBC's official Tor website. So yeah, the trend has started. We're still seeing some. You never know when the next protestware comes up. But this is what I wrote on BleepingComputer, and this is a point I want to stress. Current protestware may be centered around the war, but that may not always be the case. Developers, it seems like they have found a creative avenue that they can use open source to express their creativity, which is not just limited to delivering technical functionality anymore. They can use it to express their voice, to talk about matters that are of public interest.

So that's one theme we are seeing in open source land.

02Hijacks: real libraries tainted by attackers

Another is hijacks, when real libraries get tainted by bad guys. So many kinds of open source attacks, traditionally, it's been about typosquatting, somebody putting malware on npm, PyPI, using typosquats, brandjacking, dependency confusion, but I'm just trying to break these into newer themes. So when I say hijacks, I am referring to a real library getting hijacked. So I will come to that, but let me just give you a little crash course. Let me rewind a bit. What happened was, in October 2021, Sonatype's automated malware detection systems started flagging these three packages, which make no sense.

Look at their names: klow, klown, okhsa. I don't know what that means. Yeah. So our systems caught these packages. These packages imitated ua-parser-js, which is a very popular library. So you can see the name is klown, but the README file is pretending to be ua-parser. But if you looked at the code inside, it has a preinstall and postinstall scripts that run on both Windows and Linux. And these scripts have nothing to do with JS or JavaScript like it's saying. It's downloading a crypto miner, and it provides it with the attacker's wallet address, pool address, whatever, and it starts mining cryptocurrency on your system.

So that's what we caught initially, right? But eventually, news came out, the ua-parser-js package, which is a legitimate library used by big projects. Facebook uses it, Microsoft, Amazon, they all use it in their open source project. This library itself got hijacked. Now, the funny thing is that the code is just very identical. What we saw here, what we had caught, and what we saw in this, same tactics being used to mine cryptocurrency. And this one additionally dropped password-stealing DLLs on Windows. Yeah. If you're running Windows, it will try to steal your credentials from all these apps that I've shown you on the right side, which are very popular apps.

Smart FTP, Turbo FTP, Google Talk, messengers. Yeah. And JetBrains and Kotlin did disclose some impact, saying there, yeah. I'll say this quickly, Kotlin uses, I think, the Karma test framework. So if you were building test cases, and this ua-parser-js, the compromised version, got pulled in, then you were impacted by it. And it didn't stop here. This kind of malware, we kept seeing it in typosquats. So for example, Noblox is a very popular library. It's a Roblox game API wrapper. And its mirrors are called, say, noblox.js-proxied with a d. So we saw a fake package with an s.

Simple typosquatting. And this one it wasn't just malware, it was highly obfuscated, as you can see here, and it contained ransomware. So this was the first time we saw an open source package containing ransomware. Initially, we were inclined to dismiss this as just a prank because it came around Halloween, but later on, news followed that this actually has impacted this particular attacker. Maybe it's a script kiddie, but at least, yeah, younger users have been scammed by this attacker. The Register released a report on this. So we'll see. And then news came that coa and rc libraries were also hijacked, and again, very identical code.

It dropped the same password-stealing DLLs. So to me, it seems the attacker behind these attacks, ua-parser-js, coa, rc, and the initial packages we had caught, klown, whatever, it seems connected somehow. It's just somehow this attacker managed to compromise the npm accounts of the authors who publish these legitimate libraries so that attacker could push the code in these legit libraries. Yep. npm said that this happened because of compromised account, which has been disabled, and they're actively investigating. npm put further checks and safeguards in place to prevent future attacks like these. I think for top 100 or so projects now, if you're author of top X, top 100 projects, you will be required to secure your account with two-factor authentication.

So this is what I was saying. These attacks, at least when we saw ransomware, we were inclined to maybe dismiss it because look at the amount. It's asking you between $100 and $500. But a subsequent report by The Register said that, yeah, many more packages have come up, and the official developer of the real Noblox libraries developer says that this is a problem, that somebody keeps brandjacking his content to scam people. So that was interesting. There's a whole timeline you can look at in the slides later. And somebody, even the attacker actually taunted us on our blog over here like, "Nice try, you have to do better than that." So, yeah, this unfortunately keeps going.

Library hijacking. Yeah, so that's another recurring theme we are seeing, and hopefully, we don't see any more of it, but it looks like we will be seeing more of it. Another attack vector is repojacking and chainjacking, it's called.

03Repojacking, chainjacking, and fake “version 2” packages

This is not a new attack. It's been happening, but it kind of came to surface again last year. Yeah. As you know, Go packages, their names have the GitHub repository name in the repo itself. So what happens with, say, repo jacking or chain jacking is if you change your old GitHub username, your old repository links will redirect to the new links in your new account until the old username gets claimed by somebody else. So, this makes people suspicious, like what happens to Go packages then that get reclaimed by somebody else?

So yeah, GitHub did introduce a popular repository namespace retirement, which will put an end to this practice, but just something to be aware of because we have seen in our research, we've seen active examples of repositories, say, they were used by Intel at some point, and yeah, Intel changed their username or something, and the previous repo gets claimed by another attacker on GitHub. Okay, enough of that. Another kind of attack I want to go into is version 2, and it's a very clever typosquatting tactic. In these cases, what an attacker does is they'll try to publish a package which the name of the package itself appears like it's a newer version.

So in 2021, PyPI removed mitmproxy2 from their website, which is a fork of mitmproxy. Mitmproxy is, again, very popular. Interestingly, as you know, with open source, anybody can fork or clone anybody's package. That's the whole point of open source. But what we saw in mitmproxy2 is somebody had used this code but removed the security safeguards in it, so essentially, it made the package vulnerable. And as soon as PyPI removed mitmproxy2 package, another copycat emerged, mitmproxy-iframe, with the same problem. I'm not sure if the author had malicious intent, whosoever is copying or publishing these packages.

But surely, this did not sit well with the maintainer of mitmproxy. Yeah, he's like the more popular you get, the more people try to copy you and ruin your image, all that. So this was the problem. The mitmproxy2 package had these safety safeguards removed. It doesn't stop here. Just this year, this March, actually, Sonatype caught a lot of packages. They're called Colors2, Colors3, whatever. And again, look, to a newer developer, it may seem like this is the main problem, that Colors3 is the newer version of Colors. You see the attacker is trying to name the packages like it's the newer version of an official package itself.

If you pull these apart, some of them had code, even some had obfuscated code. It's got a bunch of Discord info stealers. Some others even open a reverse shell on your system. So definitely malicious. We reported it, but it's a whack-a-mole situation. You report a few packages, few more come up. So not sure how to resolve this problem unless you have some automation in place. Another kind is hidden malware.

04Hidden malware, Unicode tricks, and regression bugs

Many of you might be familiar with this Trojan Source made news that some vulnerabilities are invisible. This happens a lot with right-to-left, RTL, languages and character sets that, for example, to a naked eye, this looks like this part is commented out, and this is what's going to execute. If you're using an RTL language, your interpreter or your compiler may process it differently. So what looks like a comment to you may actually be code to the compiler and interpreter. And when Trojan Source became a thing, obviously, Sonatype also updated our malware detection systems to incorporate this technique.

So next time any foul play happens, we can catch it. So far, we haven't seen any big real-world attacks leveraging this because I think people quickly reacted to this before it could be widely exploited. Other kinds of attacks are, say, with homoglyphs or characters that look identical, but they're not. I may have given this example before, so I'm not going to ask, "Can you spot the invisible backdoor in here?" Because it'll take you a while. But I'll just show you it's over here, in this piece of code. These two things, what you're seeing here, it's not a space, it's an actual character.

And that alters the logic of the program because right now you would think with a comma, okay, it's just a single variable. Same thing here. You would think there's nothing here. That alters because now when you say const timeout, comma, this empty character equals req.query, whatever is in the response to this query, it gets assigned to both of these variables. And yeah, you can read more about this in the article I've linked to, but it's called a Hangul filler character, and this is one way to introduce a backdoor in your code.

Say I was to ship you this code, you may not think there's anything wrong with it, but then I send a GET request to your endpoint network health, question mark, this thing, whatever this, let's call it Hangul filler, equals my logic, and I might be able to execute code, depending on how your application is parsing it. Another example of a homoglyph, for those of you who are familiar with some basic coding, you'd know this is a not-equal-to sign. Exclamation equal, except that is not really the exclamation mark. It's a homoglyph for it.

It's called alveolar click. I don't know if I said that correctly, but that's what it's called. So actually, what will happen is the compiler will ignore the exclamation. It will actually set environment to prod as opposed to checking if it's not equal to prod. Big problem. Regression bugs that creep up because of partial fixes. Another supply chain problem. Saw this in Apache HTTPD server, and now we are seeing it with Struts 2 zero-day. So this made news, I think, last week or so, this new CVE, 2021-31805 in Struts 2 framework.

Again, Struts is a fairly popular Java framework used by apps. There was a two-year-old OGNL double evaluation flaw in it. So OGNL, it's a Java language for expressions and stuff, and the problem with it is, though, like with any language, you have to sanitize user input. So with OGNL injection, if you're not sanitizing your input, it will actually get processed by the application, so the user's input can influence what your application is doing. So it was thought that this particular CVE was fixed, but the fix was incomplete, and as such, we have this new CVE now, which was disclosed this month.

So make sure you upgrade your Struts versions again to the latest version, because don't forget Equifax hack of 2017, it occurred because of OGNL injection flaws.

05Dependency confusion

So dependency confusion. I can't believe this is still a thing because I'm going to give you an overview, but this all started, it's been well more than a year now, but this keeps happening. So dependency confusion is not a new form of attack, but it really blew up in 2021. What happened was, since July 2020, Sonatype's automated malware detection systems were catching suspicious packages. These packages were called, for example, env-paypal, something named after big companies. And on the surface, these packages showed that they said they were for security research purposes only. We really couldn't tell what they were trying to do.

They had very simple DNS lookup requests. And I saw the name of the guy who was publishing these, Alex Birsan. He had a pretty credible profile. So I reached out, I'm like, "Hey, what are you doing with this?" So, excuse me. We made sure this was all part of ethical research and that coordinated disclosures were pending with all 35 vendors or so that this guy was targeting with this ethical research. So in February 2021, when all of us broke news, I broke the news story, Alex also did together. Yeah. All these months, we had been catching these packages, keeping our clients safe by adding these to our data, but we couldn't disclose it because of the coordinated disclosure agreements in place.

So when we disclosed this news, though, I'm like, okay, great, now people are going to protect themselves against this. We are not going to see any more of this. But within 72 hours, I refresh my dashboard, 300 more packages come in. I'm like, "Alex, is this you?" He's like, "Nope, it's not me. I got my 160,000 in bug bounties already." So other bug bounty hunters who also want to get rich through this technique, and in some cases, attackers, they are now publishing packages, and you can see the figure keeps going up and up and up and up.

By March 15th, we are at 10,000 plus copycats. As of today, I'm not bragging when I say this, we have caught over 60,000 packages, including dependency confusion. A lot of them are dependency confusion. Most of these packages were for ethical research or basic stuff, but some did outright malicious things. We caught a few packages targeting Amazon, Zillow, Lyft, Slack. These touched your bash history. It exfiltrated your /etc/shadow, launched reverse shells, all kinds of shenanigans. This is important this month. I would like to happily say Sonatype prevented VMware dependency confusion attempt.

What happened was our automated systems, again, flagged this particular dependency, the API client bindings on PyPI, and it says proof by Codecov. So it looks like a proof of concept project. Now, when you Google it, what it's doing, apparently, it's a legitimate internal dependency used by VMware vSphere SDK developers. And the thing I found interesting was a lot of developers were asking about this online, on their GitHub forums, "Hey, where can I find this dependency?" Naturally, if a developer is seeing anybody, if something's on GitHub, your internal dependency name, attackers will try to capitalize on it.

Thankfully, in this case, it was an ethical researcher. VMware did give us a statement saying no user or product has been impacted, that this was ethical, and Sonatype had also reported this package to PyPI, as I said. So it was taken down, but if you're curious, this is what it did. Same thing we have seen with dependency confusion packages. It exfiltrates your IP address, hostname, your current working directory, just trying to fingerprint your environment to see if it's actually breached you. Just standard bug bounty procedure, I guess. So that's the company start getting all that information.

So yeah, this is an ongoing theme.

06Supply chain across the SDLC

So let's go into supply chain. When we say, what does it mean in the context of your SDLC? Now, this is an arbitrary chart because there's really no such-- I know we talk about upstream, downstream, midstream, but it's more colloquially created terms rather than having any concrete meaning to it. But let's see when we say supply chain, what it means. You have your development environment over here. You're fetching dependencies from public repositories, and NPM, PyPI, RubyGems, Maven Central, whatever. So let's just call this upstream. Where issues like in dependencies that can impact you.

And the thing is, anything that's published to these open source repos, they get mirrored everywhere, they get distributed everywhere. So upstream security is important in that context because whatever gets published there, good and bad, it'll get distributed. Midstream, I would say refers to, again, it's an arbitrary term here, but it refers to other aspects of your builds, package, if the application's running in a container, whatnot. And when we say downstream, you can say when vulnerabilities are exploited, those are downstream issues. Like I said, the Apache Struts2 CVE, downstream issue. But I will say anything that starts upstream, whether a vulnerability or malware, that's bound to be distributed downstream to the customer.

That's why I said it's a very made-up chart. So why would developers target open source components? Why are we seeing this theme growing? Because, well, people are investing in open source. Microsoft has acquired GitHub for lots of money, and GitHub has over 100 million repos. Things that get published to GitHub, developers may further choose to mirror it, to fork it. Some of these get published, ASAP to NPM, all these repositories, NuGet, RubyGems. So as a developer, if I can corrupt the upstream, my chances are increased to infiltrate users who are downstream.

Any ecosystem that's open to the public, it's also open to adversaries. So the barriers to entry are super low, and the returns are tenfold if I can successfully execute my attack. So, I need not give you real-world examples. There was SolarWinds, there was Codecov. At this point, it's become pretty known. At that time, it was like, okay, what just happened? Because this attack lasted two months. Thousands of people used Codecov, and this was, again, I would call this more a midstream issue because the credentials to the bash script were obtained from, say, Docker image creation process.

So I would say this was an example of midstream attack, a flaw in your DevOps or CI/CD tooling rather than an upstream dependency. And Codecov, don't think it's a small thing. A US investigator said hundreds of networks were reportedly breached because of this. This is the whole timeline. It's old news at this point, but I thought I'd mention it still because some victims, especially Mercari, they actually were impacted by this. 17,000 records, which were their customer records with bank code, branch code, account number, all that got exposed when their private repository got hacked.

But if I'm honest, I think Mercari is at least partially to blame here, because what were your customer records doing in a GitHub repo, even if it's a private repo? Was this a sample set? Because I've seen this theme time and time again. Companies get breached, and they're like, "Oh, oops, it was in our private GitHub repo." But why was real customer data in your staging environment? I think that's also a problem. So malware infiltrating OSS repos. At this point, this has become a pretty common theme.

07Malware volume and “This Week in Malware”

Last year or so, we reported on Twilio's copycats and typosquats launching reverse shells. So, as a developer, you may think twilio-npm is a legit package, but it's an ngrok tunnel that's being launched on your system. And see, attackers or bug bounty hunters, whosoever is publishing these packages, they try to go after packages that have a pretty popular following. So the original Twilio package gets 40 million downloads. To lifetime, it's gotten those many. So yeah, if I copy and I call mine twilio-npm, sneak in some funny code in it, my chances of infecting you increase.

Same with Discord malware. I've actually seen so much Discord malware at this point, I'm bored of it, and even this week, we're seeing more of it. So I did a piece on this, that attackers are targeting, I feel like younger developers and gamers by publishing malware that steals your Discord, that steals your Roblox tokens, all that stuff. So this is what I was saying. Sonatype's malware detection. As of today, we have discovered over 60,000 packages. Many of them are dependency confusion, PoC research, but many are not. They are malicious. Out of these, we have disclosed over 14,300 packages in our blog posts that you can check out on blog.sonatype.com or dev.sonatype.com.

So one thing I want to say is we've started a new digest, I think as of last month, "This Week in Malware." So every Friday we do a video and a news digest just summarizing our findings, because we needed a better way. There are so many packages coming up every week. I cannot blog about 500 individual packages. So we just started condensing this material to what's relevant and what's informative. So just recently we caught 400 in one instance and 500 packages in another instance targeting different developers. So do check these out.

08Prevention, SBOMs, regulation, and integrity checks

Now we come to what can we do about all these attacks? Well, first thing is know what's inside your code. Have an SBOM. When I say SBOM, I'm not just saying have a list of things in your code, because with dependency confusion, as we saw, the names of your private dependencies and the attacker's public dependencies, they're the same. So your SBOM should be comprehensive. It should have the checksums and all that. Follow a specific standard. There's SPDX, there's CycloneDX, there are many standards, so make sure you have a proper SBOM.

Apply latest updates and patches. That's sound advice, but I want to say be careful with that, because as you can see in supply chain attacks, like SolarWinds, like with Faker and Colors or all the protestware I showed you earlier, it was the latest updates that caused the problem and malicious behavior. And in some cases, the solution was actually to downgrade, not upgrade. So yes, apply latest updates, but not until you verify their integrity, for which you may need automation, because it's hard to manually track dependencies. So some kind of automation, deep binary analysis that can spot malicious code early on tremendously helps.

And I want to say SBOMs are not just a fancy term anymore. It's the law. If you're aware of 2021, the executive order on improving the nation's cybersecurity, and this came after the Colonial Pipeline ransomware attack. It sucks that these things are always enforced until after something big happens. So as you can see, this requires government purchasers to provide an SBOM directly or by publishing it on a public website. So that's one of the aspects of law, and I think UK government also followed lead. They at least asked what should we do about the problem.

I'm not sure if they enforced the law. I haven't seen any updates as of today, but they at least saw commentary, how can we strengthen our supply chain security? And then I think Log4j crisis further escalated this, that White House is now calling OSS a national security issue. So it hasn't stopped. And now just very recently we saw Spring4Shell, which I think may be not as bad as Log4j, but it caused some action on the InfoSec side. And yeah, for Log4Shell, this was interesting. I think it was the first time that FTC basically said to private companies that remediate Log4j or risk getting sued.

Because government has put executive orders mostly for federal agencies and stuff, but this was an interesting development that FTC warned businesses to catch up.

09Obfuscation, containers, and Sonatype tooling

So how to prevent supply chain attacks? Just back on the topic. You need tools that can dig through obfuscated and minified code. I'm not joking. It's getting worse by the day. Some of the examples I showed you, the code was obfuscated or minified. Some use variable expansion, and I hope you can see my screen when I do this, but this new kind of code we have seen, this is how it looks like, if you can see. Just like this. Couple of dollar signs, spaces. That's the whole code. And this particular code was actually It's a very good story.

A tool on GitHub, which was saying to Windows 11 users that you can use this to add Google Play Store to your Windows 11. It's called PowerShell, Windows Toolbox. It turned out it was malware. That, yes, it let you do those things, but then it downloaded a suspicious script from this particular address, and yeah, that's what it is. Very sophisticated obfuscation. So when we talk about tools, your tools have to be able to dig through these newer techniques and tactics that attackers are using. And you may need some human elements to this as well.

Because we've seen steganography code hiding in audio files. Images, and those are not images. They have executable code in it. So I think both aspects are important. As far as container security goes, well, more and more companies are deploying cloud native environments, and I'm basically saying this container security bit because of Codecov incident. Because it was using Docker images and stuff. So when you're talking about containers, like in this case, in Codecov's case, credentials were exposed. But automation is important, proactive detection is important. All that is fine. But with containers, I will say that something that can catch suspicious traffic.

And not just about patching vulnerabilities in your container environments, but say, even if an attacker could breach your container, but say a layer seven firewall could block the further connections, I think that also stops malicious behaviors. So Nexus Intelligence, we have proactive malware detection bots that run on basis of over 60 flags, actually, red flags. That's how we identify malicious components, and we are constantly improving the algorithms, when it comes to malware detection. And as far as container security goes, again, what I was saying, you should probably just check out Nexus Container.

It's easier than me trying to explain it. But if you're interested in full spectrum of software supply chain solutions, Sonatype's got a solution for everything. There's Sonatype Lift, which is free of cost. It connects to your GitHub repository. It'll analyze your commits in real time. As you're typing, if a piece of code causes memory leak or it introduces a race condition, it'll flag that for you. It can integrate with, I think Semgrep and FindSecBugs, it's called, to help with secrets management, so you don't accidentally commit secrets in your repos. This Nexus Lifecycle with premium data related to thousands of vulnerabilities that may not even have a CVE, but we already have it in our data under Sonatype IDs.

So that's Lifecycle. I already talked about Container, but again, it has a layer seven firewall. It makes sure your containers are free of vulnerabilities, et cetera. Nexus Repo, again, it helps you manage your libraries, build artifacts. It's like a cataloging system for your components. And my favorite remains Nexus Firewall, because all these, the protestware we are seeing, the sabotages we are seeing, malware, typosquats, ransomware, I have faith in Firewall. It will keep preventing these incidents by automation.

10Closing data and fast tracks

Now time for some boring statistics. No, pardon me. So we have, as of today, over 10 million unique vulnerabilities in our data. And what I was saying before, 1.4 million are Sonatype IDs. So for these, there are no CVEs. Or there weren't any CVEs when we caught them. We aim for 12-hour fast tracks. What that means is, for example, with Spring4Shell, a rumor started on the internet. Is there a vulnerability? Is this not? I was tracking this in real time. I called up the teams. We were all tweeting about it.

We were keeping eyes on this. And this 12-hour figure has actually gone down to maybe two to six hours. Because as this information evolved, we kept putting this information in our data, keeping our customers up to date. So that's what fast track means. Until a deep dive analysis, a full-on analysis is pending, we try to get something in our data with vulnerabilities that are still being disclosed under exceptional circumstances. So make sure you check out all our products, and if you have any questions, please let myself or anybody else know.

Thanks again for joining.