Stephen Magill, Hope Lynch, Sacha Labourey, and Brian Fox

Stephen Magill: Hi, and welcome to the Sonatype and CloudBees VendorDome. I am Stephen Magill, VP of Product Innovation at Sonatype, and I am here with Hope Lynch, Senior Director of Platform at CloudBees. We will be moderating this session, which also includes Sacha Labourey, Chief Strategy Officer at CloudBees, and Brian Fox, CTO at Sonatype. What we are going to be talking about today is being a developer in the age of cybercrime.

Stephen Magill: The incidence of cybercrime has been increasing year over year. We have seen increases in familiar attack types, like ransomware and data breaches, and we have also seen new attack types emerge. Several of these new attack types target the software development practice itself, either the tooling that we use in development or the software supply chain. This has increased the set of risks that developers have to consider as they set up pipelines, write code, and deploy applications. What was already a pretty big security burden for developers has been growing and becoming more complex. If questions come to mind, paste them in the Track Four Slack channel and we will bring them up to the panelists.

Stephen Magill: Let us start with introductions. Sacha?

Sacha Labourey: Hello, everybody. My name is Sacha Labourey. I am joining you from Switzerland. I am the co-founder of CloudBees. I was CEO for a decade, and now I am Chief Strategy Officer. You might know CloudBees as the enterprise Jenkins company. We have done that, we are doing that, but we are doing a lot more as well. We are delivering an end-to-end solution for software delivery. We are focused on helping enterprises, which means we want to simplify the life of organizations while also recognizing their specificities and the complexity they typically, quote-unquote, enjoy with lots of different tools and processes.

Stephen Magill: Brian?

Brian Fox: Hi, I am Brian Fox. I am co-founder and CTO here at Sonatype. My background is in software development. Early on I was working on Apache Maven, and I am still involved in Apache and those kinds of projects. At Sonatype, we have been on a journey from build tools around Maven to helping people modernize their supply chains and really manage the dependencies going into them and the code quality of the custom code going into them. For probably the last four or five years, I have spent a lot of time evangelizing focus on the supply chain and some of the novel attacks going on there. It is an interesting topic, and one not enough people are paying attention to.

Stephen Magill: Hope, why don't you tell us what Senior Director of Platform is all about?

Hope Lynch: Senior Director of Platform is about the story of the unified platform. What does that mean? How do we connect with our customers and make sure we are helping them get what they need so they can seize their opportunities? I have been in technology for many years. I have spent time in the server room, as a developer, in agile work transformation, you name it. This topic is near and dear to my heart as well.

Stephen Magill: I am VP of Product Innovation at Sonatype. I was CEO and co-founder of MuseDev, which was acquired by Sonatype in March. I focus on products in the code quality space and on delivering code quality results to developers and bringing more insight into the development process.

Expanding Developer Responsibilities

Stephen Magill: I wanted to start with the expansion of the developer domain. Developer responsibilities have been growing. That started with ops, with the DevOps movement and developers taking ownership of many operational aspects. Then it advanced into security. Secure development practices are now expected to be part of everyday work. There are also new types of attacks that developers are facing and have to be aware of. Humans only have so much attention and capacity for productive work. How do you split your day and your attention among all of these concerns? What practices and tools can developers adopt to cope with this huge array of responsibilities?

Sacha Labourey: Obviously, you need tools. You are not going to guess what goes wrong without tools, so tools are always a great support. But there is no magical tool that solves every problem. Especially in bigger organizations, you also have history. One problem organizations face is not that they have no tool, but that they have too many tools. You find differences between tools, overlap among tools, and you risk getting a plethora of information on top of the false positives we always enjoy. You get true positives too, but multiplied by maybe five tools. All of this is shifted to developers, so they get that tsunami of feedback.

Sacha Labourey: Sometimes developers feel like they are being mushrooms: kept in the dark, then the box opens, someone throws a list of false and non-false positives at them, closes it, and says go deal with it. Another topic is how you categorize that work. In some cases it is more important than features and urgent to fix. In other cases it is perceived as technical debt, so people will get to it when they get to it. The classification is handled very differently in organizations. If you got hacked in a big way, you are more likely to consider it a high priority than if you never got hacked, or never know you got hacked.

Hope Lynch: Sometimes tools just add more to your plate, and you have to think about the value you are getting and whether it is worth it on balance. Brian, how does this play out with respect to the supply chain?

Brian Fox: Part of the challenge is finding tools that work with the style of modern development and are not just bolted on. The way we develop in a continuous-everything world, a DevOps and DevSecOps world, is different than it was not that long ago. Tools designed just to produce findings and throw them at developers, or throw them into the mushroom box, do not really work. You can apply more pressure and try to force developers to do something about it, but that is not the right way to solve the problem. You need tools built natively to work in that modern environment, intended to be part of the flow that is going on.

Brian Fox: The second part is that you need to empower developers with visibility to see these things, and organizationally you need to care about them. If the only thing you care about is shipping code on time, then you are not caring about security too. You need to say, we have to ship secure, good quality code on time, and enable development to do that and measure it. You get what you measure at the end of the day. If you bolt security on at the end of the process, you should not be surprised when you do not get very good results.

Hope Lynch: Security is sometimes not treated as a first-class citizen of DevOps. You have a separate security team and a DevOps team, and sometimes they talk and sometimes they do not. What would an ideal relationship between security organizations and DevOps teams look like? We still need separate security teams for some things.

Stephen Magill: I am glad quality was mentioned. Not having time to attend to security alongside future development is not so different from not having time to address tech debt or handle architecture mindfully. When you look at practices and tools that can help, it is good to think beyond security. The practices that promote low tech debt and code quality are the same ones that help create a process for attending to security. It is about all of those non-functional requirements. Features have to get out, but there are other important things besides functionality.

Sacha Labourey: Empathy for customers, maybe more than empathy, emotion, is important. If you have never been hacked, you may behave differently after you get hacked. There is an emotional learning process that burns your brain in some way. I am not suggesting everyone try to be hacked once, but building the perception of what it means and what impact it has is important. Part of learning is hearing from companies and engineers who went through that and can say, we thought we were invincible, or we did not care because why us, and then we got caught by surprise and here is what it meant. There is trauma with it, and a good way to create empathy is to talk about that trauma and understand that you do not want to get close to that place. Secure software is like being a firefighter: you are interesting only when something bad happens, and the rest of the time people ask why they are paying you. Changing that has a lot to do with emotion and understanding what it means to be on the wrong side.

Stephen Magill: Nothing focuses like a failure. Dedicated security teams will always have a role. Tools do not replace a security team, but they may change what that team can focus on. How might the role of security teams shift with modern tooling?

Brian Fox: Joe asked in Slack: if we integrate tools into development that put the brakes on problems, we get pushback about why we did not ship on time. If the organization does not value that, you have misalignment. Deepesh followed up that false positives broke them down. Precise data is key. False positives become a broken smoke alarm. But it is not only that simple. Security and everyone else need agreement about what is important and how important it is.

Brian Fox: You cannot block builds on everything. No application we have ever analyzed started on day one with no problems. You need to think about the solution in ways that include security risk, quality risk, and legal risk, because developers have to deal with all of these things. You need to prioritize them and set different violation types at different levels, so level 10 AGPL and level 10 security problems are showstoppers, while other things are warnings. As you eliminate those, you can bring the bar down and deal with fives and above. It requires precise data and contextual policies. The policy for one application may not apply to all the others. Code that is a service is different from code that is shipped, and something buried in a bunker is different from a web-facing application. Findings presented to developers have to be accurate and relevant to their context. Otherwise, they are easy to rationalize away. If you block on everything, the tool will get unhooked and ignored.

Sacha Labourey: Tools can help define policy without necessarily being the cop. If everybody agrees on the policy, it can become more blameless. We are in a culture where the pipeline drives a lot of behavior, so we think about gates. But this is about overall risk assessment. Some things need to be go/no-go, like AGPL or an S3 bucket open to everybody. But the rest of the time it is more important to have a brain looking at everything you are doing all the time and giving an assessment: this is good, this is not good, this is actually good, and here is how to prioritize your work. Not everything is an emergency and not everything needs to block your flow.

Stephen Magill: There is flexibility and leverage in targeting different points in the development process for different warnings. You can report some things in the IDE, have pre-commit hooks for others, have CI fail on certain errors, and monitor other things in production. Surfacing the right things at the right points broadens coverage.

Brian Fox: Many people from security say, I just want to break the build. But is that really what you mean? If it is a vulnerable dependency, dozens or hundreds of developers may need that dependency to keep working on their branches and run tests, and you block all of them while one or two people make the update. Do you want 90 percent of your team idle while one person fixes it? What you really mean is that you want them to know about it during development so they do not ignore it. Maybe in some cases you want to block the release because they ignored it. It is a carrot and a stick. Tools that are native to this environment consider these things. Tools that look at it in a stovepiped way do not consider what actually happens if you fail everything all the time. It is not going to work.

Sacha Labourey: Developers have been exposed to ops, and security is not just code security. We are talking about code and dependencies, but there is data, runtime environments, identity, infrastructure as code, SaaS data residency, and other risks. If a company is European, maybe certain data cannot leave an EU data center. All of those things participate in risk assessment, and in the end they go back to developers in some way. It is getting wide. It is not just source code.

Regulation, SBOMs, and Dependency Visibility

Stephen Magill: Another topic is government regulation. The cybersecurity executive order came out a few months ago and asked agencies to develop minimal standards for things like software bills of materials and software testing. What role might those play? How does that guidance fit current industry practice? What would your recommended minimal standards be?

Brian Fox: I think it is a necessary shove in the right direction. The federal government buys a lot of software. If you include the turtles all the way down, the people selling to people who sell the software are ultimately going to be asked for a bill of materials. It has an amplifying effect and is causing parts of the industry to care about bills of materials where they did not previously. Component-centric ecosystems like Maven, Java, JavaScript, and NuGet naturally thought about this because how they assembled software was front and center. In embedded systems and C and C++, while there was reuse of code, the mentality was often more about reusing source code than modules, so thinking about a bill of materials was less front and center. Some of those ecosystems are now asking these questions. It will help move the needle. The standards wars are already on with several different standards and no real consensus, so it will be challenging, but it has fired the gun to get started.

Sacha Labourey: We are facing a radical shift in how we perceive software, much like physical assets. Armies defend countries against physical threats, but in many countries there is nobody to defend organizations or companies from cyber attacks. In Switzerland, we acquired F-35s and trucks, but have zero cybersecurity units in the country. It is going to take a while to realize that what used to be a physical threat handled at a country level through police or the army now needs to expand to include cybercrime. In many countries we are far from even thinking about those problems.

Hope Lynch: Joe asks how to balance speed from the open source ecosystem, with Node, NuGet, and so on, against the new requirements to understand dependencies and the risks from them. You have wonderful tools you bring in. How are you monitoring all of it and making sure you stay secure?

Brian Fox: I feel like that is what we have been doing at Sonatype for more than 10 years. You need precise data and contextual policies. You need tools that give visibility all the way down the dependency stack, even if the manifest is not correct. We see cases where things get into applications that were not listed in the POM or package.json because they were included by copying and pasting. Also, if someone inserts malware into your supply chain, your manifest does not say, give me the hacked version of this component, and yet that is what you got. The analogy I use is that it is like judging a recipe for a cake as safe to eat because it does not say there is poison in it. The cake might still have poison in it. How do you know? You need tools that can actually look at that in addition to what the manifest says is there. Once you have an accurate bill of materials, you can manage violations to your policy and put gates in place.

Sacha Labourey: CloudBees works with companies like Sonatype on dependency management and recently launched a compliance tool. The idea is to look at many aspects, including inputs from tools like Sonatype's, but also what happens post-production once you deliver. How do you scan that the right keys were used, that containers are not still two months old and running in production, and so on? Leaving the pure pipeline view is important. The pipeline is an amazing step, but you also need meta-level analysis all the time. Solutions based on OPA, Open Policy Agent, can scan a wide array of constraints and feed that information back to teams. The challenge is balancing so you do not add five times more work to developers, but filter, categorize, and prioritize things so people know what is urgent and what is less urgent.

Training, Culture, and Simpler Defaults

Stephen Magill: Another Slack question is on education. A lot is expected of developers in security, but there is not as much training on secure development practices. What could be effective?

Sacha Labourey: We do not need less training in anything, but it is about balance. If you think about compliance in a wider sense, there are things that may be not allowed, for example in a bank, that are not necessarily a security issue from a coding standpoint. You might write perfectly okay code, but the bank does not want those things done. Compliance rules constantly evolve. Teams get burned out because compliance teams constantly retrain them: now you cannot do that, now you can do this because we fixed another problem. It becomes a constant stream of noise. Tools should not just say, you did wrong. They should say, here is something we need to fix and here is why. Tools can act as a medium to learn as you go rather than requiring everyone to get a PhD in secure programming.

Stephen Magill: The right tool is the one that lets you do the right thing without all that training. The security team has expertise and can guide and advise, but many things are formulaic and you want automation around them, so developers do not have to spend manual effort learning to recognize things that can be automated.

Hope Lynch: Gene asks: what is the most optimistic thing you have seen to help developers rise to the level that the outside threat demands? He says he is in awe of the carnage from SolarWinds, Codecov, and how software supply chains and CI/CD have been put in the spotlight.

Brian Fox: It is hard to find optimism in this topic, but developers generally seem to be aware and care about the problem, as evidenced by the usage of free and open source tools to help solve it. That contrasts with around 2011, when we tried creating Eclipse plugins to surface this information and nobody was interested. Well-informed developers would say they only had to worry about AGPL because they had a security team and a firewall for everything else. That is where we came from. Now in 2021, developers are at least scrambling to find tools to solve the problem. We still have a long way to go, but the awareness is there.

Sacha Labourey: Once you know it, it seems obvious. Before that, you had to spend cycles trying to make sense of it. Embedding quality and security as part of the process, so you inspect the process rather than the output, seems obvious. Gene has written about this for more than a decade. But in the field, you still see people compile source code and upload bits from their computer to the artifact repository, completely bypassing embedded security. Maybe SolarWinds and Codecov were a cost the industry paid to get a change. We talked about empathy, emotion, and learning through emotion. Maybe that is what it took to learn.

Stephen Magill: What about risk to the development process itself: bits uploaded in untrackable ways, dev environment or release artifact issues, and attacks like Codecov that corrupt the development process. Does the security team own that, or will there be a team within security focused on securing the development process itself?

Sacha Labourey: To me, it is part of the DNA of an organization. Not every team needs to suddenly get smart and learn how to do everything. There needs to be shared DNA, something built as an organization that everyone agrees is the baseline. That is how we do things here. It is part of your values and everything you do. Any team that gets onboarded needs to be onboarded this way. It needs to be hard not to do it. You need to make it easy and desirable to go this way rather than another way. If the environment is set up so everything goes with the flow, why would you do something else? It needs to be a key tenet of the organization's DNA.

Stephen Magill: Christopher asks how to make this area simpler for developers, because simpler is more secure.

Stephen Magill: One example from Sonatype's software supply chain report is that it is much faster and easier to do the right thing at organizations where there is an official process for approving a dependency. If you have workflow and tooling around evaluating a dependency, and picking something approved by that process gets you unstuck within an hour instead of waiting a month, you have made the easy path the more secure path. The easiest dependency to manage is one you do not have in your codebase. Think carefully about everything you pull in. The same goes for code: the only bug-free code is code you do not write. Keep things simple and small.

Attacks on the Development Factory

Brian Fox: Gene asks about my reaction when I first heard about the scale of SolarWinds. I have flip answers, but it felt inevitable. Since around 2017 I have tried to raise awareness of supply chain attacks. Many modern attacks are focused on the supply chain and your developers. Around 2017, we started seeing attacks focused on stealing keys of open source publishers, which was new and novel then. It continued, and it evolved into supply chain attacks trying to get into developers. Not so much trying to get stuff into your code to ship to end users, although that happens, and it happened with SolarWinds. Some attacks use development entry points to attack the rest of the company. Development infrastructure often has keys to the kingdom, especially in a cloud-native world and production, and those can be used to transition into other parts of the organization.

Brian Fox: You need to think holistically about developers, development infrastructure, and all of that, not just the product coming out. In the factory analogy, Deming principles are focused on making better cars, and you should do that, but just doing that does not make the factory safe from someone trying to blow it up. Many supply chain attacks do not care if they get into the end-user product. They do not care if they pass unit tests and ship, because all they are trying to do is run on your computer and put in a backdoor. Many traditional security practices are not designed for that. They are trying to inspect quality into the car.

Stephen Magill: Staging is an important environment too. It cannot all be about production. All of these are entry points into the process. Codecov and other attacks show the vulnerability of the development process itself. Dependency confusion is also in this space. The automation that enables us to be more efficient and productive is exploited to surreptitiously bring things in. If there is automation without proper monitoring, there is less awareness of what is happening, and you need those checks.

Hope Lynch: Laura from American Airlines asks: how do you get developers to care about security vulnerabilities even if their application is not critical to the operation but is still exposed to the internet?

Brian Fox: In a world with crypto, any of that stuff is directly monetizable. If you can get a CI server to run mining for you, that produces value. The old adage that my application has nothing of value is no longer true. Your visitors have CPU cycles if someone can put a JavaScript crypto miner there. Your build servers have CPU cycles. All of these things can be directly monetized in a crypto world even if you have nothing to steal. In some cases it is easier, because thieves often get caught trying to sell what they stole. If you stole cash directly, you do not have to sell it. That is the analogy in crypto.

Sacha Labourey: If you want to do an attack, you need proxies along the way, not necessarily technical proxies but steps that get you closer to the target. Those are often the easiest way to achieve things because people do not feel they are a target. In traditional hacking, a lot of it is social hacking: giving a call and saying, give me your password, I need to reset something. It starts with extremely low tech. Everybody needs to be aware of that. I have seen training material using 360-degree VR scenarios, like training pilots with alarms to see how they react. You need to be trained to react when you least expect it, because attacks can happen anytime.

Remote Work, Cloud, and Hybrid Reality

Stephen Magill: Has this risk changed in the current work-from-home era? How does that complicate the situation?

Brian Fox: For the attacks I was talking about, if developers are under attack and you were in a building with traditional perimeter defenses, backdoors might have been blocked by something else. If something got installed, it might have been limited in scope. How many people are working at home and no longer have those basic perimeter defenses in play? How would a company find out that one of their developers' machines was compromised? It becomes much harder. We tend to take perimeter defenses for granted because they have been around so long. They are not perfect for everything, but when you take them away, many of these attacks become more likely to land successfully.

Sacha Labourey: There was a lot of implicit embedding of security because the data center was king. Everything was funneled through the office network, VPN, and so on. Cloud or not cloud, that was how you were shielded. COVID and remote work showed that the cloud was the best data center because it is as close or as remote for everybody. It forced organizations to rethink the right perimeter. Hope mentioned zero trust in Slack, and that is right. We had many implicit assumptions about how things were secure.

Stephen Magill: Is adopting cloud the way forward for mitigating endpoint threats?

Sacha Labourey: No. In some ways it does not change it. If anything, it offers so much power that it offers a way to blow up your world in an unlimited number of ways. Developer environments, preview environments, and test environments get created and shut down. That is fantastic, but if there is a threat in any of those, there are many entry points into your world.

Stephen Magill: There are cloud development environments where the IDE is entirely cloud hosted, like Google Docs for code. Nothing lives on your laptop. Does that shift the perimeter back to something more defensible?

Sacha Labourey: The thread throughout the conversation is training and giving developers what they need. You can have the best tools, but if someone configures something the wrong way, suddenly you may have a problem. Buy the best tools in the world, but take care of your developers and integrate the tools into the process.

Stephen Magill: There is no single solution. Everyone's environment is different. Nobody shifts everything to the cloud; everyone is in some sort of hybrid environment. You need customized solutions, and that is why these conversations and audience questions about unique circumstances matter.

Sacha Labourey: It is good to openly talk about these things. Once a topic becomes a topic, like supply chain attacks, it feels like it should be solved everywhere and never happen again. But we know the velocity at which things happen. There is inertia, backlogs, business constraints, and so on. We are likely to see issues like this for the next decade. We need empathy and a safe place to speak and exchange about these struggles, maybe anonymously in some cases because it could expose a company. You do not snap your fingers and you are done with it.

Threat Modeling and Closing Optimism

Stephen Magill: Where do people go for guidance or conversations in this space?

Stephen Magill: Conferences like this are one answer. That is where I have learned most of the insightful things I have come across in the last few years.

Hope Lynch: Threat modeling has come up a couple of times: supply chain pipeline threat modeling. There is some pent-up interest around this. Sacha or Brian, comments?

Brian Fox: That feels like a whole webinar. Based on conversations I have had with people over many years, I would say most people are not modeling that threat. Some of what I brought up is probably new to people who had not thought of that door in the back corner of the factory. The number of ways the supply chain can get totally hosed is incalculable, especially with transitive dependencies and upstream repositories. How do you know what is on the repository? How do you know who put it there?

Brian Fox: An element of this is making sure you can detect and respond as quickly as possible to the inevitable problem. Then focus on locking down the things we have talked about. Have policies to control them, ways to detect in near real time when malware is published or when one of your developers pulled something down, and understand the blast impact. Assume one of these holes, including something you did not think of, will happen, and ask how quickly you could turn around even if you knew. Many companies still do not know what is in their software, and even if they did, they do not have mechanisms to turn it around quickly enough to respond. Many companies need to focus on the basics, but then not stop there. Keep moving down the stack to understand the rest of the implications.

Sacha Labourey: I want to share numbers from a recent global survey of C-executives. Ninety-five percent of executives claim their software supply chains are secure. That is amazing; I would not expect it to be that high. But when you double-click, two-thirds say it would take more than four days to fix a problem if they had an experience of that nature. Four days is huge; a lot can take place in four days. About 58 percent say that if they experienced one, they would have no idea what their company would do. At the high level there is a perception that the problem is being handled. When you ask specific questions about what would happen and measure things, it is a much less mature situation.

Stephen Magill: The first step is awareness. Even once you become aware of some things and address them, there is more to discover about the organization. Christopher asks: if magic happens and everyone knows how to secure the software supply chain, what is coming next? What big security threats do you forecast?

Brian Fox: I reject the assumption. We are not going to fix it this year. It is going to get deeper. That is the trend since this started in 2017. It is a giant cat-and-mouse game. Every time we get good at closing one hole, whether network ports, application coding errors, or dependency problems, attackers look at the next soft target. I would look for places where we are seeing weird one-off things, like the start of supply chain attacks in 2017. I do not know what they are, and if I did, I am not sure I would tell everybody, because as soon as these things are announced, attackers and copycats pile in right away. We saw that with dependency confusion.

Stephen Magill: Threats continue to evolve. I remember when we just had to fix memory errors and everything was due to C and C++ memory issues.

Brian Fox: Buffer overflows.

Stephen Magill: Exactly. JavaScript does not have buffer overflows, but we still have plenty of problems.

Hope Lynch: Brian, you are being constantly accused of not being optimistic. Gene wants you to say something optimistic.

Brian Fox: The fact that everybody is talking about it, that standards are evolving, and that open source projects are solving it tells me everybody recognizes the problem. That is the first step to fixing it. For so long, there were few of us pointing it out and people dismissed it. It is not as bad as before. At least we are talking about it and moving in the right direction.

Stephen Magill: Practices like CI/CD, tooling integrated into pipelines, and multiple points of monitoring really do make a difference. The attacks have had to evolve, which means we are doing something right. In this year's software supply chain report, we analyzed mean time to update: how quickly open source projects update dependencies when a vulnerability is released. That metric has improved steadily over the past 10 years, at least in the Maven ecosystem. The community as a whole is getting better. Modern tooling and practices help, they are being adopted, and there is focus.

Brian Fox: I have seen companies start to focus on the leading edge of the problem and look at it as a developer experience problem instead of just throwing more tools into it. They are thinking holistically. There are not enough of them, but the fact that it is happening now dovetails with the open source part. That is more good news and leads us toward finally starting to solve this problem.

Stephen Magill: Sacha, what inspires your confidence?

Sacha Labourey: I am born optimistic. It is great that we are talking about it and seeing crazy innovation in security: solutions geared toward developers, ops, runtime analysis, and so on. It is a complicated topic. It takes time, energy, and money. It is not going to happen by snapping your fingers, but it is becoming a real topic. We are on the cusp of a change.

Stephen Magill: We are close to time. Hope, final comment?

Hope Lynch: Thanks, everybody, for participating so much in the Slack channel. It made it a lot of fun.

Stephen Magill: This was a great discussion. Thank you.