Panel: The Dirty Truth of DevSecOps

Derek Weeks (Moderator)

Hello everyone at DevOps Enterprise Summit. I'm Derek Weeks. I'm Vice President at Sonatype. I'm also the co-founder of All Day DevOps. I am really excited to be here because this is my fourth DevOps Enterprise Summit, fourth year of it. I've also enjoyed going to London for a couple of other events over there. And this is by far one of my favorite events of the year. And I have the pleasure of being joined by Brian and Stephen today, who we'll get to in a second. But we are going to be talking here about the dirty truths of DevSecOps and the experience that Brian and Stephen have in this industry, in this arena, talking with the customers, and people out in the community that they've spent time with over the years. I think the really cool thing about this session is, one, we are live. This is not a recorded session. And we are taking your questions in the Slack channel, as we've just posted. So Nick, yes, the "Mad Max" reference is right, VendorDome, Thunderdome, and we're about to have a great discussion here. So, when you have questions on DevSecOps, post them live in the Slack channel. I have it right over here on my other screen, so I will see it there. So some of the dirty truths of DevSecOps, things that we can talk about. Tribal wars, if you will, between organizations. How these different organizations merge with one another, come together, provide information to one another. What kind of training opportunities are there? What are the CISOs asking these guys when they go and speak to teams of CISOs? What are the developers asking or anticipating, when we're talking about, DevSecOps? The cool thing is we can get a CEO's view and a CTO view through this conversation, so I think that will be an interesting dynamic as well. And we'll also talk about how different practices related to DevSecOps have changed over the last five years or so, certainly, that it has become more of a buzzword or topic of conversation or presentation topic at various conferences. So, with that, I am going to hand over to Brian for an introduction to him and a little bit about his background.

Brian Fox

Thanks, Derek. Hi everyone. My name's Brian Fox. I'm one of the co-founders and CTO here at Sonatype. My background is nearly 20 years in software development, C, C++, later Java. I also was heavily involved in Maven. I'm still on the Maven PMC, and I was a PMC chair for a number of years, and so I have a lot of experience dealing with enterprise software as well as open source software and how those things intersect. In the early days of Sonatype, we did a lot of training and consulting around Maven, helping people sort of uplift their build systems and modernize it, and we saw a lot of companies struggling to make effective use of open source without introducing new risks, and that led to our product portfolio and a lot of things we're going to talk about today. Trying to help people leverage that open source and do things in a better way that doesn't introduce new security risk. And over those, what's it been, 13 years or so, the industry has moved along quite a bit. I think people generally recognize that there's risk in open source from security aspects. In the early days, that wasn't true. People would say, "We don't have to worry about that because we have a security team and a firewall, so my stuff is cool. Just don't use GPL." So I think it's an interesting conversation to be had about how the industry has matured over those times, but that's my background. Stephen, over to you.

Stephen Magill

Yeah, thanks. I'm Stephen Magill, CEO of MuseDev. I've been doing software security and program analysis research and tool building for over 15 years now, ever since I did my PhD work in that space. And over the last few years, I've just been getting more and more interested in the practice of software and in getting advanced analysis tools into the hands of developers and really making security easier and lower friction, and just understanding more the problems that the developers face, that security teams face, and the interaction there. And so we've done joint research on that with the State of the Software Supply Chain Report. Sonatype and Muse together partnered with Gene Kim and IT Revolution to do some analysis of how enterprises use open source, how enterprises approach governance of their own software production workflow. And at Muse, we focus a lot on the relationship between security teams and development teams and what can sometimes become an adversarial relationship, right? Where the compliance team and the change advisory board and so forth is viewed as this big blocker, right? And I think a theme of this conference, something we've found a lot in this community is that often when you have those... processes and workflows that people grow to hate, it can often be fixed with the right application of automation and the right sort of standardization, and putting the right platform in place. And so we really focus on building a platform for continuous assurance, which is how we refer to this process of automating your governance workflows, automating compliance, and really letting tools manage the workflows and making things automated and self-serve from a development standpoint so that you don't have to have these complicated human-driven processes that slow things down. So yeah, so that's what we focus on is delivering, via that process, really deep insights into the code your dev teams are working on. So Sonatype focuses on open source, as Brian was talking about, which is a huge risk surface. So much you need to understand, in terms of open source usage and how that impacts security and compliance of your development process. And then we look at the code itself and what you're introducing from a security, reliability, performance standpoint in terms of potential bugs and issues in your software development process.

Derek Weeks

Yeah. So Brian, or no, I think I'm going to go Stephen on this, but so why Vendordome? Why did Sonatype and MuseDev come together on this Vendordome, I think? Stephen, we'll go to you, and then Brian, we'll go to you.

Stephen Magill

Sure. Yeah. So I think it's because of those sort of complementary focus areas. We're very focused at Muse on understanding the code that you're writing, and then Sonatype focuses on both analysis and then all the data that feeds into that analysis on the open source side. What's the vulnerability surface look like? What CVE reports have there been? What's the impact of those? And, yeah, Brian, then maybe you want to say more about how you can combine these two to get even better insight.

Brian Fox

Yeah, what we're doing is leveraging the call flow, data flow technology from MuseDev in our product. And so coupled with the bill of materials and the known vulnerabilities that we find in that, we can then produce prioritized lists for people to say, "You not only are using these components that have these vulnerabilities, but using the MuseDev technology, we can see that your code paths lead to those vulnerabilities directly. So you're more likely to be directly exploitable to the specific vulnerabilities." So that's how we're working together to take these two pieces of technologies and combine them for better outcomes for our users.

Derek Weeks

So this is the fun part of the session. So I had a couple of staged questions just in case the audience was a little slow to pick up, but Denver Martin out there had a really good question to start off. I'm going to go to Brian on this first, but Stephen certainly chime in on this as I'm sure you have experience in this space as well. So Denver says, "How do you get info security or InfoSec to help fund projects like web application firewalls and other crossover items that DevOps wants to use, however, it's an additional cost?" Right? So this is the, "I have a DevOps team. I know we need security. Maybe we don't have as much budget, but the security guys have a ton of budget over there. Can we go and borrow some of that, leverage some of that?" How does that conversation go? Or any advice for Denver and others out there?

Brian Fox

Yeah, it's a great question. I think we did some round tables last year, right, with primarily CSOs who were describing the problem that they can't get more money to do better security, and better security by meaning hire more people to do more scanning to help development teams. And some of the more enlightened conversations we had was when those CSOs recognized that if they were able to use their budget to help fund the platform, in some cases, some big banks, we had seen that the security organization had basically funded the entire SDLC, the modernization of the tool chain, because they recognized that by helping to do that and helping development do a better job of picking the components and turning the components around and being able to do releases more quickly, not only did that mean that development could do a better job of fixing memory leaks and bugs and things like this, but also when those bugs turned out to be security vulnerabilities, they could turn those fixes around in the same way, right? So being able to make that case to your security team that if we work together, everybody wins, that there isn't a natural reason why you need to think only about security vulnerabilities as this second-class citizen. They're ultimately just bugs, bugs that have different impacts, sometimes profound impacts. But from a development perspective, the way you fix it is often the same as you would do for these other types of defects. And so recognizing that, helping to combine those budgets can provide much better outcomes versus trying to buy more, playing defense and spending more money trying to chase the stuff out after the software's built and trying to, what ultimately will lead to a bigger battle between the two tribes. That doesn't work, right? Mm-hmm. So I think it's a great question, but it can be hard to change people's minds into a more progressive stance. But when it happens, we've seen it be tremendously effective.

Derek Weeks

So Stephen, if development or DevOps wants security tools and they're going to a security drive or security organization saying, "I actually want some security tools right? I want to embed more security into my development life cycle." Isn't that security's dream? Development never wants to use security tools. Why wouldn't you be behind this? Maybe you can shed some light on conversations you've had in that regard.

Stephen Magill

I think that's right, and I think we see that more and more as teams become more enlightened and security especially recognizes that the better relationship they can have with the development team, the smoother that process can be, the better, and I think in a sense, that's what DevSecOps is all about. It's not just tools, but it's how do you set up a culture where security is on everyone's mind? And so it's, yes, developers have security tools that they want to put in place and have opinions about how those should integrate into their workflow and then be part of their daily work. And security has their opinions there too, and everyone's working together to really come up with a process that everyone can unify behind that addresses everyone's needs, right? Because security and developers and compliance and audit and QA, and there's a lot of different stakeholders in this process, and so we try to encourage everyone to-- in terms of what we put together, the analyses that we focus on, security is an important piece of it. But we also focus more broadly than that because developers care a lot about reliability. No one wants to get paged at 2:00 a.m. because something went down, right? And so you can address all of these issues with one platform. That's obviously the best, and I love Brian's point that even things you don't necessarily think of as traditional security issues can certainly be security issues in the right application and the right context. If you care about denial of service, then performance issues suddenly become a security issue.

Derek Weeks

Yeah. Denver, thank you for your question. Hopefully, we answered it. If you have follow-ups, certainly chime in. I see a number of people chatting in the channel. I'm going to go to Mark Fuller's question next. And this is a really good, interesting question. I think there will be different perspectives on how to answer this, because I don't think it's tightly tied to either Sonatype or MuseDev's portfolio per se. But many of the latest DevSecOps tools work poorly or not at all with legacy systems, for example, COBOL on a mainframe. At the same time, these systems run critical applications that we use every day. Do you see this as somewhat of an enforced strangler pattern, or do you think DevSecOps tools will fill in the gap and keep legacy systems running for years to come? So we've had this discussion over time in other conversations about there's legacy infrastructure out there, but there's also legacy security tools that are out there. And then there's kind of this new modern DevOps experience out there, process, practice out there, and there are new tools that are being developed specifically to address that. Do these stay as two separate tribes or two separate ecosystems, parts of your infrastructure, or does one expand to fill into the gaps? Do the legacy guys catch up in the security game and come into DevOps practices to expand the solution offerings, or does a more modern DevSecOps tool expand into cover legacy environments? Either one of you want to go first on that?

Stephen Magill

So, my thoughts there are you certainly can do DevOps in a mainframe context. Rosalind Radcliffe at IBM has spoken at DOS events about doing just that, had some great talks there. So I think DevOps practices and the value that those can bring from a reliability and security standpoint and speed of delivery standpoint, those are all within reach. I guess I'm less optimistic that we'll see really a sort of flowering of many different tools targeting security issues in, say, the COBOL ecosystem. It may just be not a large enough focus area, but I think there's a lot of value you can get from very narrow, specialized tools, right? And so we've actually talked to a lot of companies where they see a gap in the tooling space, or they see a gap in the market and their developers write a linter or something themselves, right? Do some simple checking of properties of the code that they care about given their coding style and patterns, and you can certainly do that sort of thing, even if you're on more of a legacy technology base. And I think then if you can adopt those DevOps practices and have some sort of DevOps strategy and platforms, then you can start to incorporate those the same way you would a tool from a vendor.

Derek Weeks

Brian, what are your thoughts on, is Sonatype going to do anything to cover the legacy environments, or is just covering the more modern DevSecOps practices challenge enough?

Brian Fox

Yeah. I think there's certainly a market element that works against you, right? Nobody's going to take a new tool seriously if the only thing it supports is COBOL. So everybody has to start with the modern languages, which then means you're working down the long tail. I think the other element is some of the older languages, and this applies not just to older languages, but even you could have the same thing with really old Java. The way it's being developed isn't really compatible with a more of a modern agile kind of thing, not without a huge lift that may not be warranted for an older application where you're largely in maintenance mode, right? So specifically on COBOL, I don't think there's a lot of open source modules that people are out there picking, so you don't have that same problem. You certainly have challenges with vulnerabilities in the code, right? Where other tools might be more well-suited. So I think that's the problem is that, and it's not just a problem with legacy systems, it's all of these ecosystems. The way the developers work with them is just different. And you try to find common patterns and build tools for that, which creates sort of this dynamic. So yeah, if you're supporting a significant outdated legacy portfolio, I honestly don't think it gets easier. Yeah.

Derek Weeks

Maybe not the answer that Mark was looking for- The current reality, right? But yeah, these are the dirty truths of DevSecOps and what we were talking about, right? Right. We're not here to kind of spin, "We have a positive answer for everything." One of the other questions that came in that got a couple of thumbs up was Steve Jones. He said, "Is data privacy, data security classification becoming more important to your organizations and customers?" So this is about where is the data, how are we protecting it, how are we protecting secrets within our infrastructure? How are we protecting the customer's data that's in these applications? Yeah. How much is data privacy, data security becoming an issue or a standing point in conversations? Yeah. So

Stephen Magill

we hear a lot about that. It's a huge compliance issue for one thing, right? If you're dealing with credit card information or customer data and you're PCI compliant, there's requirements in terms of how you handle this stuff. And so we've done certainly a lot of looking into what can you check when it comes to the code? How can you enforce policies during development on logging of particular transactions or encryption of sensitive data, ensure that certain types of data don't go to certain endpoints. And that's all doable and is definitely a best practice I would recommend in terms of ensuring that's in your code scanning pipeline, that you are looking for those sorts of, the key word is information flow analysis or data flow analysis there. And then there's security in depth, defense in depth is important, too. So it shouldn't be the only thing you rely on, but it can definitely be an important part of the strategy.

Derek Weeks

Brian, anything to add there, or you want to go to the next question?

Brian Fox

No. It's a concern. It doesn't directly come into play with some of the things that we're doing specifically, but it's definitely a concern, both of our customers and how we're handling it, how we're thinking about it, and how we're thinking about their data. But that's about all I'd have to add, I guess. Yeah. Part of it is, at Sonatype, we have data from our customers that we're collecting, but we also have a policy that we don't keep customer data around for very long because we're sometimes touching on security related issues or security vulnerabilities on things that they know. So we have very explicit policies as a vendor that when we're collecting specific customer information, that we don't hold onto that in-house for a long time because we actually see it as a liability to hold it for too long. Like someone had this open source component that they used in their code, and we've known about that for the last two years. If anyone got ahold of that, that would be difficult. So we try and purge that information on a frequent short-term kind of window so that we don't hold that and leave our customers more vulnerable as a result. I know it's part of that.

Stephen Magill

This brings up one thing I want to ask the audience about, which is as you say, Derek, the best way to protect data is to not hang onto the raw data at all, right? Yeah. I'm curious if anyone is using differential privacy. A few years ago, it was just a research topic in the academic community, but it's been deployed by Google and Apple as part of their privacy protection schemes when it comes to Chrome data collection and Apple device collection. I'm curious if it's making its way more broadly into the ecosystem of things people look at when it comes to protecting sensitive data. So if you've looked into differential privacy or you're using differential privacy, message me. I'd be interested to hear about it.

Derek Weeks

Yeah. Okay. I'm going to go to, Aras had asked a question. If I'm interpreting this question right, he said, "I'm usually seeing security teams going to development saying, 'How can we help out? We have tools, we have practices, we have training,' and so forth, but unless he's mistaken, he's not seeing a lot of development teams going to security saying, "Hey, we actually need your help. We want to engage you in this." Do you guys see it as a kind of one direction thing of development is controlling and building these applications, building out the infrastructure that they need? Does security have to come that way, or are you guys seeing other situations where development is actually going to the security team saying, "Help. We don't know exactly what to do, but we need some help with- Yeah... adding security into our practices."

Brian Fox

I see these challenges a lot. I think there's a history that we have to recognize, even if we wish it wasn't true, that a lot of security teams in organizations are perceived as friction points or blockers getting in the way of getting things done. And there's been a history of bad tools producing lots of findings, right? And I've heard examples of people saying, "Well, we had a mandate from security that we had to fix everything that our SAST tool provided." Which is just kind of ridiculous on its face if you understand that the tools are not perfect. But if somebody is using that data incorrectly, weaponizing it, and then expecting us to fix all of these things, it's going to create new problems. And so there's the history of this, so it's not surprising to me that development teams don't go asking security. I think it's sort of that they'd rather just fly under the radar. But the reality is, like I talked about before, working together on those things is what provides the best outcomes. And fortunately, I think we've seen over the last handful of years specifically, that development has become generally more accepting and more aware that they can do a better job around the component management that will reduce the number of security findings they have to deal with later. But it almost has to come from the top. Because when somebody can say security is friction and they're just slowing us down and stopping us from doing our job, there's an implied definition that their job doesn't include security, which is just wrong, right? But many development organizations have, for whatever reason, that culture where the only thing they're goaled and measured on is producing features and shipping stuff and fixing bugs. And if the definition and visibility of that doesn't include all of these other dimensions, then that's what you're going to get. You get what you measure, right? And so there can be subtle misalignments that then end up with these magnified culture wars that I see so many different things. I'm sure, Stephen, you've seen similar aspects, right?

Stephen Magill

Yeah. And I think one really important area, and I don't know how much this is happening, again, I'd be interested in stories here, but I think the most important time for developers to go to security is before any code has even been written, right? When you're planning a new feature, you're planning a new component, and you want to make sure that from an architectural and design perspective, you're setting things up to be secure, to be easily auditable, to have sort of separation of concerns in the small piece of critical code that handles security. There's a lot of aspects of that initial design process that I think security has a lot of insight into, and it can be really useful to make sure you cover those early, rather than you get to the end and you find out, oh, because of how we designed this API, it's actually really hard to meet our requirements.

Derek Weeks

So, I'm going to try and interpret Brad Appleton's question here. He said, "Anyone having any luck with a combination of GitOps, DiffOps?" And I think this is pointing to, as a means of eliminating manual change approval bottlenecks. So I'm interpreting this as where you're combining security as code along with infrastructure as code, where these two concepts are merging together. We care about the application, or we care about the infrastructure, and can I do this in a way that eliminates more manual checks and approvals within the infrastructure, or any of these concepts coming together out there to maybe use fewer solutions or vendors cooperating together, et cetera. Yeah. Or even teams cooperating together, not just vendors.

Stephen Magill

Yeah, I think infrastructure as code helps a lot. And in terms of DiffOps and GitOps, the thing we focus on the most in terms of recommendations and how Muse works and the workflow that we recommend is exactly sort of diff focused, right? You look at the code changes that are coming in, you look at security issues, other sorts of issues that are potentially being introduced by that code change, and you sort of handle it in the moment as the code is changing so that you don't have this lengthy process at the end, and you can sort of certify that process and keep your release cadence up and not hit these barriers that slow down work. And I think infrastructure as code helps that too, because it just brings more of the platform, more of the software within the purview of being able to be addressed by that sort of diff-oriented workflow. Brian, any comment there on-

Derek Weeks

Yeah, I would say the infrastructure as

Brian Fox

code element of this is kind of the newest part to this dimension, butSo many things. First for us it was licensing, then security, then architecture and quality, and infrastructure as code. These are all dimensions of things that ultimately development needs to be concerned about. It's best solved within development, especially if the new paradigm for how you're building and deploying your applications is in the cloud, then something like infrastructure as code is your build system. There's no package, right? You don't produce an installer or a ZIP or a WAR anymore. You produce configuration in Amazon that affects running of your application, which means in order to test it, development has to do it. The way to solve all of those problems is to be able to provide the insight and the guardrails and the information to development as real-time as possible in the environment they're doing. Coming later with tools, whether it's a security SaaS tool or an infrastructure as code tool that basically says, "Oh, you got it wrong, do it again," is both ineffective and super frustrating. And especially in an infrastructure as code scenario, by the time you scan it after the fact might be too late. It might not be manifesting itself as a security issue, although we see this all the time, where companies have databases that are wide open to the world, or S3 buckets that have data dumps in them for anybody to consume. That certainly could be caught with some type of checks up front, but it could also be, did you really mean to spin up 10,000 extra large instances and leave them idle and cost us a ton of money, right? Those kinds of checks done early can prevent both security and just simple misconfiguration problems. But again, it has to be done in a way that works with the agile development environment.

Derek Weeks

I'm going to go to one of the questions that I had before, and I always find this interesting. So you guys are out talking to people in the market all the time and having conversations around how to bring more security into DevOps practices. Stephen, is there a standout conversation that you've had with someone on this topic, whether it's a developer or someone at a conference that you sat down with and happened to have an hour and a half conversation because you were so enthralled by the topic that you remember as like, yeah, this is the- Yeah... the one standout conversation moment?

Stephen Magill

Yeah. There's a few things that stand out to me. Actually, there's a number of conversations I've had in this area that all fall under this umbrella of automated governance and focusing on that. And there's now a lot of different organizations in the DevOps community that I've talked to leaders at those organizations and heard about their efforts to automate their governance process and improve that. And I think that trend is the most direct attempt to address this at an organizational level. Baking security really into the entire software development process. And I think the other thing that's encouraging about that is these efforts that I hear about are largely developer or DevOps teams are driving them. They're bringing in security and they're bringing in the auditors, and they're making sure that this workflow works for everybody, but in particular, they're making sure that it works for developers, which is strangely an often neglected part of the process. Making sure that these processes are actually helping developers. And so, I think there's some great examples there. Capital One has done work on that, PNC Bank. There's others in the community too that are really pushing this forward. There's some great DevOps summit talks. I'll try and find and send some links out. But yeah, it's really the fact that I've had so many conversations in that general area that I think is really exciting and gives hope.

Derek Weeks

Yeah. You talk about security working for developers. I'll try and find the link to this, but the story I was reading that a developer at Google had written up how they do security around their code and how they check for security vulnerabilities within their code, and that they were getting open source products, and they were bringing them in, and if they worked fine, they stayed in. If they didn't work, they went to the open source projects and said, "Here's what we need you to improve." Or they ended up building them themselves. But it was this whole process of like, here's how we get security to work as development wants it. And it didn't involve the security team at all. It was actually just the development team saying, "If we're going to do this, and we want the tools to behave like this, then how do we want them to behave?" So when they were kicking out a bunch of false positives, they were like, "This tool is trash, and either if the vendor or open source project can improve it, fine. If we can improve it, fine. If not, we're just going to turn it off because it's just this noisy thing that we don't want to deal with." But I'll try and find the link to that.

Stephen Magill

Yeah, I'll send out, I think I know the article you're talking about, I'll send that out in Slack, too. It's really great, and I think there's a couple of key things that always stood out to me about how they approach it that I think are really important. And one you touched on is really being data-driven. Paying attention to, we're running all these tools, which ones are providing values, which ones aren't? When a particular tool reports a result, how likely are developers to fix that, and really tracking that and using that to adjust what tools you run and how they're configured. And then taking that platform approach and running multiple tools. So I think in that article, they sayAs of the time that was written, which was a couple years ago now, they were up to 146 different tools running on their code analysis platform, which is just really incredible, right? And a lot of those were in-house tools developed for very niche purposes. But it just goes to show that if you have the platform right, and you're data-driven in how you adjust and turn on tools and turn off tools, you can really do this at Google scale, right? So, yeah.

Derek Weeks

So Brian, I'll give you a choice of questions that you can ask. One is either the standout conversation, because I know you were thinking of what's my standout conversation, but there's also a question from Sruji about AI and ML. So, this person asked: Is there any luck with ChatOps, like automating end-to-end automations pipelines with integrating chatbots or any AI, ML? So part of it, whether it's that explicit example of ChatOps specifically with AI or ML, or just more general, how have you seen machine learning and artificial intelligence applied to making security practices better within a DevSecOps context? So I know that we're doing work at Sonatype applying AI and ML to different scenarios. So if you didn't have a specific answer for that, I think it would be interesting for the audience to hear how we're engineering AI and ML into our construct or architecture.

Brian Fox

All right. How about I do both? Okay. So, a standout conversation was, I covered it in the beginning around the CSO who actually paid for the whole central tooling. That was surprising and made a lot of sense. A different conversation was interesting a couple of years ago at one of our events, I think it was in Dallas. I've been so used to talking about the challenge with open source components is that development often doesn't know, they don't have the tools to easily help them understand that they should be making better choices around the dependencies. It's why are you still using Struts after all these years? And, "Well, I didn't know it was vulnerable." Right? Those kinds of things. I had an interesting conversation with somebody who was more on the ops side of things, and they said, "Look, we've got all of these intelligence feeds that come in that tell us when there's a new vulnerability in one of these things." Think common collections. But their problem was they had no idea what was inside the application. So it was this situation that they were tasked with defending the perimeter, and it was sort of the analogy of like, "Well, I'm defending this perimeter, and the bad guys are shooting at these tanks. I don't know if the tanks are full of water or if they're full of gasoline. I literally have no idea." And so it's a really interesting insight onto the other side of things, and it's like, well, okay, that's really neat because our tools can help provide that depth of visibility into what actually is in this application, so you know you should be looking for cross-site scripting or not, or SQL injection or not. And really, it can make it much easier for them to do a better job of defending the perimeter when they realize where the soft spots actually are. So that was a really interesting standout conversation that enlightened me. In terms of the ML/AI challenges, one of the things that we've seen in the past couple of years is this new dimension of intentionally malicious components being injected into the software supply chain. There's a ton of examples that I've been talking about for a while. And it's new and novel in that two things, that rather than waiting for vulnerabilities to be found and disclosed and then trying to exploit them at scale, which is like the Struts, the Equifax problem, right? These things we've been dealing with for years. That they're introducing these vulnerabilities on purpose. And that in and of itself isn't necessarily new, but what seems to be new to me is that a lot of these attacks are focused on the development environment itself, right? And trying to exploit the software development environment, either to use it to replicate, like you saw in the Octopus vulnerability malware, or in that they're trying to steal credentials and other important things off of the development environment. And to make it real for people, I've started to try and describe in DevOps, DevSecOps, we talk a lot about Deming and the principles there. And those are all great principles, but we have to recognize that those principles were focused on the end product, the car, making an assembly line that could produce cars that were cheaper, better, higher quality, right? But what it was not doing, it was not trying to protect the factory against some kind of malicious attack, either a bad actor or parts shipped in that blew up the factory or contaminated the factory. And so when you think about security, you need to think about it in a different way. And what we've had to do is try to use ML/AI technologies to look at the new components that are being released into the wild and try to understand elements about them. Who released it? Have they worked on this before? Did they add a dependency that's never been seen before? There's so many different kind of attributes that you can collect that it's impossible to build heuristics around like if this happens, then this release of this component is inherently dangerous. But the machines are really good at detecting that, just like our credit card companies are for detecting credit card fraud at the moment of point of sale and send you a text, "Is this you? Yes or no?" Right? So we've been trying to integrate technologies like that to be able to give our customers an instant adjudication on a new release to understand if it's fishy or not, and maybe they should wait until somebody can confirm if it's malicious or if it's justSomething we've not seen before. So that's what we've been doing with ML/AI techniques.

Derek Weeks

Yeah. No, it's super cool in terms of that scenario and the others that you explained. Stephen, I'm going to throw a question out to you that I've heard pontificated around the industry, but is security going the way of QA? So we don't have any QA departments anywhere because DevOps figured out how to automate QA and build testing in. Is security going that same way? Does security as a practice or application security as a practice disappear into DevOps or development?

Stephen Magill

I'll say, I hope that we're moving that direction for the portion of security that can be addressed by automated tooling. So I don't think there's a lot of value to a tool reporting something to a security team that then has to file an issue with the dev team and participate in a prioritization process to get something fixed. There's no reason that result can't be communicated in a way that makes sense directly to the developer and enables a fix to that issue without all those people being involved. And so, my hope is that security teams shift to being, and I think security professionals would by and large hope this happens as well. They don't want to be combing through thousand-page lists of results and filing issues. They'd rather be focused on pen testing and red teaming architectural decisions and things like that. So my hope is that we do get the tooling to the point where those teams can focus on those sorts of issues. Yeah. And to the ChatOps issue, I think a good example of this is, and the Google paper mentions this too, this process of reporting issues directly to developers in the code review system, because that really is one of these key areas where it is conversation-driven. It's a social process. The team is coming together to decide what changes need to be made to the code before it's merged and deployed upstream. And so, it's the perfect place for automated tool results to participate if you can do it in a low noise, very actionable manner, and so we always try and do that to present results there.

Derek Weeks

Okay, Brian, I'm going to shift a question to you. This is an interesting one. So, I'm not going to read the whole thing, but is there a place for ChatOps to be applied where an application vulnerability would come up, ChatOps surfaces that vulnerability in the code to a developer, AI/ML supports this with saying, "Hey, by the way, we've spotted this vulnerability there. It looks like there is a potential fix out there. Do you want..." Is this ChatOps the trigger for troubleshooting or remediating this to say, "Problem came up, computer told me there was a potential resolution path, I'm just going to click accept and move on"? Do we see that as the path to automated remediation where machines are just remediating these code vulnerabilities for us, or is a developer always involved in kind of let me verify that change before we go through and place a new version of an open source component in that code because we update it to remove that vulnerability, and AI/ML is intelligent enough to know there's not a breaking change in that upgrade path? Yeah. How far does this go?

Brian Fox

Yeah, I think the natural conclusion of that is that you get to a level of full automation for certain things, but the question becomes how do we trust it? And I think ChatOps is that mechanism. I read the story of elevators back in the day when they first installed the automated systems for elevators, people didn't trust them. So what'd they do? They'd put a bellhop or whatever on the elevator to push the button for you. But that's literally all he did. He wasn't driving the elevator anymore. And then eventually it got to the point where people were like, "Why is there just a guy here pushing the button? I can push the button myself." And it was more about us as humans trusting the new technology, and in the beginning it felt too much like magic. We didn't believe it. But then when it becomes natural and obvious and redundant, then you get to the full automation. So I think the ChatOps element in the pull requests is the bellhop in between the technology and getting to the point where we just go, "Yeah, that's obvious machine. You're right. Just do it for me." No, that's a great story. I haven't heard that one before, but a great perspective on why you have- Yeah, I think about all the things we do now that are just buttons, someone else pushing the button for us. That's right. Yeah. So,

Derek Weeks

a question I had, Brian, you touched on this earlier a little bit, but I've said part of DevSecOps and security is it's only around in here because we have adversaries. So what are adversaries doing that we need to be aware of? There's all this, can we bring teams together? Can security and dev work closer together? Can we use different new evolution of tools in the DevOps space to find vulnerabilities? But what are adversaries doing that we need to be aware of outside of our walls that's a crafty move that you've heard of lately? And Stephen, if you want to chime in after Brian, please do so as well.

Brian Fox

Yeah. Well, I think it's a lot of the stuff I was talking about before. Recognizing that there is this tribal warfare, that security often is focused on defending the end result and not defending the factory, and then using that to exploit it. So if you can insert some malicious code into a component, especially in an ecosystem where people tend to grab the latest version, like NPM as an example, it means you get it in there, you instantly have users that are running your bad code. And if you know that most security practices might run a scan daily or weekly or only just before release, that's a long time and a soft target that is unprotected for a long period of time where that stuff can go undetected. That's why we're trying to use the MLAI technology to detect that something's fishy to prevent that from happening up front, without putting in place a blanket statement that you can't use any code that's less than, what? Pick a number. A week, a month. How old does something have to be before you can trust it just across the board? There is no good answer to that. So that's, I think, one of the things that the adversaries are looking at. They're understanding that our desire to evergreen and keep up to date actually can be a weakness because of the way traditional security is looking at the problem. So that's, I think, the newest, biggest kind of challenge that I feel like not enough people recognize, because they're so focused on just doing a better job of making that car safer and not looking at leaving the factory door wide open on the other end. Right? Mm-hmm.

Stephen Magill

Yeah. That's interesting that different communities, different ecosystems have sort of different... Vulnerabilities look different from a supply chain perspective in each. So, in the supply chain report, we found that in the Maven ecosystem, by and large, the sort of dangerous behavior is not being up to date enough. Right? Correct. There's a lot of libraries not updating their dependencies, but NPM looks quite different, and so your threats look quite different as well. Mm-hmm.

Derek Weeks

Okay, so I think we only have a couple of minutes left, and one of the questions that I had down was, can you automate all application security practices? Which ones might be the easiest, which ones might be the hardest? Can you automate everything in regard to application security? Or, I'll extend that to infrastructure as code security as well, security compliance.

Stephen Magill

Yeah. I think you can't, and there's a few things that work against you in terms of automating everything. So one- Stephen, no one wants to hear that. Just kidding. It's the dirty truth of DevOps. That's the cops. Yeah. I think you can automate a lot of it. And automation obviously helps you do more with the manual resources that you then have left over. But, yeah, there's the things that just can't be automated, at least not right now, maybe in the distant future. Things like questions around design and how security is baked into the system from an architectural perspective. A lot of just sort of logic errors, like application or domain-specific logic errors that can lead to security issues. There aren't general scanning tools for those because they are so specific and ad hoc. But then there's also on the infrastructure as code topic, I think this is an example of this, there's the gap between how things are described in the code base and then how they look when they're deployed, how they're ultimately deployed. And so if you really want to take a broad view of the threat space, you have to worry about things like problems with the VM or a particular version of Docker, or a particular version of Linux providing some vulnerability that maybe isn't reflected in your configuration. That's your configuration plus the deployment infrastructure, plus the versions of things that you're running that all sort of work together for that. And I think we'll get closer and closer to that from an automation perspective, but it's something that's not always recognized in terms of threat space.

Derek Weeks

Yeah. No, there's lots of specific tools out there that are solving different parts of these problems. But you're right, that kind of in the end is this combination. It's like going to the pharmacy and like, "Okay, I need this drug because my doctor said I needed some inflammation medicine. But does that interact with the two other medicines I'm taking?" And that's kind of the same thing in these environments is just because you might have solved one problem doesn't mean it's not creating another or hiding another problem within infrastructure or another layer of the environment.

Stephen Magill

Yeah. But also, it may be the case that the problem you can solve with automation is the most important problem for you right now, in which case, you know.

Brian Fox

That's right. Yeah, I was going to say that. I wanted to say to Derek, no you can't, and almost it doesn't matter. Why does it not matter? Yeah. Because I see so many organizations that are so far from that nirvana that they can't even answer, "What applications do I have and what components are in them?" So that they're not even in a situation where any automation can even help in that automated kind of remediation sort of way. But they can- Yeah... automate understanding where they are. Don't rely on the humans to report what components you're using, as an example. That is something that you can automate. So don't let the fact that you can't get to 100% solution stop you from taking the obvious first steps and move further along that spectrum. Because the horrible truth is, so many organizations are so horribly broken in so many of the regards we talked about, that that's what needs to be fixed first. Not automating that pull request at the end of the day. But as engineers, we like to focus on the hard-to-solve problems, not the obvious need-to-be-solved problems sometimes.

Derek Weeks

Yeah. It made me think back to a conversation with Shannon Lietz at Intuit, that they went through and they solved a whole bunch of problems and applied state-of-the-art technologies into their infrastructure to look at how do you protect Intuit's applications and environments. But once they did that, they took a huge focus on the adversaries. What are the adversaries doing? What tooling do they have available to them? What are their attack patterns? And they began to measure things like adversary return rates. So they said, "If we're doing a really good job, and we know adversary attack patterns and can identify those, if this person comes and tries to attack us today, do they actually come and try and attack us tomorrow or next week? And if we're hard enough to break into, they'll leave. If we fortified ourselves and protected ourselves enough." So it's interesting to see this dynamic from once you've automated as much as you can, even if you can't get to everything, look toward how do we know if we're being successful and how do you apply metrics into these environments? And part of that thing is the adversary return rate. If you don't have adversaries, you don't need to invest in security. But if you know how your adversaries are performing, and if they're coming and visiting you often or not, then you're going to know how successful you are. So I'm being told from our host that we are out of time, but I know Brian and Stephen and myself are around at the conference. We have a Birds of a Feather session later this afternoon on security compliance governance, that you guys can drop into as well. I hosted that conversation last night. There were a lot of really good questions and experiences shared with the team, so add that Birds of a Feather session to your calendar, and that's BOF-sec-audit-compliance. So you'll see that pop up later this afternoon in the agenda. But thank you, Brian. Thank you, Stephen. Always great as usual to have your experience and share it with the audience. And thanks to everyone for submitting your questions along the way for these guys and them doing it live. So perfect. Yeah. Thank you. This was fun. All right. And come take a look at the Sonatype and MuseDev booths if you haven't. Take a look there. I know they have a number of offers, whitepapers, live demos, and other giveaways, so definitely head to the Sonatype and MuseDev booths as you can, and hope everyone has a great conference.