Paul Holt

Good afternoon, good morning, good evening, wherever you happen to be listening to this from. My name is Paul Holt. I am Anchore's General Manager here in EMEA, just based outside of London. And welcome to this Vendordome, which is putting the security in DevOps. I'm joined by two of my esteemed colleagues today, Neil Levine from Anchore, and Sam White from GitLab. And we are going to spend the next hour or so debating and asking some questions around what it means to have DevSecOps, how you want to implement that, and some of the most recent attacks and the implications that has on just the supply chain, and just generally taking questions as well from your good selves. If you want to post questions, we are in Ask the Speaker track number four. As I say, we're live, so we are going to be taking questions as they come through, so please do not hesitate in posting the questions in there. So we'll kick things off, and the first thing which I want to pose to our speakers is essentially what DevSecOps means to you. And perhaps Sam, do you want to take the baton first and kick things off for us?

Sam White

Yeah, absolutely. So DevSecOps really means that security is incorporated entirely into that software development life cycle. So, rather than the traditional approach of developing some code, throwing it over the fence to security, having them run some scans against it, and then eventually some of those results make their way back to the development team to action on, means that it's really tightly integrated within the entire process, so that developers are able to continuously scan their code. They're able to see results right there as they get them, and that security continues both, not only during that development process, but really before the development process and hardening the developers' workstations all the way through afterwards when you ship it into production, and you continue to secure it. So DevSecOps to me really means that security has become part of that entire process, and it's just a piece of what we do as we develop software.

Neil Levine

Neil, I'll pass it over to you. Yeah. I remember when DevOps came up as a term of art in the industry that, I think Adam Jacobs, who's one of the people who helped promote it, always said it's a cultural movement as much as anything else. It's about creating culture where, for DevOps, it's about having developers and operations collaborate together and not see each other as siloed organizations, and that they should collaborate together. So, he always emphasized the cultural aspect, which was that organizationally, you had to change things up and see yourselves working as a group. And so I think, DevSecOps, if you extend that concept or way of looking at it, is absolutely about bringing security into the process and having developers and operations, and see them as part of the working group to ship an application, and not as a group to throw something over to. So, Sam's absolutely right. In practice, what that means is you need to have security checks implemented everywhere. Security need to advise on where you need to be doing security checks and helping structure those such that it is part of the day-to-day workflow. But it's important not to overemphasize the tools and the technology, because we're vendors, we tend to do that, of course. But it's absolutely about a collaboration thing, where even when you're sitting down and you haven't written any code and you're having your design meetings, everything else, that security's there as part of it. So it's important not to strip out the human element. But yes, what it means for most organizations today is that they're integrating security into development and not leaving it right till the end, which is traditionally where it's been implemented.

Paul Holt

Yeah. And I think, on that very point, in terms of some of the challenges that we have seen as vendors, as we talk to organizations about implementing security into their CI/CD and their software development processes, what are the typical challenges that we've seen as organizations have tried to embed security into these development workflows?

Neil Levine

Neil, please. Yeah. So, I think this is where when I say some organizations are not doing DevSecOps, it's because we still interact with security teams who are playing catch up to where the developers have been. So often the developer or/and the operations side has moved faster in terms of building automation and building that whole end-to-end pipeline, and the security team are playing catch up to that and working out where they can add things. And the telltale sign that we often see is a security team who don't know what's going on in the development process but have seen a container registry appear with a whole bunch of content, and they've got to go and work out what's inside it, or they're starting to be asked to produce reports or audits. And so, often that's a sign of, well, you're not quite doing DevSecOps even if you do have a few security checks implemented in the system. If the security team don't know what's going on, then you don't have that cultural integration. So, that's definitely been one of the markers where there is a problem. And I think it just reflects the fact that inherently, developer teams do move faster than security teams, and so whether it's picking new tools or just writing code, they tend to be far more nimble and less constrained than the security teams. And so I think that's-

Paul Holt

Yeah... one of the frictions is just inherent burden. The security teams have a lot to look after. They can't unshackle themselves from things once they've involved in that application. It is an ongoing process for them. Yeah. Sam, any additions to that?

Sam White

Yeah. I would say that's absolutely true. I find that from just a practical standpoint, most security organizations report up through an entirely different reporting structure from the development team. And so, where you have a security team, an application security team, reporting up through a CISO or a CIO, they're just not talking a lot with development, at least in most traditional organizations. And so, they may go throw things over at development and say, "Hey, we found this critical vulnerability. You need to go fix it." But I find that traditionally, that's really where the collaboration starts, and that's where the collaboration ends. And it becomes this unfriendly relationship where development says, "The security team is trying to make us go do this work, and they're trying to make us fix things." And the security team's perspective is, "The development team is creating these vulnerabilities that now we have to deal with, and they're not fixing them." So it becomes this a little bit of an unfriendly relationship, at least the way that I find that most legacy organizations work. And again, a big part of that is just a function of that reporting structure. It's those organizations that are a little bit more mature in their security development, I find that they've broken down those barriers. They've broken down those walls between the two organizations, where there's more collaboration than just, "Hey, you've got a vulnerability. Go fix it." Right? There's actually a dialogue happening there on a more regular basis. There's education happening around best practices regarding coding and security, how to code properly to avoid these things in the future. And there's a two-way dialogue where development is also informing security about vulnerabilities that may or may not actually be applicable, depending on the nature of the code. Perhaps if there was a finding, but the code isn't actually exercising that area of the product, and so in practice, it's not really an exploitable finding, even if there is an underlying vulnerability there. Yeah. So again, I see those more mature organizations tend to have more of that collaboration there. But a lot of the legacy organizations that struggle, it's almost like they're from two different companies trying to interact.

Paul Holt

Yeah. No, it's interesting. One of the challenges of DevOps was obviously to get developers and operations to work closer together. And now with DevSecOps, it's getting developers, security, and operations to work together. And I guess if you've been successful in adopting DevOps practices, then you should have the cultural foundations on which you can do DevSecOps, hopefully.

Neil Levine

Yeah, although I think there's an interesting challenge there, which is that development operations, I mean, to Sam's reporting point, I think it's probably easier to get alignment between those teams because they're using the same, ultimately, I think the center of gravity was closer to them. Whereas security has a much broader remit. It's inherently always carrying a lot of the legacy stuff, too. So I think there may be some additional challenges with security in particular. Also, its mindset tends to be very risk-averse, and obviously for obvious reasons, whereas developers and lesser operations maybe. But certainly, the DevOps teams have learnt that you can move fast and break things is an okay model to have. If you've got the automation, you can resolve it. Whereas security teams, I don't think, they don't want to absorb a break things attitude because the cost can be so high. So I think there are some specific cultural maybe resistant factors, the security teams, which means that even if you've got DevOps, it still may be a challenge to bring in security in some sense and some things.

Paul Holt

Yeah. I guess they could also be different, just in how organizations arrange themselves. There could be different reporting structures, whereas security development operations could ultimately report up into the same hierarchy, whereas security typically would end up with a CISO type.

Neil Levine

Yeah, although it is interesting. I think once again, what's an interesting signifier is whether they have people tasked with security responsibilities actually outside the security organization, and it's like sometimes you have champions, right? You have a security champion inside the developer team or the operations, and folks who are not trained as security experts, but are essentially there as a frontline and have taken on the responsibility of trying to triage things for the security team and trying to actually be the first among equals when it comes to security within the developer or the operations organization. So I think, again, that's another sign that you're doing DevOps well, is actually where you've recognized that you do need to vary things up and you can't just have a homogenous group of developers and operations folks. Some of them do actually need to take on, without becoming true security experts, at least some security responsibilities within that team. So again, I think that that's a good sign that you're either well along the path or maturing towards DevSecOps.

Paul Holt

Yep. No, that makes sense. And we've heard this term, you hear this term of shift left, as organizations shift the security further left into the development process. Just thinking that through and what that means for organizations, how do we see that manifesting itself within the developer organizations themselves as we try and get developers more familiar with security? I don't know, Sam, if you want to take that one.

Sam White

Yeah, sure. I would say, really, shift left is all about giving the ownership for security at the point of where that vulnerability or where that security problem is potentially introduced. So if it's the developer writing the code, that's where we want to have that visibility into the security impact of what they're doing and any potential vulnerabilities. That saves the developers a lot of potential embarrassment, potential conflict. It reduces a lot of that back and forth with the security team. If you're actually able to bring that visibility into what the security impact of their code is right there in the forefront of the developers as early in that process as possible. While they're writing the code, when they're submitting their merge requests, when they're going through those review processes, all of that can happen well in advance of them actually being ready to ship the product. There are some aspects of security that don't necessarily belong with the actual development team, but I would say the underlying principle of shift left is really get the ownership for the security in alignment with the group that's actually owning that area of the product, whether it's the code, whether it's the infrastructure, whatever that may be.

Neil Levine

Yeah. I think one of the, shift left is obviously a marketing term, so it can be variously interpreted, but certainly Sam's identified the essence of it. But I think the thing that is most compelling certainly for senior members of an organization is it's essentially an economic argument, right? Which is if you identify things early, it tends to be cheaper and quicker to fix them. And so ultimately, shift left is an efficiency and economic argument, which is just the cost of effort is much lower, just because the fix is closer to a time of when it was created. So it's just developers haven't context switched out of that chunk of code or whatever it is they're working on. So it tends to be you can resolve it faster. Obviously, if you identify it earlier in the pipeline, you prevent things going into production and everybody ultimately wants to do that, too. But I think one of the, again, coming back to the previous question about sort of maturity and what is DevSecOps, I think for a lot of people, though, they think if I just do a check in CI/CD, I've done shift left and then I'm doing DevSecOps. And I think there's a little bit of a lowest common denominator effort here, which is I've shoved some things into my pipeline checks, into CI, and so here I'm doing DevSecOps. But it's not doing the check, it's actually fixing it and doing it in an efficient way such that the automation is there, that the speed of response is there. So if you haven't actually put in the back-end organizational infrastructure to actually do something with the data, then you've just got more data and you've may have failed fast, but you haven't resolved fast, and so you haven't got your automation and the true goal of what you want from a DevSecOps pipeline, which is still ship fast with high quality. But yeah, shift left is definitely, as Sam said, get the check where it's so the team who introduced the problem gets it as quickly as possible.

Paul Holt

Yeah. And as two organizations with Anchore and GitLab right at the center of this shift left mentality, what are a few practical examples of organizations trying to shift left the security into their development? Well, just give us a few examples of what you can do.

Sam White

Yeah. So from our end, the examples that I see obviously are organizations using the GitLab product to do that, but typically that would be not only running those checks as part of the CI pipeline, but actually surfacing those forefront to the developers as they write it, as they submit those MRs for review. And so they are able to see those results in line in their natural workflow that they would go through to submit an MR and have that reviewed and eventually merged into the code base. They can see that upfront, and they can make a decision, is this added risk that I've introduced newly into the code because of the changes that I've made, is that something that I'm willing to stand by and we want to continue moving forward with that, and I want to recommend that to the security team to still be merged into the code base anyway? Or was that just a mistake on my part of coding and I want to go back and fix that, and I'm going to correct that before I even start the review process, before anyone else has to get involved? And to Neil's point earlier, the time and efficiency savings, we've seen some companies that actually are implementing this, and they've saved just an incredible amount on the back end because those vulnerabilities start to be resolved by the developer who created them right upfront, and no one else has to get involved. So you save five, 10 other people's time who don't even see that vulnerability coming out on the back end of it. And in the end, what actually is being submitted to be merged in goes through an approval process, and those are things that the development team is confident that there's a reason. If there's a vulnerability there that's severe enough, there's a reason it's there and it's justifiable. Either it's a false positive and that code's not being executed and it doesn't apply in this situation, or it's absolutely necessary and the risk is worth the benefit that it provides.

Neil Levine

Yeah. So I think Sam's general point, which is that I think developers now are a lot more used to having commits thrown back at them. And it's not just security issues now, obviously. There's lots of other things that are checked for, and so I think security's managed to sort of slipstream itself into some of the automatic pull and merge request bounces that developers are now used to getting. So the sort of source code management tools or the source code management tools have been very good at implementing those, and I think just the whole pull and merge request kind of mentality has really lent itself to actually adding security checks into it. So that's definitely the area where developers experience it. And then the second area which we touched on before is in the CI/CD systems themselves, so when the build is happening. And again, a sort of interesting thing of how quickly do they get that back if your build takes a very long time, and again, to my earlier point, the developer may have already context switched and they can't remember what that piece of code was. But typically, I think failing builds where most mature teams who have been doing DevOps are used to having to worry about the nightly builds. Have they passed all the tests? Are they good to go? Can QA pull them down and test on them and so on and so forth?And so again, I think another easy area is just that the nightly build failed because of security checks, and so you've got to resolve those. That's your top priority because essentially the pipeline's been shut down. So I think that's the other area that most developers start to experience things. But when you sort of cross into the operational domain, again, this is where it's a slight sort of shift left aspect of it is worrying about at deployment time, which is distinct from runtime. So actually, when you're about to run your application, are there any security concerns at that point? Because obviously circumstances could have changed, that the application could've been absolutely 100% passing all of its tests and security tests at developer time and build time and when it was stored in an artifact registry or whatever it was. But actually when you're deploying it, we've now got new issues or there's maybe a policy change that organization has. So I think that deployment time is now sort of a new thing that the operations team is sort of aware of, because there's a lot of essentially static code checks you can do on your artifacts and your deployment scripts and so on. And then your traditional endpoint. At that point, it's runtime and it's very traditional. Yeah. Yeah, those are the three main areas, I'd say.

Paul Holt

Okay, fantastic. Yeah, there's a couple of points I want to drill down on there. But first, on a related note, there's a question just from somebody listening in here about, is there a position to take now to take a more proactive approach to the security in DevOps and adopt things like penetration testing with respect to your CI/CD pipelines and think of it like an attacker? What do you think about that type of approach?

Neil Levine

Yes, I can see there's, we've got another question about Codecov and all of the sort of alarm that that introduced. When was it, last week, I think, or the week before last. I'll keep track. There's so many breaches these days. No, I think just the actual software, the developer infrastructure tools, and the overall pipeline tools are now a major vector. SolarWinds highlighted this, obviously, that actually endpoint protection was, as the name implies, is just so focused on the end of where the application was run and realizing that actually the tools which are supporting the SDLC are a huge vector is just like, I think become-- SolarWinds has just driven everybody to sort of focus on that now. And I think this is one area where the industry is still a little bit far behind on this. I think two areas got surfaced out of this. There's this element of trust in the supply chain, and can you trust what you've got, and is there a sort of a chain of authority that you can work out what you have and where it comes from, and even once where it came from, where do they get it from, and so on and so forth, but then the actual developer infrastructure itself. And so yes, I think pen testing against your CI/CD systems and your registries and pretty much anything that a developer touches is just now table stakes. And as an industry, we're still catching up on that. But it's certainly the case that this does lend itself to a sort of everything as code approach, right? We certainly have policy as code is kind of fairly well understood, but actually even the pipeline jobs which are creating and doing your builds, that should be stored as code. You should be able to audit that. They shouldn't be random bash scripts that are just not in a source control management. Again, I think it just, everything should ideally be stored in your SCM so you can scan it statically there as much as possible. But then, yep, you have to pen test all of these platforms now. Everything that potentially can introduce a vulnerability, you've got to go pen test it. There's no doubt.

Sam White

Yeah, I view a big part of that as an effective pen test involves running multiple types of scans. If you've got SAST running, you've got DAST running, you've got fuzz testing, you've got your traditional dependency scanning running. All of these come together to help give a more holistic view. It's not really sustainable to run a whole bunch of manual pen tests against your application. Although that's good information, it tends to be pretty slow-going, and it's really hard to keep that up. It's just not scalable or sustainable. So I would say as far as pen testing goes, there's a huge place for that. I think the biggest area is in writing rules for some of these other tests that are out there and that are available, so you've got a good 360 degree of coverage on your applications that you're shipping out the door. As Neil mentioned, these recent attacks have brought into light that we need to consider more than just the code that we're writing and shipping. We also need to think about our supply chain. We even need to think about the infrastructure that we're building that in, the workstations that we're developing the code on. So there's kind of a pretty broad surface area there. I would say the underlying principles are, again, run as many different tests as you can and automate as much as possible, just so that it becomes sustainable and scalable. And that way the focus can shift more towards how do we write new rules? How do we add new automated tests, add additional checks into that coverage that we're already getting?

Paul Holt

Yeah. And so we start to head down the conversation around software supply chain. So let's just touch on that because that, again, I think is an equally timely topic given the executive order which was pushed out last week in the US. Neil and Sam, what do we mean when we talk about software supply chain, and why is that so important for our organizations nowadays?

Sam White

Yeah, so your supply chain is really anything that feeds into what you're delivering. And that actually can be far more extensive than you think initially. Just thinking outside of the box, you've got your initial code that you're developing. You're probably including some number of third-party libraries as part of that. Those typically undergo some degree of scrutiny. But as we've seen in these recent attacks, it's more than just that. It's the infrastructure that you're using to develop it. It's the infrastructure that you're using to test your code. As I mentioned, it's even the workstations that you're using to develop the code on. And so there's a lot around that. Everything that goes into getting that software built and shipped and delivered is a potential point of attack. And so you've got to think holistically about the security of all of those aspects and make sure that you're adequately scrutinizing them and applying a zero trust model across the board to every single component all along the way.

Neil Levine

Yeah. So I think if supply chain is just where stuff is coming from in general, then, again, to come back to it, it's both the infrastructure that you're using, where does it come from, who's providing it to me? SolarWinds was an infrastructure piece, essentially. But I think from an application, if you're writing the application, where are the ingredients that you're putting into the application coming from? I think people always tend to think of it as, "Well, it's all my code," but of course it isn't. It's dependencies, and this is where, in particular, the open source community has sort of had so much success over the past 20 years, but it's actually now realized we've brought along just a tremendous weakness along with it around security. So I think supply chain, it's equally off-the-shelf software. Can you trust what you're given? What does the vendor that you're interacting with do from a security perspective and understanding that? And that's a large part of this executive order is trying to give confidence as best as possible that your suppliers are adhering to best practices and having a certain level of competence. But then it's not just off the shelf. It's if you have no commercial relationship, can you trust the stuff that you're putting in? And I think what's particularly challenging in our space has been as we've moved to things like containers, we're now back with compiled languages like Go becoming popular after sort of the Python era. It's just the opaqueness of what you've got is now really high or low, depending on which way you look at it, I guess. But it is a high opacity because a developer may think they know what the dependencies they're putting in because they're listing them explicitly, but when it goes into a build, a whole bunch of other stuff are getting pulled in completely unseen to the developer and often unseen to anybody else because these containers can be lumpy, sort of monolithic binaries, essentially. So I think there's this sense of what's going on with my vendors and what's going on with the open source tools, and they have different challenges, and you can have different approaches to managing the supply chain when it comes to security with those two groups.

Sam White

Yeah. I would say that these things have really exposed those organizations that have been living by zero trust and those organizations that haven't. Zero trust is a bit of a marketing buzzword, but it's also a really good practical security principle to follow by. Verify everything that comes in, and even those things that we've learned in the past month or so here, even things that come from reputable companies, you still have to verify because there's not a guarantee that the contents of that binary that's being handed over are safe. So I suspect we're going to see overall more along the lines of the executive order that was passed recently, both in other countries and even in companies individually, where security teams are likely to start just applying deeper scrutiny in security practices for their software vendors. And that's going to be across the board, not just dependencies for the code that's being developed, but the software that's being used to develop that code. And even in the case of SolarWinds, like IT management software. Any software that's being used can be a point of attack. And so there's also likely, I suspect, to be a shift towards increased behavior monitoring where, okay, we can verify that this actually came from a trusted entity, but can we verify that the contents of that haven't been tampered with? In a lot of ways, the insider threat, if you will, is very real, whether it's malicious or even just accidental or a compromised account that is able to go in and change some of that source code. And so a company can do their best to protect their own employees from that and monitor that. But that's another thing when you're accepting software from a third party, you don't necessarily have full line of sight into all of their practices. And how susceptible are their emails to phishing campaign? Are their employees to phishing campaigns, and just exactly what protections do they have on their networks to make sure that attackers haven't gotten a foothold? And so I think the solution to all of that tends to point more towards behavioral monitoring type approaches to these things, where a piece of software, whatever that may be and whatever its purpose, comes in, and it goes through some sort of scrutiny or analysis before it's accepted, and possibly even scrutiny analysis after it's accepted on an ongoing basis, just to monitor its behavior and verify it on a continual basis to make sure it's behaving in an appropriate way.

Neil Levine

Yeah. So I think- I think it's a good point... yeah, I think there's two things there. So, I think, yes, in terms of, you're going to see a whole bunch of vendors focusing on tamper detection throughout this SDLC now. I think there's no doubt about it. There's going to be a really interesting question as to how do you work out whether something's been tampered with because the software changes as it's going through the pipeline. It's just inherent as you're deploying things and credentials get added into a system or there's configurations which are added. So detecting tampering is definitely going to be a challenge, but it's absolutely something that I think we, as vendors, are definitely going to be focusing on. I think the zero trust one is in some ways actually a harder one because there is, especially as with the open source sort of commons, in some instances, it's going to be very difficult to actually validate, to feel confidence that you know who's behind something, because often there is nobody behind it. It's somebody who wrote something five years ago, and they disappeared off, and they're not responsible for it anymore. So I think this is where it's interesting to see the Linux Foundation in particular, as you know, with Sigstore and the work that the OpenSSF is doing, is really trying to sort of raise the level of the security status amongst these open source projects. But ultimately, there's always going to be a gap there, I think, which is going to be difficult for lots of organizations to address with a zero trust model, unless they literally rebuild every piece of software they bring in, or they vet every line of code. There's going to be some challenges there. And so I think it goes back to Sam's point, which is even if you try to go for a zero trust model, you're still going to have to hit everything you bring into your domain with all the testing you can to sort of raise your level of confidence. But it's never going to be complete. And so, again, I think this comes back to this slight variation or shift left, but you're going to have to accept that there will be risks somewhere. You're bringing them down as much as possible, but you've got to be continuously looking at this stuff over and over again. There's never a total seal of approval you're going to have on anything. And so, that continuous testing you're doing through shift left, as soon as there is a problem that's aware, you find it immediately, and that speed of response is going to come. That doesn't change in terms of urgency, importance for your planning.

Sam White

Yeah. Security always has and probably always will be a little bit of a cat and mouse game. It's really just a question of can you be faster and more agile than your attackers. Right. It's almost impossible to have a perfectly impenetrable system of any kind. Almost everything is hackable with enough time and resources. And so it's really a question of, given the events over the last month, how do organizations then go and up-level their game to take that next step and regain the advantage? And, yeah, it'll be interesting to see how all of that unfolds.

Paul Holt

Yeah. And we'll get to that in terms of key takeaways. But interested, particularly as organizations continue just to use more and more open source within their own application development, what's the best practice? What's the minimum threshold in terms of checks and due diligence that people should be doing as they bring open source code into their own CI/CD workflows?

Neil Levine

Well, I think the one thing that seems to be on the rise, and is sort of important, is you actually essentially curate your own library of open source content. I think the first thing that we see more mature organizations doing is not allowing people to just pull things down randomly off the internet which was a de facto model and has been, and still is, for many organizations, that the more well-resourced teams, should we say this, developers should be able to say, "I want to use this library." But as quickly as possible, you bring it in, you scan it, and you essentially take responsibility for that open source project and, or the package, or the library, whatever it is, and host it in a trusted repository inside your organization. By taking responsibility, that means you've got to stay on top of bringing the latest version and making sure that gets scanned and made available. And then, implementing the checks and policy rules to ensure that people can't go out to random registries. They have to use the secure registries that you have inside your organization. So that's one practice we've seen. We're starting to see a bit more due diligence even before contents may be brought inside the four walls of actually saying, "Look, is there a security mailing list or contact for this project? When was the last commit made? Is there a governance structure?" Sort of checking who has commit rights. There's a little bit more due diligence on the security health of an open source project. I think at the moment, that's still far too manual. It's not actually easy to automate that. And I think that's, again, part of what the OpenSSF is trying to do is give badges and other things so you can have essentially an automated way of determining that no, this meets a certain level of competency when it comes to security response. So these are early things to do, but, yeah, I think just preventing people putting stuff down from anywhere they want to and just getting insight into what's actually, maybe even before you do any of this, trying to see what are you even using. Coming back into the SolarWinds thing, do you know what libraries you have? Do you know what third-party code you've managed to pull in is often the first step. Yeah.

Paul Holt

Sam, anything you guys would add to that?

Sam White

Yeah. One of the benefits of open source is that you have access to the code. You don't get that with closed source. And so you actually can apply the zero trust model a little bit better with an open source project than you can with a closed source one, because you have visibility. There's some level of transparency there. Typically, you are able to seethe past commits for that project. You are able to see the actual code of that project. And so, like Neil mentioned, just a good starting point, if you're doing nothing else, at least do some form of security review of the project to get a sense for how security conscious are the maintainers of that project and what's their level of security maturity. Again, just to echo what Neil said, next steps would be running some basic scans against that. Are there open, known CVEs against that project? If you have the means, bring that code, make a clone of it, and sort in-house so that you can run additional scans against it, SAST scans, DAST scans. You can run some of those scans because you actually have access to the source code. So there's a lot of power there. And, lastly, I think this would be beyond first steps, but the long term would be implementing some more behavior type monitoring, as we discussed earlier, to actually observe the behavior once it was compiled and running in your environment.

Paul Holt

Yeah. Now, I know we collectively have a very interesting mutual client in the United States DoD, who have done something very, very similar in their creation of a hardened repository called Iron Bank. I don't know, Neil, if you just want to say a few words about that, because that I'm sure would be of interest to some of the people listening in.

Neil Levine

Yeah, DoD is probably one of the more paranoid organizations out there. But yes, they implemented a DevSecOps methodology and created a platform to support it, and actually even built a reference architecture for others to try and copy. They really tried to seed this approach inside the federal government in the US. And, yeah, the Iron Bank contains commercial software and open source software. But yes, essentially, they try and rebuild everything. They don't trust a container that's given to them. They want to rebuild it from scratch themselves. And so, they need access to all of the components to do so. So, it's quite heavy. There's a lot of resources to do that, because there's an awful lot of open source software, a lot of commercial projects out there. But they're a resource organization to be able to do that. Sometimes that might be too heavy a lift for a lot of other organizations who ultimately, I think it's what do you trust least, focus on that first. You can't go and rebuild everything from scratch. You're going to have to accept some content is given to you. So again, this comes back to if you can push the burden onto the supplier to prove their competency, and actually, you have to sort of trust their internal transmission of their competence demonstration over to you. But the DoD is essentially saying, "Take it or leave it. We're not going to have any of that. We don't trust anything. We'll rebuild everything from scratch. We're going to go inside and look at all the contents." And they try to do it in obviously, as machine-driven way as possible, but they still have humans. At the end of the day, you often have humans going, looking through, and making a judgment call around certain things. All software has some vulnerability of one type or other. It's just what is the risk, which is what do you still have humans essentially trying to do? But yeah, so it's an interesting model, and the reference architecture is public, so people can see how they're doing it. And enterprises can maybe learn and take some approaches, but they certainly represent the extreme end of zero trust. They really don't trust anything.

Paul Holt

So on a similar theme, and where I think we were starting to head down this track, we're starting to hear this term software bill of materials. And I think, actually, once again, a couple of weeks ago, in the executive order that was published in the US, the software bill of materials was starting to be referenced as a critical piece of data which software vendors will need to provide. Perhaps, I don't know, so Sam or Neil, either one of you could take an explanation of what we mean by the software bill of materials, and why is it important?

Sam White

Yeah, sure. So I can take a pass at that. Really here, we're just looking for a comprehensive listing of all of the different components that have gone into producing a software package. And so that would include any third-party libraries, any other dependencies, and their dependencies that may be involved in that. If it's a containerized image, obviously you've got packages installed in the underlying container operating system. So you want to have a listing of those, as well as, again, the dependencies for the software that you wrote. It's not a terribly hard list to pull together, but having that list just shows that you have an awareness of everything that was included. And, if you're not able to put that list together and you don't even know what's going into your application, obviously that's a red flag. That's concerning. And so, an initial starting point is just to put together that list so you have a comprehensive inventory of what's comprising that software.

Neil Levine

Yeah, that's right. And I think where the industry's working on right now is trying to create some standards, because as Sam said, creating this software bill of materials, it's not difficult. It takes effort, but it's not rocket science. But actually, it's what you do with the bill of materials is the most important thing. And so there's a movement to try and standardize some formats, because ultimately you want to be able to track this across domains, across teams, across organizations, and everything else here. So, there'sSPDX and CycloneDX are probably the two main standards, and I think you're going to hear a lot more about those from vendors. We produce SBOMs in this format for customers or partners to look at.And then coming back to what we were talking about earlier with tampering, I think, the SBOM, again, this is where the SBOM gives you something to compare and contrast between stages in the pipeline to see has things changed, and then to be able to ask the question, why has it changed? Your SBOM won't necessarily be static from one end to the other. That's, again, the build process is inherently going to change some things. But it's, I think, going to be an interesting source of truth around have things changed which we expected to change, and did they change in the right way, versus there's unexpected change. Why did the fingerprint or the hash on a certain binary change when it shouldn't have? There's something strange there. So I think SBOMs are going to help with some of this tamper detection, and spotting for variation over time. But yeah, the big thing in EO is that, the executive order is that, again, to try and just raise transparency and visibility about what people have. It's just to help organizations get better visibility. But before you do anything else, just do you know what you're running? I know a lot of the organizations, their first questions were, "Do we run SolarWinds' agent, which is compromised?" And that wasn't an easy question for many organizations to answer quickly without having to go through many lines of organization. Whereas an SBOM can sort of give you that answer very, very quickly. So even from a response perspective, they can be useful.

Paul Holt

Yeah. So I guess, the SBOM is going to be foundational, really, to understand your software supply chain. It feels as though it's going to be a really foundational component.

Neil Levine

Yeah. The analogy is ingredients, right? Which is, you've got to make sure that if you're producing food, that at any point it's going through a factory which has got peanuts in it, that that's immediately identified because you've got to advertise that that's a risk on your product, so people with allergies can know that. And so, again, it's just that transparency end to end, so any risks can be surfaced and addressed, either to the end consumer or just even as you as a producer, so you get better insight. So, back to our point earlier, if you're using an open source project which you know is inherently insecure, like an old version of OpenSSL or whatever it is, that you immediately see that and can respond to that if your tests have failed to spot something.

Paul Holt

Yeah. Great. So I want to respond to, well, actually one of the poll questions we did on the Slack channel, which suggests that most of the people listening are doing security scans only in their CI/CD process. That's where the majority of scans are taking place. Just let me pose the question as to why you should potentially broaden that and look at either ends of the workflow, and the importance of doing scans elsewhere within your software development processes. Sam, do you want to take that one? I think that probably pulls together quite a few of the things we've already spoken about already.

Sam White

Sure. Yeah, I can talk about that. So, that is interesting and actually, in some ways, that response to the poll is encouraging because if I were to pick any one place to do it, that's a really good place to do it. Because if you're doing it as part of your CI/CD workflow, it's automated, it's part of the development process, and it's happening before it goes out to production. If there's anywhere to add next after that, I would say, is to make sure that you're scanning regularly in production, because it's one thing to make sure that everything you ship is clean, and it's ready to go, and it goes out the door clean. The challenge is that the security world doesn't stop. The attackers don't stop. Vulnerabilities don't stop. And those packages, the software that you ship with today, tomorrow, somebody's going to find a vulnerability in some component of that, and all of a sudden it's insecure again. And I've talked with organizations that are a little bit embarrassed to mention it, but they actually have code that is two, three-plus years old, and it's still out there running in production, and it hasn't been touched, it hasn't been updated, it hasn't been scanned. And guaranteed, the number of security vulnerabilities that exist in that application are sky high. And so, CI/CD is an absolutely great place to start. Like I said, that result actually is fairly encouraging to me. I would say the next step then for most of the viewers here would be start thinking about that production environment and scanning that. How do we work to shift that left to make sure that the developers are aware of what's actually running today in production and what vulnerabilities exist there so that, again, that can get back to the team that's responsible for making the fix?

Neil Levine

Yeah. No, I agree. CI/CD is definitely, if there's only one thing you can check off the list, that's a pretty good one. I think coming back to, I think, what we said at the very beginning, or I may have commented at the beginning, though, is shift left doesn't mean, "Hey, I'm doing endpoint, and I've got some checks running in CI/CD." It really is a much more holistic approach. And so, as long as that's not the end of the journey for organizations, that they feel we're doing DevSecOps, we've done a shift left thing. We've got a CI/CD check, that actually, ultimately, and this is where shift left may have not been the best term, but really, you want to be doing security continuously everywhere, right? That's the goal here, is that every step along the way, you've got some security checks. Now, they may be fatter in some areas, thinner in others based on the context, but really, you need to expand across the spectrum and just get away from just runtime, which I think we're starting to do now. And I think there is a danger that people overcorrect and say, "Hey, we're just doing everything in CI/CD. We're good." You really do need to check everywhere at all times. I think this continuous security is actually, that's the sort of defense in depth effect, really there. Whereas even if it's the same check you want to be doing it. Scan the registry. The content could be tampered with in a registry. Scan when you're deploying. Things could've changed since the build time. CVEs are coming out at such a fast rate now, even if it's a day between the build and the deploy, something could've happened. So very encouraging, but I would hope that that's not the end of the journey for people, and that's the beginning.

Paul Holt

Yeah. And I guess one of the frustrations that we hear is obviously things like false positives and just too much information, and really working out what's important and what's not important. How should organizations start thinking about really working out what should we be focused on, and what actually is not that important? Neil, do you want to take that one? I know that's probably-

Neil Levine

Yeah. False positives, they're the bane of security teams. They're just a fact of life, until we have Skynet taking care of this, it's always going to be something we're going to be dealing with here. The burden is definitely on the vendors here. We've got to do our best to try and reduce this as much as possible, and it's a constant process to try and get better at it. I think, and this is where I'll create a rod for my own back, but I think a lot of this is around reporting and the metrics. What I'm starting to see, which is really encouraging as part of RFIs is like, are there business metrics which can show how many false positives you're generating and how quickly we resolve those, and if they're getting better over time? So I think there is a sense of don't just accept the pain and let the user deal with it as a natural byproduct, but actually show that you're getting better and how you're getting better. It doesn't mean you're going to get down to zero. I don't think that's ever practical, but at least that you're cumulatively working through the pain, even if it's in thematic areas that we've got better at, say Java, or you've got better with your containers or whatever it is. So, there's a lot you can do around this, with giving customers automated policy recommendations or helping developers write things, maintain their metadata in the right way, which can then ensure that we can do detection more efficiently. But yeah, I would say, and as my sales team are sort of already girding themselves here, but ask your account reps, "What do you do around this? How do you get better? How can you show me that you're getting better?" The burden is absolutely on us as vendors to try and deliver that to users.

Sam White

Yeah. Yeah, I agree. I think a lot of the burden is on the vendors to help improve this. In the long run, I see a lot of the false positive rates dropping as users start to consolidate their scanning tools. Right now we see a pretty diverse array of tools that are out there, and there's a lot of different interfaces. There's a lot of different places to go to explore these vulnerabilities. As you bring all of that into a more centralized place where you've got all of those results in one location, you're able to start doing cross-correlation and even pair that with observing things along the way of the CI/CD pipeline. As you're executing your tests, was this library ever actually utilized, or is it just present there on the system because it's required to be as a dependency? But is that code ever actually even utilized, right? Or even in production, being aware of and knowing what firewall rules are in place, knowing what intrusion detection and intrusion prevention rules you have in place. Those can be mitigating factors that may make even a valid vulnerability non-exploitable. And so, I don't think the industry is there yet, but in the long term, I see this converging in a way where all of that different intelligence comes together to help inform the vulnerabilities that are being found by the original scanners and to help add just some additional light there to say, "Look, this is a critical vulnerability, but by the way, we saw that this code was never exercised during any of your tests, and there's this mitigating factor in production." So maybe it is a critical, but priority-wise, you want to drop that really low on your list of things to take care of because it probably is, in this case, a false positive or at least a non-exploitable vulnerability.

Neil Levine

Yeah. Yeah. We'll give the standard vendor sort of magic pixie dust around, this is one area where at least we at Anchore anyway, where machine learning can definitely help with some of this, right? Machine learning's very good at pattern recognition and classification and such things like this, and so I think there are good opportunities to look at this kind of data to see where it has been a false positive across many other customers because they flagged it, and then bring that back into the wider customer base. So there are some techniques where machine learning can certainly help with this. Both the correlation point that Sam raised, which is absolutely critical, but just to see how users respond and how the human brain is analyzing and doing the classification to determine it's a false positive, and then use that aggregate set of information from end users to flag things up earlier. So some interesting things emerging on the technology side.

Paul Holt

Yeah. And I know we spend a lot of time talking about vulnerabilities, but in the very short time we've got left, in terms of other things that we should be looking at as we go through our CI/CD process from a security perspective, what else do we need to be considering other than just vulnerabilities?

Sam White

Oof. There's a lot. I'd like to take that one. So yeah, vulnerabilities are important. Configuration is also important, right? So with the shift to policy as code, infrastructure as code, code as code, right? Pretty soon everything is headed towards code. That's just the direction of things, and so being able to do that additional scanning and say, "Look, has somebody opened a network port? Has somebody modified their Kubernetes instance to expose a service that really shouldn't be exposed?"Doing some of those basic checks, there are vulnerabilities, but there are also just security best practices and hardening techniques, I would say, that also need to be monitored and watched for. And that applies not only to the applications we're developing and pushing to production, but again, as we talked about earlier, that applies all the way across the board to the tools we're using to run our testing coverage, to the SCM tool that we're using, to our workstations themselves, even to our IDE. Does that have open loopholes? If I'm working from home and my home firewall is wide open and I've got a service running, maybe I've got a home Apache server that I'm running out of my own home that's riddled with vulnerabilities because it hasn't been updated, that's a huge entry point for an attacker to come in and gain access to my account and exploit that. So, there's a lot out there. I think you could easily start to try to boil the ocean, trying to cover all of the security areas that could potentially be coverage gaps in an organization. The key there is just to prioritize them and start with the most likely, the highest coverage areas that you can action on to help reduce that security exposure.

Neil Levine

Yeah, I think one of the things like SAST and DAST and things, it's very good at just looking for an objective fact of, "This is bad. You should not do this." I think Sam touched on this with configuration is like every single software project has a set of best practices about how to secure it. And often right now, they're scattered across the product documentation or they're just culturally known amongst its users who absorb it through time. And I think with this trend to everything going into code, literally every configuration option being managed statically inside your source code, it's going to lend itself to actually doing some good objective security testing of like, have you got encryption turned on everywhere? Have you turned off all of the default? Have you changed all your defaults? That's really just a good one. It's like, have you changed your default password in all your tools? What if the password's actually held in a code file you can scan? So I think, again, best practices is a generic category. It's like actually the configuration for all these applications. I think that's still the challenge we see with companies. I would hate to be running an organization because there's so much technology, there's so many different projects you've got to be an expert on now. It's no longer the LAMP stack, right? It's like a huge cornucopia of tools. Understanding the security best practice for all of those is really hard. And again, I think that's the area where I think if you're going to put your effort after you've done your vulnerabilities and your zero trust networking, it's actually, are you doing the right things in all your applications and trying to do that in an automated way so you're scanning, detecting when you have bad best practices is maybe the next challenge for us to climb.

Paul Holt

Fantastic. Great. Guys, we've run out of time. It feels as though, actually, we could probably go for another hour, but alas, our time's coming up. So Sam, it's been an absolute pleasure. Neil, thank you so much. I think if anybody who's listening in wants to continue the discussion, please swing by the GitLab Slack channel or also the Anchore Slack channel, and we'll be very, very happy to pick up the conversation over there. But thank you for listening. I hope it was a worthwhile hour spent. And as I say, if you want to continue the conversation, please drop by the relevant Slack channels. Thank you.

Sam White

Thank you.