Sal Padilla

Agenda. What we'll talk about here a little bit: life before open source, if anybody remembers that. I do, but how could that be? Because I'm only 30. Risks with open source, high-level stats, reality, the journey from no open source, current state, where we are, key takeaways, and my favorite, open discussion. I love open discussion because I get to learn from you as much as you learn from me.

So who remembers life before open source, right? Yeah. Yes. I'm not the oldest guy in the room. Fantastic. Life was simpler back then, right? You wrote a ton of code, you tested it, hopefully, and you deployed it, probably to a mainframe. I remember that.

So what did you think about using open source, right? What were your thoughts around open source and using open source?

Audience: It's not secure.

Sal Padilla: Yeah. Exactly. Embedded hacker code, right? Legal to use, maybe, right? Who supports it, right? Who supports that open source? Is open source good or bad? Is it safe or vulnerable? In the spirit of Halloween, is it a trick or a treat, right?

I had one colleague ask when we were thinking about using open source, did I review every line of code in the open source component? Yeah. That was exactly my reaction. He didn't care for that back then.

So I work for Duke Energy. It is 150-plus years old, one of the largest power holding companies in the U.S. We have approximately 7.6 million customers. We generate almost 50,000 megawatts of electricity. We actually have gas. We have about 1.6 million gas customers in four states, and we operate six nuclear plants. So security is quite important to us, actually.

So I'm the enterprise DevOps architect/engineer/evangelist. Probably a few of those in here. I've been doing this for over 30 years in IT, and 10 of those about DevOps. So I've been doing DevOps for quite a while, when it first started.

So life before open source, right? We had basically 0% of your code was from external suppliers, right? Open source components, open source libraries. All of your code was written from scratch, right? Then we had this mind shift when we started seeing open source. It didn't make sense anymore to write code for functions that's already been written, right? Why not just reuse this open source code?

But what drove us to open source? With open source, of course, we didn't have to write as much code, right? We were more efficient, faster, and we could focus on the business logic, the logic, the code that really mattered, right? And all that silly code connecting to databases, reading data, reading files, that was all taken care of for us with open source, right?

So where are we now? Now, life after open source. So now we went from zero to almost 85%. Almost 85% of modern application code comes from open source libraries, which is quite a lot, actually. The average enterprise will download 500,000 components annually from about 3,000 open source suppliers. Of that, 30,000 will have known vulnerabilities, which is really shocking.

In the Java world, 146 billion download requests for Java components, and 11 billion package downloads for JavaScript from NPM, which is our new favorite library. On average, developers have access to more than, or access more than, 21,448 new open source components released every day since the beginning of 2018. 21,000 a day.

So your average developer will download 60,000 packages, about 6,000-some packages annually. Being a developer, that's probably true. I know that's true because I download a lot. A little over 30,000 with known vulnerabilities, a little over half. Half of the downloads have known vulnerabilities. Maybe this wasn't such a great idea.

The good news is developers believe security is important. I know this to be true because I talk to them. They are conscious of what security means to the company. The bad news is they don't have time to spend on it. Who here works for a company that has embraced Agile? Right. And what do they focus on?

Audience: Features.

Sal Padilla: Features. New, cool features. Security is not cool. That's the problem. But it is important. But features, that's all they ever talk about. So how do we protect the enterprise in this new paradigm?

So everyone's familiar with DevOps, right? DevOps sits between the developers and operations and provides a system for continuous integration and continuous delivery. Build it faster, deploy it faster. DevSecOps is about building security into the DevOps pipeline, baking security into the pipeline.

There's two aspects of DevSecOps. There's the static analysis within the pipeline, right? You have SAST, static application security testing scanning; OSS, open-source software security scanning; and to some extent, code quality scanning. Then there's the dynamic aspect of DevSecOps, the runtime aspect, where you have tools like DAST, RASP, IAST, WAF. I'm not going to talk about that part. I'm going to focus more on the tools within the pipeline, the static part, because we don't have time for the dynamic.

So a typical build, typical Jenkins build, right? Here is a typical build. We check out the code, we build it, we run unit tests, and then we run three scans. This is how we do it today. This is current state. We run a code quality scan; a static application security testing scan, which checks for unsecure coding written by your developers; and an open-source software or software composition analysis scan from Sonatype Nexus IQ. So we run all those in parallel. You pass the quality gate. You go build a Docker image. You make it to the end, you're good to go.

So with every project build, a project team and build manager, if you have a build manager, can view the results of the OSS scan and act on the findings in the dashboard for the build. From within the build dashboard, they can click on the link to view report and go directly into Nexus IQ and get more information on the scan results. Application teams and cybersecurity teams can view the scan results too and view details of a particular vulnerability if they so wish.

So let's look at a sample vulnerability, sample component found to have a vulnerability. Developer or cybersecurity analyst can view the graph of all the versions available of that component. So right there. They can see which version is currently being used by the application and the highest policy threat. But wait, there's a clean version at 3.4.1. The developer can choose that version for their application, and 3.4.1 has no policy threat. Use that version. There is a caveat to this, though, by the way. Sometimes a component may not have a clean version of that component, or it can be a transitive dependency that the developer may not be able to change. So it's not perfect, but at least the things you can fix, you should fix.

If you can't remediate the vulnerability in your application, the application team can request a waiver. So our process is the application team submits a waiver to either sourcing or compliance. Sourcing and compliance can waive the finding and include documentation about the waiver so we can make this issue go away. For example, sourcing may have bought the license for the component, and we may waive it and then document the contract ID for the component for the purchase in the waiver so that we can link the two systems together. So it wasn't just waived just for the heck of it.

So as you can see, we have two main governance groups. We have sourcing and compliance. I'm not sure what you guys call it in your companies, but those are the folks that work with our licensing and purchasing. And cybersecurity. Everyone has cybersecurity. Cybersecurity for security vulnerabilities and sourcing for license violations. This tool will scan for both. That is very important. The dashboard allows you to filter for either one. So the cybersecurity folks filter on their security components, and sourcing filters on the licensing issues.

From a governance and enforcement perspective, this is what sourcing and compliance and cybersecurity look at, this dashboard. And our goal is to keep this dashboard clean. In essence, this is our open-source vulnerability management tool. This is how we keep track of all the open source in the company and what our risk exposure is.

Within the tool, you can see what policies are being enforced. App teams, compliance, and cybersecurity teams can see all of this, what policies are being enforced. We can set alerts to compliance and cybersecurity when an app team reaches a certain stage in their life cycle. When an application team tries to go to release stage, for example, the scan fails and the issues have to be addressed. You can't get to production. Ideally, it would never get to this.

I touched on this a little bit, but when we talk about open source, we almost always talk about security, but we always forget about the licensing. We forget about the licensing sometimes. But licensing could have potential huge financial impact to your company, so you got to really watch that, too. It's not just security, but it's also the licensing.

So who owns remediation? So now that we know the vulnerabilities, who do we need to discuss remediation with? This dashboard allows us to identify which application was a source of the vulnerability so we can have a discussion with an app team. But imagine what if the CIO of your company asked you if the company was vulnerable to Struts 2 or EventStream from NPM. What would you say? Right. Fortunately, I was not asked that, but what would you say? How would you know? Right.

Nexus IQ gives us a bill of materials inventory, and we can see if, A, the component is in the ecosystem, and B, where is it? So now I can answer that question. No, we don't have it; or yes, we have it, and we know the application it's in, and we're going to fix it.

Other things we do is we're educating our developers on shifting security left, right. We're educating them on available tools that they can use while they're developing their applications and getting them into the habit of writing secure code and using safe open source components. So they can see the same component details in their IDE that they can see in Nexus IQ. So while they're developing, they're making intelligent choices, and hopefully it's not getting in before it even gets to build, right.

Other tool within our pipeline is code quality scanning. So the app teams can see on their dashboard for the app and quickly assess their technical debt. The other scan we do to round out our DevSecOps within the pipeline is static application security testing, which focuses on secure coding practices and OWASP Top 10, among other things.

So where are we now? Well, we just finished educating and training our sourcing and compliance team. We're training our cybersecurity team on these new tools and processes. We're always educating and socializing these tools within the development community. And we're moving to active governance and enforcement. So you're warned. Now we're going to really start failing things and start encouraging you to fix your applications.

So here's some key takeaways that I've experienced while doing this. Sourcing, compliance, and cybersecurity will have new roles and responsibilities. They may not have experience with open source licensing, for example. I know when I worked with the compliance folks, they didn't really get how open source licensing worked or how it got into the company, and I had to bring them up to speed on all of that. That was something they didn't understand: the dynamics of application development, how components can just come in any time of the day and it may have a license vulnerability, license violation. Or they may not be staffed to handle the new workload. So there is going to be additional workload. They might not have the staff for it, so you have to account for that.

Sourcing/compliance don't need to be engaged at the first occurrence. Give applications time to remediate. So you find something, don't go over there first day and beat them with a wet noodle because they pulled in something bad. They see it, they're working on it, hopefully they're finding a good component library to use and working it into their application. Or in their case, putting it on a backlog and prioritizing it.

So how do you know your company's threat risk without a tool like this?

The last thing that I actually heard this morning in the keynote was: convincing product owners that security is more important than features has opportunities. They do not yet quite grasp, the product owners, the importance of security versus features, and they're putting features way ahead in priority rather than security. And I hear this all the time with the project teams, the app teams that I work with. Like, "Yeah, we know about it, but our product owner hasn't told us to work on it, so we didn't fix it." That is something that I'm curious how would you deal with that? I'm getting to the open discussion part, and that's one of the questions. How would you deal with that, convincing product owners to emphasize putting more priority on-

Q&A

Audience: Risk registries.

Sal Padilla: Risk registries? Risk registries. I'm not sure if I'm familiar with that. What does that mean? Speak up. Sorry.

Audience: You document the risk, put it down that it's in there, and for them to ignore it, they're explicitly saying, "I'm accepting that risk."

Audience: Someone signs off.

Audience: Someone signs off on it. When it blows up in their face, it's theirs.

Sal Padilla: Okay. That's a good idea. Risk registries. Risk registries. Documented, right?

Audience: Yep. They put their name on the line and their jobs, essentially, on the line.

Sal Padilla: I like that. Any idea? Yes.

Audience: One of the early principles of Agile was that a user story was only that had customer value. All framework-level concerns, non-customer-value concerns, were actually baked into the development. You would bake it into a story that had customer, and you would work incrementally on those. SAFe calls them non-functional requirements.

Sal Padilla: Non-functional requirements.

Audience: Yeah. But with the advent of SAFe, which is imposition of project management ceremonies around what was Agile development practices, now we track everything and it makes it really hard. In the early days of Agile, we never had these discussions. Because the product owners focused on value, and we would just weave in framework-level concerns into our velocity. It was non-negotiable. You weaved it in.

Sal Padilla: But how did the product owner- how does the product owner say you should work on that?

Audience: They don't. The engineers and infosec says you work on it, and you work on it. The product owner is not involved in the conversation around that.

Audience: They decide what operating system is used on your servers.

Sal Padilla: So this other group has as much clout as the product owner to decide what the team works on?

Audience: Because that's what an empowered Agile DevOps team is. The team is empowered to make all the decisions. Have your own budget or a percentage of the spread or something, however you want to do it, but that's what I'm accustomed to as well. You have 15% or whatever to spend on technical debt. The team decides what the technical debt is that they want to work on because who are they going to ask? What other subject matter expert is there?

Sal Padilla: Okay. So I hear empower the DevSecOps, cybersecurity, essentially, and compliance team as the same level of power as your product owners.

Audience: I wouldn't say that. That sounds more like they are a center of excellence or whatever. What they should do is get whatever it is in their head codified into policies in your organization. You can scale the people, but you can't scale the policy.

Audience: As long as there's a role on your Agile team, right? Dan North is our- People are already talking about merging that role into the team as a whole, and it's just a hat someone wears. It's like certified Scrum Masters are cons with a lot of Agile teams now, and the team just shares that responsibility.

Sal Padilla: So what's the incentive in that model for the developers to prioritize security into their sprints?

Audience: Well, it's the same as the incentive is to write quality code. You have to teach developers to be good developers.

Sal Padilla: That's right.

Audience: But you bring that in. I remember the days when there were entire QA organizations that I was working with, and you threw code of imprecise, unknown quality over a wall to a bunch of testers. And then we just absorbed SDETs into the teams. And developers who weren't SDETs couldn't ship quality code. SDETs added to the team had huge value to the quality of the code. And I think security people on the team add huge value to the quality of the code. Ops people, same thing.

Sal Padilla: Yes.

Audience: Where we see it actually get sideways, to your point, is if you give the product owner too much power, they'll hold your product hostage from getting better. They'll add feature after feature. But a good product owner will do exactly what you said, which is democratize themselves to the point where they really should just be a member of the team. Same with a Scrum master. We've actually had pretty rough going product model because we've defined that there is a Scrum master and there is a product owner, and if those two don't get along, it doesn't matter how good your engineering team is. But in places where we don't have that defined, we've actually had success. Riot Games, I highly recommend reading the story of Riot Games. For two years, they've been dealing without product owners and Scrum masters as the only two specialized roles on an Agile development team, and wondered why. It was counterproductive, and they got better.

Sal Padilla: I don't necessarily disagree. I am not a big capital-A guy. Yes.

Audience: I think security has to be prioritized at a higher level, the solution train level or management level. Because the product owner is in charge of the backlog, but he's not in charge of the product. So the management has to prioritize security higher and say, "Okay, I need security more than features."

Sal Padilla: Right. Like from this morning, Bill Gates, right? I think I heard it right.

Audience: I did too.

Sal Padilla: Oh, I think they said Bill Gates. Yeah. Feature freeze. If you have to choose between security versus features, choose security. We're not there yet. We got to get there because you saw, right? 50% has a known vulnerability.

Audience: The key thing that you just said, and I'll repeat that, is it wasn't synonymous. Security is not a feature.

Sal Padilla: Right.

Audience: So security is built in. It just should be assumed as part of it. It's incorporated. It's not a feature because we all know, we've been in the business long enough, what's the first thing that gets deprecated or kicked out in release cycles? Documentation and quality. It can't be a feature. Security can't be a feature.

Sal Padilla: Can't be security versus features?

Audience: I actually have a different problem, the opposite problem. So we work with healthcare data. We're PHI compliant. We work with HIPAA, and we process billions of transactions every year, of healthcare transactions in our product, clearinghouse. Our InfoSec team is designed for on-premise software delivery, but it's not really... They have lots of manual. So we have an InfoSec team that in order to make a release, and that's now flexing its muscles, cloud development has had a period of freedom for a while. The InfoSec team is flexing its muscles and putting brakes on any release into the cloud that doesn't meet their review, which takes a minimum of two months.

Sal Padilla: Mm.

Audience: Requires us to produce all of the waterfall-style artifacts. They take OWASP and turn it into a bullet list of disconnected requirements that's 100 items long. I don't know that I want to go back to there. I don't know. We've been there.

Sal Padilla: How did you fix that?

Audience: We just went away. The pendulum swung the other way.

Sal Padilla: Legacy versus brownfield, we won't go there. Only got 30 more minutes. Any other questions? Yes.

Audience: DevSecOps. What do you say to the people who say if you have to say DevSecOps, you were doing it wrong in the first place?

Sal Padilla: If I have to say DevSecOps? Oh, because it wasn't built in?

Audience: Because DevSecOps is a thing rather than just secure DevOps. The joke is like, so then start calling it BizDevSecOps.

Audience: I just call it DevQAOps.

Sal Padilla: Yeah. Because we like to make things up in our industry. We call things that we used to do 10, 20 years ago some other new because for marketing. No offense there, Matt. Anything? Time is up. I finished. Thank you.