Tomer Gershoni

Hello, everyone.

And Gene, thank you for inviting me and putting a security person inside a DevOps conference. I promise not to bite if you won't.

Ruly was talking with you about the Agile DevOps transformation he was going through with his team. Now I would like to share with you the cultural journey that we've been going through inside HP Software, throughout embedding security into Agile and DevOps practices, and the lessons learned that we've made through that.

My name is Tomer Gershoni. I've been in information security for about 15 years now. Now it's also called cyber. And in the last four years, I've been leading the engineering security program for HP Software, supporting about 60 to 80 product teams, more or less, throughout embedding security into their daily activities.

In my 15 years in information security, I've seen a lot of organizations, IT environments, technologies, and security solutions to, theoretically, every security threat that exists out there. But there is one security problem that we still see, that we still recognize, which is the disconnect.

This disconnect between the builders, which we consider as the dev, the QA, the IT operations, or the DevOps teams, to the defenders, which are the CSOs and their delegates. And when we inside HP Software did a root cause analysis of the reason that we see this disconnect, the reason for that seemed pretty obvious.

We have two different management systems. We have two different methodologies. We have two different tools to manage. And obviously, there will be a disconnect. Those are not connecting together. They are not speaking together.

And the outcome of this disconnect was that for every release, at the end of that release, when we are just about to push the code to production, we have one big security problem. That security problem was not only costly, hard to manage, but it even caused a fire drill for every release. And managing that fire drill was very hard.

So we said, "Okay, so how do we resolve that?" In theory, the solution was very straightforward: let's take this one big security problem and break it into chunks. But we looked into the industry and looked for methodologies, solutions, technologies that will support that. None. Almost none.

And so we said, "Okay, so if the problem is that straightforward, how come there are no solutions for that problem?" So we started thinking about solutions, internal solutions from a process perspective, policies perspective, tools perspective.

But even the bigger challenge coming through that is the fact that you need to build that solution in scale. HP Software is a $3 billion, $3.5 billion business. We have 5,000 engineers, 80 enterprise software products. Some of them are 20, 30 years old in market. We have product release cadence from weeks to months to years, even.

And just to give you a sense of that size, after four years of operations today, we have scanners that scan on a daily basis an average of 12 million lines of code for security vulnerabilities. And that's what we have in production today.

So that's a really large scale, and every process tool that you need to build is something that can scale to the size of that organization.

So we set up the team that I'm leading. I'm leading a team that is named Security and Trust Office. So it's not the regular security branding. Its objective is to build trust with our internal and external customers.

And we started that journey by trying to break out the silos and embed security into the organization from a cultural perspective. But we wanted not only to be the central police. We didn't want to be the guy that you're afraid of. We wanted to be part of the engineering teams.

And we tried by branding ourselves as, instead of the central police, the community police. And we branded ourselves as the geek with the shield, more or less.

But we continued with training 2,500 developers globally. Not an online training that you need to go through, and you go through the slides, you know how it works, and go through an online training. But put the kids in class, train them on how to protect the code that they are writing, put an entry exam and a final exam, and track that progress with their management.

In addition to that, we've built a security champions community. The champions were selected carefully inside the R&D teams to support this transformation. Those are senior developers or senior architects that have the influence to the R&D teams. And in addition to that, we developed that community, we improved that through time, so that allows us to have the right footprints inside those teams.

But then we thought about, okay, so how do we embed security into the Agile processes? How do we do that shift left of security so security will not be a problem at the end of the release?

What you see here is our security lifecycle framework. Pretty straightforward framework as an industry standard, but that's regular security lifecycle regardless, whether we're talking Agile, waterfall, DevOps, et cetera.

We started to think of how do we take that and adjust that to the Agile and DevOps speed? So we started with planning and worked with the different PMOs to align the security activities' timeframes to the Agile sprints and release timeframes.

We continued with threat modeling. So threat modeling, for those of you who are familiar with it, is a very heavyweight, slow process which takes months in certain cases. We broke it down to be very lightweight, feature-based threat modeling, so we'll be able to identify design risk early in the design stage.

We continued with building a security test automation service that the developers will be able to seamlessly scan their code, scan their apps, scan their infrastructure for security vulnerabilities without any impact on their daily operations.

And we adjusted the hands-on testing to the release timeframe and content readiness. So once the content is ready, it's being pen tested by professional hackers.

And that was the final outcome of that. You have the software product lifecycle, and you have security embedded through that, and that really works. It's being governed by the product lifecycle governance framework, and it really works continuously and with less fire drills than what we had.

And the last example that I want to use was, how do we build transparency and control over risk?

And the first thing that we wanted to avoid is the continuous arguments that you have with the pen testers or the security assessors about the severity of the risk.

How many of you have came up to the office, saw a pen testing report or a security report, and didn't understand how the risk severity was calculated and how it was actually measured?

So in order to avoid that gut-feeling risk determination, that when a hacker comes up, or a security assessor comes up in the morning, and if he's in a bad mood, the risk severity will be high. If he's in a good mood, it will be low.

We built a risk calculator to ensure transparency in regards to the risk severity determination. In addition to that, in order to give the risk and the overall product some business context, we built a term called product criticality, which is based on the revenue, the data type, the deployment model, and the breach history.

So whether the product is already in the focus of the future hackers, let's say. So it gives business context to the actual security vulnerabilities that we found. And those two together, for our business leaders, provide them with a very clear distinction of the risk profile of the product.

And that's a very dynamic value that could change on an hourly basis or daily basis according to the fix that the R&D are implementing throughout the product.

To summarize, after two and a half years of operations, we started to see the change happening. 84% of our engineers said that the training and the materials that they have learned can be implemented in their daily operations.

The R&D effort in auditing the results, reviewing the results that are coming from the security scanners, dropped from months to hours.

In addition to that, throughout time, we've seen a dramatic decrease in the amount of vulnerabilities that are found throughout the hands-on penetration testing. This is a real product graph that was released just recently, with zero security vulnerabilities that are found in the hands-on penetration testing.

And we are seeing 30% year-over-year reduction in what we call vulnerability score, which is our metric to measure success.

But bottom line, what cares is our customers. And as part of what I said, building trust with our customers is a major activity and focus of our business.

And to finalize my session, I would like to give an example of the Heartbleed incident. And in Heartbleed, we were able to fix it, fix and provide a public response within 48 hours to all of our customers due to this very tight and intimate relationship that we built with the product teams.

Thank you very much. Thank you.