Ann Marie Fred and Siddharth Pareek

Ann Marie Fred: Hey everybody. Thank you for coming to our talk today about rebuilding security culture with security champions. I'm Ann Marie Fred, a senior principal software engineer at Red Hat.

Siddharth Pareek: And I'm Siddharth Pareek, Senior Vice President, Consulting for NatWest Group.

Ann Marie Fred: A little bit about ourselves. I have more than 20 years of software development experience, of which more than 10 years were in the DevOps area. I was a DevOpsDays Raleigh conference co-organizer for three years, and I'm also active in the Linux Foundation and CD Foundation open source projects. I was an HR manager for three years, and for the last four years, I've been acting as a security focal. Most of my career was spent at IBM, but the last nine months I've been at Red Hat.

Siddharth Pareek: I am part of the DevOps Center of Excellence practice for NatWest Group. What I'm doing in NatWest Group is driving chaos engineering, security champions, and most importantly, cultural transformation track for the organizations. I'm also part of the governing board for Ortelius, an open source project, co-authored book on site reliability engineering, and published white paper on digital skills. I'm also on board of expert panel for Cloud Credential Council, global ambassador for DevOps Institute, on the influencer panel for DASA, which is DevOps Agile Skills Association, and community manager for Atlassian in NatWest Group and a regional chapter.

Ann Marie Fred: A quick disclaimer: these are our personal experiences and not the official position of IBM, Red Hat, or NatWest Group.

According to former FBI director Robert Mueller, there are only two types of companies: those that have been hacked and those that will be. In fact, in the 2020 Sonatype DevSecOps Community Survey, 24% of the organizations they surveyed either suspected or had verified a security breach within the past 12 months.

Most of those breaches target data that the company owns. The cost of a data breach can be astronomical. GDPR and other privacy regulations hold companies liable for security breaches, and ignorance of the problem is not a defense from the liability. This can cause major damage to a company's reputation, if you think about famous breaches at places like Facebook, Equifax, LinkedIn, Anthem, and so on. In fact, according to IBM's 2021 Cost of a Data Breach report, last year, the average cost of a data breach was $4.24 million.

According to Derek Weeks, the time between a vulnerability being announced and the exploits appearing in the wild used to be 45 days. That has compressed by 93% over the last decade to three days. So how do we keep up with this? Well, it's a race, and the only way we can be fast enough is with continuous security practices.

Unfortunately, according to Toby Irvine's research, for every 100 developers across the industry, on average, there's only one security professional. Because of these scarce resources, security is often neglected in transformations. Security is treated as a last-minute gate. Say, a month or two before shipping a product, the teams will do a threat model and a manual penetration test, fix as many vulnerabilities as they can, but because there's a fixed end date, eventually they just ship it.

Siddharth Pareek: Security experts report about 18,000 vulnerabilities caught so far. Moreover, 2021 was marked with a significant increase in reported zero days. At least 66 zero-day issues have been publicly revealed this year, doubling the total number of those identified in 2020. So the question is, how can we reduce our cycle time right from identifying vulnerabilities, or stop them from being exploited within a duration of three days? Let's hear.

Ann Marie Fred: In order to move faster while being more secure, we have to up our game. And to do this, our technology teams need to become more self-sufficient with respect to cybersecurity. We need to scale up the number of application security subject matter experts from that 1% to more like 10% of our technical population. These would be the people who have the background information that they need to be able to dig into the details of security tool findings, penetration tests, and hacking reports.

Specifically, what motivated us to make a change here? At IBM Digital, we had done some internal DevOps surveys in 2019, and it identified security work as one of the top three pain points of our developers. At Red Hat, a lot of this is coming along because we're restructuring our entire product security programs, and that's largely in response to the May 2021 Executive Order on improving the nation's cybersecurity. Red Hat software is an important part of the supply chain of many other companies, and so we need to make sure that our security practices are state-of-the-art. And I know at NatWest Group, this came about largely because of responding to the Log4j and Log4Shell vulnerability in 2021.

Siddharth Pareek: Yeah. So, interesting story from NatWest Group and a little critical in the context of this theme of security champions. A bit of background: NatWest Group, like any other financial institutions and technology organizations, uses different versions of Log4j and Java libraries. The first challenge was, we were using it very actively, directly, but it was also being consumed indirectly through the tools, through the coding frameworks, which were in turn calling back Log4j. The worst part: it was not even known whether we were using Log4j or not. So that compounded the issue across the industry and financial groups.

The second challenge of a global financial industry like ours is the way we are built. The way we build our toolset and the way we implement our technology into them is sort of in a dependency mode. Our core capability is used by the other middleware, which is then used by the other applications. So it might happen that you're not using Log4j, but some of the services you consume are in turn consuming it. There is a multi-version, multi-layer, and multi-facade issue and problem that exists.

Coupled with this, consider our bank is close to now a 400-year-old banking group. So we have legacy and complexity to it. Complexity is what I've spoken about in the previous few points. And legacy, as we end-of-life toolsets, or there are certain languages which are at the end of their life. As a result, upgrades become quite difficult, and the more you get into the detail, the more you start getting drowned.

The story of Log4j and its relationship with the developer, DevOps, security champion piece of things is very much around the base-level and core understanding about what the job is. The job is to develop applications, and to do so in the safe and secure way. You can't be safe and secure unless you know the toolset, the libraries, the underpinning things you are importing into your application. If we are able to take away all those challenges, then life becomes much easier in addressing not only Log4j, but other vulnerabilities as well.

So what is it all about? It's all about building a culture of DevSecOps. Why? DevOps culture brought a lot of innovation to software development. Security couldn't always keep up with the new speed at which code was developed and reused. DevSecOps is an attempt to address this by fully integrating security testing into your CI/CD pipeline, as well as developing the knowledge and skills required in the development team so that the testing results and fixes may be done intermittently.

It has always been ideal to integrate security as an inherent component of the entire software life cycle, whether you name it DevOps, DevSecOps, or SecDevOps. DevSecOps is about security that is built in rather than security that acts as a perimeter around programs and data. If security is left until the end of the development pipeline, DevOps-adopting organizations may find themselves back in the protracted development cycles where they were hoping to avoid in the very first place. So the question comes up: why do we need security champions?

Ann Marie Fred: Well, as we said before, one cybersecurity professional for every 100 developers leaves them spread too thin. Furthermore, the people who are closest to the code are the ones who are best equipped to keep it secure if they have some basic security training.

In a little more detail, what are security champions? What we're talking about is one management-designated person per small team or squad. That's at least one tenth of the development population, including developers, architects, testers, SREs, and so on. This person will have about 10 to 15 hours of extra application security training. That's enough for them to be a local subject matter expert for their team.

If I'm going to compare security focals and security champions, two similar terms you might hear: a security focal is somebody who has a role more like mine, where it might be a 50% or even 100% time commitment, advising teams, maybe five to 15 teams, on security. Security focals work closely with the other focals across a division or business unit. They're the primary point of contact with the corporate cybersecurity experts, and they're responsible to ensure that security champions are reacting quickly and reporting back status on the urgent and important security work.

On the other hand, security champions is more like a 25% time commitment. So in some weeks, like when you're doing threat modeling or penetration testing, it might take up 100% of their time, but in other weeks it might only be an hour or two of work. A security champion only needs to advise one small team or squad, and they work closely with the other security champions and with their security focal across a unit or a single product team. Security champions monitor communications from their security focal. They make sure the work gets done within their own teams and they report back.

There's a playbook that we found that works for rolling out a program like this across your organization. First, you need a small working group for the program. Just a handful of people is enough. Then you need to create and publish a program description that describes what you're doing and why, and explains all the details that people need to know. Using that documentation, you can share it with your management and executive team to get their buy-in so they understand what you're asking them to sign up for. Your team also needs to choose a good training plan, make sure they've paid for it, and set a due date for the training to be completed. Then all the HR managers will appoint a designated security champion from each team and squad. Finally, our working group will make sure that we've set up the communications channels that we need and will kick off the program.

I have in the backup some more detailed information about what I've put in program descriptions before, but a few highlights I wanted to call out. Of course, you're going to have to have introductory information. You also need to have the expectations of your security champions in terms of time, training, communications, and what their responsibilities are. I like to call out that they're highly valued subject matter experts. This is something that's really good on your resume. Also, security champions don't have to be the technical lead or the most senior person, but they should be motivated and want to learn more about security. And we want to make sure that everybody understands that these are not the only team members who are responsible for security work. Some of the security work can be honestly tedious, especially patching and fixing bugs, and we need to make sure that's the responsibility of the entire development team.

When choosing a training plan, there's a few layers to this, and we're going to put some examples in Slack if you're here listening to our talk. Everyone in the company is hopefully getting some training on cybersecurity basics. That's an hour or two on things like how to have good passwords and recognize phishing attempts. Hopefully, all developers are getting some basic security and privacy for Developers 101 information: things like how to make sure you're encrypting the correct data and handling personal data correctly and not checking in secrets into your source code and things like that.

Specifically for the security champions, we're going to add a little bit more. One is security and privacy by design, and this includes things like all of our corporate cybersecurity standards and how to meet them, how to make sure that your applications are secure by default, how to run good code reviews, checking for possible security problems, and so on. We also give them maybe four or five hours of training in the OWASP Top 10 and SANS Top 25 common vulnerabilities in software so that they know what those vulnerabilities look like, so they can recognize them when they see them, and they also know how to mitigate the problems or fix them in their own code.

Finally, we want them to have an hour or two of training in the basics of threat modeling. This isn't enough for them probably to go out and run threat models all by themselves, but it's enough for them to be very helpful to a qualified security architect in running threat models with the team, and it's probably enough for them to keep threat models up to date over time as well. The good news is, according to Sonatype, developers who receive training on how to code securely are actually five times more likely to enjoy their work. A nice bonus.

Another thing that's really helpful to us is a secure engineering guild. At this meeting, which is usually weekly or bi-weekly, we're expecting security focals and security champions to attend, but I also like to leave it open to anybody else who expresses an interest in application security. We want to make sure that it's either recorded or at least it has a really good agenda and minutes so that people who miss the meeting are still getting the important communications that happen there. A few examples of topics that we've covered in the past include status updates, corporate initiatives, security alerts, security tools, and how our teams are adopting them, like very specific information on how to make it work within our own environment. Pen test and remediation and how that's going. Interesting security news. And if the meeting's long enough, we'll also put in five to ten minutes of timely security education.

Siddharth Pareek: We don't recommend any tools out here. It all depends upon your company's culture and the technology it's currently utilizing or approving or has available for collaboration. So whatever it is, go for it. However, depending upon the technologies you choose to utilize, and it may be even a standard email mailing list, what you need to ensure is that the tools are easy and straightforward.

Next, you'll almost certainly need to capture the interest and participation of the teams. Security champions should have efficient communication channels between themselves and with the rest of the security teams. Lastly, about the guidance, make a time and place for champions to meet on a regular basis. These meetings are useful for discussing any challenges that the security champions are having or adjusting the target as needed. You may start with, let's say, a biweekly session, which could be initially beneficial.

Now what we'll be talking about is the challenges that we have faced or are facing in bringing this security culture in the organizations. Let me start with the first challenge. People are busy. Things are really complex, and prioritization is always hard. There are no special challenges for security. It all depends on our colleagues' or developers' roles in day-to-day life and how it reflects on security. The only special thing is that it's fast-moving and complex subject matter, and that's where and why security champion has important role to play.

What is a recommendation based on our experiences? Four things: education, awareness, involvement, and responsibility. Let's start backwards, which is the responsibility of everyone within the organization for the security of the organization, and that covers all aspects of security, for example, even closing the door behind you to the extent of developing your code safe and secure for your application. It comes down to involvement. Do the people understand it or feel it and know that it's serious? That's where the tone from the top comes in. For example, our CEO, CIO, and CDIOs come and talk about this part of our goals and objectives to be safe and secure, and it aligns with the NatWest Group purpose, which is to serve our communities and customers. Setting the tone from the top is vital. The next is education and awareness. Everybody needs to be aware of undertaking their security responsibility within the group, and certain groups need to be educated further, and that's where developers and security champions come into the picture.

The next challenge is about lack of awareness, and it could be about security principles, or it could be just knowing what is truly susceptible or how exploits occur. Taking an example from NatWest Group, and maybe some of your organizations would be doing something similar: what our security teams do is they send across our colleagues some purposeful phishing emails, let's say. When you click on them, knowingly or unknowingly, you're asked to take up some predefined training programs. The benefit of this is this leads to beneficial adjustment in behavior, which lowers the risk to the business and possibility of re-offending.

The third risk is about the buy-in. Being a bank, it is heavily guided by compliance, risk, governance, and extra layers of regulations, making security of paramount importance. What we are doing is, rather than going to every person and team and asking their views and opinions, we have formed a working group which is trying to build a security champions model. In the words of Steve Jobs, "People don't know what they want until you show it to them." Or in the words of Henry Ford, "If I ask customers what they wanted, they would have told me a faster horse." So the summary is, build a security champion model with the other like-minded, diverse SME groups, and then put it to test with the rest of the audience or colleagues.

There's an interesting paper which was recently published which talks about the human element as an important factor in IT security. As for the findings, the lack of interpersonal interactions and shared language is a key impediment to the functioning of connections between the various groups. In order to develop empathy, trust, and cooperation, it was suggested to appoint security champions to function as a channel, explaining security to your coworkers, assisting them in mastering new behaviors, and reporting security that isn't working.

In all, the challenges for our organizations are: IBM, Red Hat, and NatWest Group all have globally distributed development teams working across time zones. You need either to find a time zone which suits all, or let's say, two different time-zone meetings could be possible. The other recommendation is encourage and support asynchronous communications with instant messaging, shared docs, meeting recordings, email, and so on. More on the challenges, let's hear out from Ann Marie.

Ann Marie Fred: One unexpected challenge I ran into at Red Hat was getting the training program set up. The training program we chose cost about $250 per user, but it took us some time to get the budget approval for that. By the time we got the budget approval, there were only three months left on the contract for the training plan, and they weren't allowing any more new users in. So we had to wait for the security team to negotiate a new contract with the training company. What I learned from this is that you need to choose that training plan and request the funding as early as possible because it could take a while. Also, if you already have a good training plan available to you, start with that. For example, at IBM, I was lucky that we already had several good training modules in our internal learning system, and so our working group just had to evaluate the modules, choose our favorite ones, and chain them together into a training plan without worrying about funding.

The biggest problem I probably ran into at IBM was compliance fatigue. Our developers were already spending up to 25% of their time on things like data subject requests, privacy and security reviews, internal audits, patching software with fixes, and to them, this just felt like piling yet more security work on. So what we did is, when we formed our secure engineering guild, we set one of our primary goals to be automating as much of the manual toil as possible. We shared new automation with each other in the guild meetings, and we pair programmed with each other to implement changes. We also partnered with our CISO organization to pilot new tools that they were considering to make sure that they worked well for us.

What are some wins we got out of this? The oldest program of the three is the one from IBM Digital. When I look back 12 months after starting it, we found that indeed about 10% of our developers had become subject matter experts. They completed their training programs as asked. When handling critical alerts, which we called CISO overrides, those could often be handled within one to three days. That is from the time that the cybersecurity team would notify the security focals of a problem, to when we told the security champions about it, to when they scrubbed all their applications to see if it applied, when they pulled in the cybersecurity team if needed for further investigation or fixed the problem themselves and then got the change pushed out to production. All that could very often be handled within one to three days.

Importantly for me as a focal, because I was responsible for more than 100 applications, I knew that no teams were falling through the cracks. By spreading the communication out to my 10 or 15 security champions and having them report back to me, I knew that we had touched all the applications. We also saw that far fewer security reports from things like pen testing and tools and hackers were coming back with false positive marks or could not reproduce. Usually, when you get a false positive or could not reproduce, that's actually code for the developer saying, "I don't know what this means, and I can't seem to reproduce it myself." When we give them the education they need, then they were able to actually fix the problems instead. People also uncovered new vulnerabilities in their own code reviews. We saw broader adoption of threat modeling, and our teams were thinking about security every week.

Key takeaways: the Security Champions Program is straightforward and repeatable, but you need management support, a few people, a few months to get started. You need funding for online training. One security champion per team. One or two secure engineering guild leads per organization, and that's usually an architect or security focal. In terms of help that we need, we'd love to hear if you've tried something similar yourself and what challenges you ran into and how did you overcome them.

Siddharth Pareek: I have certain help that I would definitely require. I would love to hear out what is the ideal number of security champions per developers that you have? What sort of education and time are you providing the security champions to enable them to perform their role nicely? What sort of contacts or information and security resources do the security champions in your organization need to discharge their accountabilities? Lastly, what tooling, or let's say extra tooling, is required for the security champions in order to build security champions in the life of the developer?

Ann Marie Fred: Thank you for coming to our talk.

Siddharth Pareek: Yep. Thank you. Bye now.