Hart Rossman

My name's Hart. I work with Amazon Web Services, and what I do is I look after our global security, risk, and compliance practice in professional services. My team's focus is helping our largest enterprise customers around the world make good risk decisions as they migrate to the cloud.

And one of the interesting things about so many of our customers is that as they're on their journey to the cloud, they're on a journey of learning and adapting to Agile and DevOps as well.

And so when we're talking about making good risk decisions, it's not just about what are you doing specifically with AWS services, right? It's: what are you doing in the DevOps practice that you're building and in the Agile methodology that you're rolling out across the teams to make sure that whatever you build and deploy performs as expected from a security, risk, and compliance standpoint?

So what I'm really here to talk to you about today is how do you begin that conversation with the security community in your organization? How do you encourage people to start thinking about risk and compliance in constructive ways, instead of just worrying about what you can't do, or what you think the regulator might say, or what you think your boss is going to be concerned about, right? It's really: how do we establish a proactive, positive story that we can then demonstrate through the DevOps and Agile process, right?

So the first thing is you've got to establish that vision, and you've got to be comfortable telling the story. I'm going to share some examples with you today to arm each and every one of you with some of that.

You want to start to build that enterprise-wide Agile security culture. Security needs to be everybody's responsibility when you're doing DevOps. It can't just be the role of a centralized security function.

You want to always start with the customer and work backwards. The customer could have a lot of different personas. Customer could be the regulator in one scenario. Customer could be the security team in another scenario. Customer could be the actual end user of your application in another scenario. What does that security and compliance and risk experience look like to those varying customers, and how can you serve them best through the DevOps process to remove any concerns or objections, right?

You want to be able to provide examples through consultation and code. I think one of the things that we often struggle with in the security community, right, is we want controls and frameworks that we can audit. And audit absolutely has a place and a purpose, but audit isn't often consultative. It doesn't necessarily help people raise the bar. It only checks the work you've already done.

And so if you want to be successful here, you've got to tell that story, you've got to bring people on board, and you've got to help them along, right? One of the ways you know if you're making progress is by having metrics and having everybody be able to review the metrics together. So we're going to talk a little bit about dashboards.

And then finally, as we all like to do, right, we want to celebrate our successes and do the best we can to learn from our mistakes. So that's what I'm going to talk about for the next 28 or so minutes.

So the first thing is, when you're moving into a DevOps type of shop, security really has to be part of everybody's job, right? It has to be baked into the DNA. You've got to promote a culture where everyone is an owner for security. And that's often scary a little bit to the centralized security function, right? They view that as their job and their level of expertise, and they have a trust-but-verify model, right? So they're kind of okay letting you do some things, but not other things.

And what we've got to do is get to the point where the DevOps teams own development, own operations, and as a function of that, own some measure of security, risk, and compliance, right? We've got to make security stakeholders part of business success, and we've got to enable easier and smoother communications, right?

One way to do that is to begin by setting up in your DevOps culture, right, a mechanism by which you can call out to the centralized security function when you need help, right? So instead of them always sort of feeling like they have to bust on in like the Kool-Aid Man and see what you're doing and poke their nose under the tent, right, set up a mechanism.

One mechanism we've seen work very successfully is part of a ticketing and workflow system, right? So instead of having security in every single standup, right, every single meeting, every single sprint, say, "You know what? We know what our role is as the DevOps team, and we have a mechanism by which to call you in when we need support."

Email's not a great mechanism. Scheduling meetings isn't a very great mechanism because it doesn't tie the record to the accountability. Ticketing systems or workflow are fantastic. I need help with a threat model, I ticket the AppSec team, right? I have questions about my security group or firewall rules, I ticket the perimeter protection team, right?

So you have a way to build a consultative relationship where security is everybody's responsibility, and you can be held accountable, you can manage the communication, right, and you can bring in support, and then you've got metrics to show you're doing it.

So that gives you an interesting thing to look at right out of the gate, right? Which DevOps teams are proactively reaching out for support through tickets? And then where are the cold spots, right? The cold spots are where the security apparatus might want to go poke their nose under the tent and say, "Hey, lots of DevOps teams are asking for help with their threat models. You guys have never once filed a security ticket. What's going on over here?" Right?

So it's kind of a great way to build that culture, to manage that culture, to empower teams without getting stuck.

If you're going to do that, though, you do have to now operate in a shared responsibility model. You have to be really clear about who owns what and who's responsible, right?

So I like to break it down really into three simple areas, right? Do I really own it? A lot of people talk about ownership in DevOps and ownership in software development. At the end of the day, for me, you own it if you carry the pager for it, right? It's very simple, right? If you're going to get woken up at 3:00 a.m. because something went wrong, you own it.

On the other hand, you might not own it, but you might govern it, right? That's a very common place where people get confused. They think they own it, but they really govern it. They set the policy for it. They make the rules on how it should operate. They set the expectations for the way something should occur. And that's great. You've got a role to play too, right? Own the fact that you govern it, as it were.

And then the last place is everybody else who wants to help, right? It's really not their project. They're really not responsible for the outcome, but they've got an opportunity to add some value in some way. And that's great, too. Those are the people who should be consulted or kept informed in the process, right? And if you can clearly delineate those three things, right, it's easy to understand how you play your role in building that security culture and delivering success, right?

So we'll sometimes talk about DevSecOps, or DevOps, or DevOpSec, or SecDevOps, however you want to call it. I happen to like DevSecOps because it puts security right in the middle of the conversation, right?

And so when you take it from that perspective, you've got development, who's responsible for building things faster. You've got operations for keeping things up and running and stable. And then you've got security, whose job is to help you protect it and demonstrate that you're protecting it, right, from an audit and compliance point of view.

We talked earlier about telling the story. One of the things you have to do in telling the story is take the stuff that you all know in the DevOps community and start to put it in language that the security community can engage with, right?

So I'm going to spend the next couple of minutes telling you a bunch of things you probably already know, like why it's good to be agile, and what CI is, and what CD is. But what I'm going to do is layer on just a little bit of language that'll allow you to engage with the security community, right, as you're trying to bring them into the fold.

Before I do that, though, I do want to get a little bit of a sense for something. Who in the room has the job security, risk, or compliance in their title?

Four people. Okay, it's a good start.

Of the four of you, and the camera's not on you, so you can answer honestly, how many of you write code weekly?

No. No. All right, so this might be a little new to you guys. Fantastic. But that's part of the journey, right, is getting people involved and helping them understand how to be part of the process.

So the first thing that we want to have everybody understand is that moving fast doesn't equate to more risk, right? Often in the security community, you're going to find people who think that older things are better because we've used it for a long time, we've never changed it, and we have an expectation of what happens every time we use it. And so when it looks like something's going to go a little faster, change more frequently, that makes security people nervous, right?

So one of the things you want to be able to do is talk about, right, larger efforts where it takes longer to see the result means there's a much riskier outcome. And it also means if you have to make a change to correct that, like a patch, right, or deal with the surface area or the attack surface of an application, you've got a lot more work after you find the problem to resolve it, right?

Whereas in an Agile methodology, right, using DevOps, smaller, more frequent releases means you're minimizing the risk of exposure during the promote-to-production process, and it minimizes the time to remediate if you detect something that's not right, right? So ultimately, it lowers your risk, right? Security people love to lower risk, right? They love, love, love that, right?

So you want to change it from, "Look how fast we can move and break things," to, "Look how we're going to help the organization lower risk." Right?

You also want to get people in the habit of understanding that you actually learn the most in production and operations, right? In the security community, often we want to try to learn as much as possible before we go into production. That way, we prevent anything bad from happening in production. But something we know in the DevOps community is that you are going to learn the most in production.

And if you've got frequent releases and the ability to remediate quickly, and immutable infrastructure, and blue-green deployments, and all the other goodness that we've got here, right, what it really means is that you can learn the most about the security, right, of your application during the release cycle and from how your users and also how some of your ne'er-do-wells are interacting with your application, right?

So you want to get into that production mode as quickly as possible. You want to get interaction with your users as rapidly as possible.

Right? So now that we've kind of set the stage a little bit. We've said we can lower risk. We can learn a lot from production. Don't be afraid to launch, right? Now let's talk about some of the things we should be, right, adopting as tenets, right?

So the first thing, and this is really coming from my point of view as being part of AWS, is we think of not just DevOps, but also obviously security in the cloud. And so the first thing is that you want to use the cloud to protect the cloud, right? In other words, don't deploy all of your applications in AWS and try to protect them all from on-premise, right?

You want security to be as close to the data as possible. It's a pretty well-understood tenet. That means you've got to forward deploy your security infrastructure and automate your security operations in the cloud using the cloud.

The same principle holds true, right, in any DevOps environment you're deploying into. You also want to make sure that your security infrastructure is cloud-aware, or I should say more generally, environmentally aware, right? Too often, we deploy security solutions that are essentially a closed box and don't interoperate or take into account the context in which they're deployed, and so they only partially protect the deployment, right? We now have an opportunity to correct that.

You want to expose all of your security features via an API, right? Security's contract with everybody else is the way by which we interact at the API level. So you've got to have the capacity to do that.

And finally, automate everything so everything scales, right? You want to make sure that all of your security processes that you can automate are automated, and all of your security infrastructure that you can deploy as code, you deploy as code.

One of the things that does is it brings the security community into the DevOps process themselves. Now, all those guys who raised their hand up earlier and said they don't write code on a weekly basis are going to be writing code on a weekly basis. So they're going to be building and contributing to everybody's success.

So I think many of us in the room know what continuous integration is, and we also know what continuous deployment is. But why do we want to have that conversation with the security community? We want to be able to talk about the value continuous integration and continuous deployment brings to lowering risk in the process.

So we want to talk about the confidence it gives us that our code changes will build successfully, a high degree of integrity. We want to talk about the velocity of the feedback cycle through iterative change. Incident response, time to remediate, reducing dwell time of attackers. Those kind of speed and Agile things in the security environment.

Bugs are detected quickly. So are vulnerabilities. Automated testing reduces the size of the testing effort. You don't have to wait to do your static code analysis or your vulnerability or pen testing at the end. You can build security unit tests that tie directly to security stories or use cases and abuse cases in every single sprint, which is kind of awesome.

And you get very fast feedback, and you can test things immediately. If something fails a build, you can rewrite the code, improve the security unit test, run it again.

So CD now gives you the ability to have a repeatable process to push changes into production, whether they're security-relevant or otherwise. You can harden and de-risk the deployment process. It allows detection of failure really quickly, supports things like A/B testing, and then gives us a breadth of data to manage our applications.

You're not just relying on a vulnerability assessment or a pen test in production. Now you've got a wealth of information about the deployment process and the operations of that application to feed into improving your threat models, changing the way you operate, reducing the risk for the overall deployment.

Now that you kind of got people on the hook, the security community is going to ask you, "Well, what do I do? I'm sold. Pretty compelling case. I can be a part of the process. We can build things, we can break things, we can fix things, and we can build them again. What do I do, though? How should I be thinking about this?"

And so, I like to think about these three core principles when it comes to DevSecOps.

The first thing you need to do is secure the tool chain. That's your supply chain for your software. So you want to make your source repo, your CI/CD tool environment, your testing tools. You want to be developing secure baselines for the building and deployment and operations of those, just like any other piece of software.

So identity and access management. If you need any integrity checking or digital signing of code, if you need to do any testing or evaluation of your pipeline itself, pen tests or vulnerability assessments. Each one of the tools in your tool chain emits a ton of telemetry. Is that being backhauled into your security operations center so they can monitor the strength and integrity of the pipeline? So first of all, secure that tool chain.

The second thing you want to do is armor up your workloads. Now you have an opportunity to take some of the things that I talked about earlier and begin to empower the DevOps team to make sure every line of code that flows through the tool chain is better, smarter, faster, and more secure than the last line of code.

You can empower the team to build a threat model. Every threat model gets checked into the source repo. Then what you can do is from the threat model, you can develop use cases and abuse cases. Use cases are things you want the software to do to add functionality based on your threat model, and abuse cases are things you want to make sure can't occur as the result of a malicious user trying to subvert or manipulate your code.

You can then take those use cases and abuse cases, write stories, sprint against them, and develop security unit tests that map directly to the use cases and abuse cases.

So now before you've even done something generic like static code analysis, before you've done anything generic like a vulnerability scan or an app scan or something, you have an application-specific, controlled by the DevOps team so you know it can be implemented quickly and effectively, straight line in your story from my threat model to the code that gets written, to the specific test to validate. And then, oh, by the way, we have the institutional checks, the static code analysis, the vulnerability scans, the requests for a pen test.

That's a story the security community can get behind, and they can see it every single time you do it.

One of the things that means is that security doesn't have to physically be there every time, but what it does give them is an opportunity to check and collaborate. That first threat model you write is probably going to be abysmal. Why is it going to be abysmal? Well, one, it's probably your first time, but more importantly, you're probably a really nice person. You don't think how to compromise your own stuff. You don't think about all the bad ways to do things.

So you probably have a kind of weak threat model. But that's okay. You're going to get smarter every time. You're going to get user behavior data on how they're using and abusing your app. You're going to ask for a consultation from your security team to come in and tell you all the bad things they'd do if they were working with your application. And over time, you're going to build that awareness, you're going to build that capability, just like with each sprint, your application gets better.

So it's okay to start with a weak threat model. You know by the time you're done, it's going to be rock solid.

And then the last thing you want to do is really encourage your security organization, risk, compliance, engineering, audit, whomever they are, to build and deploy every single one of their applications through the same tool chain you're using.

That does two things. The first thing is it puts them in a position of being builders themselves, not just sort of critics of other people's work. Right? The second thing is it forces them to really scrutinize the tool chain itself and make sure that it's at the right bar for building and deploying software, right?

If a security team says, "Well, I can't deploy my software through this tool chain because it's missing separation of duties," the answer is not for the security team to go build a separate way of building and deploying software they're more comfortable with. The answer is armor up your tool chain to enforce the separation of duties they're expecting, so everybody benefits from that. Right? And then the security team can comfortably use it just like everybody else.

Right. So how this might look, I told you the story, is you've got that security repo. You're doing your security infrastructure and application tests. You're doing those vulnerability scans. You're able to take that threat model and kind of walk it through the whole process, right, and then align it with the things we talked about in kind of the CI/CD approach of doing things. Right?

So at the end of the day, you're left in a kind of cool situation where you're not just building DevOps teams, but you're building DevSecOps teams. Right? You're making DevOps the security team's job. You're encouraging developers to participate openly in the automation of code, whether it's security or otherwise. You're emboldening the ops guys to do the same from a security standpoint, and then everybody's able to take pride in how fast and frequently you deploy those security features, right, and how quickly you're able to shut down adversaries.

Along the way, you want to keep track of your progress, and there's a couple of ways we can think about doing that. It's helpful to have a cloud security strategy or a DevOps security strategy. It doesn't have to be a 50-page document that took you six months with a management consultant to write. It can be one page with a set of principles and tenets, or what you might call security expectations for the DevOps community. Right?

You want to have a stakeholder communication plan. Engage in telling that story. Get better at telling the story every time you tell it.

Do your security cartography. Do the mapping between what you're doing and what the rules and regulations and compliance requirements are, right? Visualize it, bake it into your code, make it easy for people to consume. Don't just show up with a checklist. Checklists aren't helpful.

Document that shared responsibility model. We talked about at the beginning. You got those three layers. You might have more. Make sure you really have names and org charts and responsibilities on there so people know what role to play.

Consider building security operations playbooks, and then distilling them down into code as runbooks. Right? You have general themes around the way your organization ought to be building and deploying security infrastructure, and make that available in the repo for everybody to use, not just the security team.

Next thing is you're going to ask, well, what are some of those common epics or stories that I ought to be thinking about? What are those? How do I get started? What's my roadmap?

And then make sure you're testing it. Do incident response simulations. Build and deploy, automate, and then test to see if it's working well. Cause something to fail. Introduce an adversary to the environment in a controlled manner. Make sure that software you're writing and the incident response processes you have in place are performing as you expect they are.

Right. So at AWS recently, we've started to talk about these 10 security epics, and they're themes you can apply as you're thinking about your DevOps build cycle over the next several months to focus on security, risk, and compliance. You can treat them as individual stories within any other epic, or they can be epics in their own right, depending on the level of maturity and build you have to do or who's doing the construction.

So I'll just try to call them out because this slide may be a little cluttered. You want to focus on identity and access management, logging and monitoring, infrastructure security, data protection, and incident response first.

And there's a reason for that, particularly in the cloud. Right? Unlike on-premise, where you rack and stack gear and then you provision access to it, in the cloud, you start by provisioning access, and then you provision infrastructure.

So the first thing you want to do is do a couple of sprints getting your identity and access management the way you want it to be. Then you want to enable logging and monitoring and get that infrastructure set up and backhauling it to your SOC or wherever you're managing it. So that way, when you get ready to deploy your first asset, it's access controlled and it's under observation.

Now you can start provisioning infrastructure resources, focusing on configuring them to meet your needs. Now you're ready to put data in the environments. Now you can think about encryption and tokenization and access control at that level. Think about your data protection requirements. Layer that into the build. Right?

And then incident response. Now that I have something up and running in the environment, am I confident and comfortable that we can work together if something goes unexpected? If there's anomalous or malicious behavior. Right?

And then there's a number of other epics. Resilience. Compliance validation. Here's the idea where you want to be generating evidence as part of each and every sprint to satisfy the needs of your audit compliance community. No checklist required. Right? They ought to be able to look at the dashboard. They ought to be able to look at the artifacts. They ought to be able to see some of the data, and that ought to give them enough evidence to confirm your compliance with the rules. Right? Shouldn't have to be a separate exercise.

Also reduces the load on the team that's doing the builds and reduces the interference by people constantly coming in and asking you a whole bunch of questions, right?

You want to make sure that, again, you're focusing on continually armoring up the tool chain, thinking about configuration and vulnerability analysis, your patch management strategy, reducing over time application surface area or attack surface, and then thinking about all the telemetry you have available to you, from the DevOps pipeline, to the infrastructure deploying, to the monitoring systems that you have available to you.

And if you've got a big data project anywhere in your org. Who here is doing a big data project in your org?

Okay, I've got quite a few people. How many of you are doing a security big data project in your org?

Right. That's often an easy miss. If you think big data is important, if you're building and deploying the infrastructure to do that for genomics, or oil exploration, or fish and wildlife management, or whatever it is you do all day, click streams on media and advertising, the security team can benefit from that to lower the risk and enhance the security posture of the organization using those same tools.

So think about doing a security big data analytics project, especially as you get a couple of projects in this kind of DevOps frequent release cycle.

You can also provide them sample user stories. So you might say in an IAM epic, what do those first couple of sprints look like? And what specifically are the user stories that might go into the backlog? And who's the persona? Who's the customer? How does all that work?

The danger here, I'll tell you, though, because I do think this is a phenomenal approach, is that if you script all this out for everybody, it stops being DevOps and Agile, and it starts being Waterfall. You hand somebody a six-month plan: go build this.

So that's not the intent here. The intent, though, is, again, you're engaging with a community that may not be hip to DevOps and Agile and may not know how to get started. So you're not trying to give them a six-month Waterfall plan. You're trying to give them an idea of what do those initial sprints look like, what are our initial epics and priorities, and get them started. You're not trying to give them an inflexible roadmap that says you must build these and only these things. Okay? You're just trying to give them a primer.

When it comes to security cartography, you want to be able to take what you're doing, as I mentioned earlier, emit the evidence as part of the process, and map it to the code.

So in our particular case, we've actually launched enterprise compliance accelerators for things like NIST Special Publication 800-53 in the U.S. and PCI compliance, where you can go to our website and you download three things. They're all in one place.

First thing you download is instructions on how to use these CloudFormation templates. The second thing is the CloudFormation templates themselves. And the third is a map. And what it does is it literally tells you this line of code in CloudFormation, or this line of code in the IAM policy, meets this specific PCI requirement.

So now you know, one, where you're compliant, and two, if you need to make a change in the future, which specific line of code impacts what. And then you can make a determination whether or not the change you're going to make allows you to continue to be compliant or not.

So you want to drive that security cartography all the way down into the code, not just the effect the user sees. And that also gives you a way, again, to generate the evidence as you go.

Another thing you want to do is demonstrate to the security community that doing DevOps allows them to react faster. So here's a simple example where we did a live demo a little while ago where we showed that you can deploy a memory imaging tool really quickly using the same deployment tool you would push a web server out.

So you identify an anomaly, you're really concerned, you want to snapshot memory. So you might try to use a tool like LiME, and you use the exact same deployment tool to push LiME onto the server and do the memory snapshot as you would as if you were pushing out Apache. Same thing.

And so you want to be able to demonstrate to them that they can use those tools to be fast, efficient, and you can show them the dashboards, and you can show them that there was integrity in the launch process, and you can show them that the binary for LiME hadn't been manipulated in the source code repo. Show them all the things they need to see and make it easy for them to want to be part of that DevOps process.

So I'll end with two little lists.

One is, think about being planetary scale from day one. It's often difficult in security to go from the first thing to the second thing, but if you're doing DevOps, Agile, and automation, going from the second thing to the thousandth thing is easy. So think planetary scale.

Expose your features as API, particularly around security. Know who your customers are and make sure you're doing things in the process to service each of those customers in the security community, including the end customer who cares about security and the privacy of their data.

Utilize that feedback during your iterations and internalize those metrics and make them available for everybody to view.

So make DevOps the security team's job, harden your tool chain, plan your security epics, write your first security user story together, and then go ahead and sprint.

We're just about out of time. Thank you very much today. I can be reached at hart@amazon, or you can hit me up on Twitter @hartdanger. Thanks for your time.