Ed Bellis

I am the co-founder and CTO now over at Kenna, a software-as-a-service in the security space. Not as important.

However, what I'm going to talk to you today about... There we go.

Normally this is the slide that I almost always gloss over. But in this case, it's actually a little bit important. And it's not so important that I am the co-founder and CTO of Kenna, but rather why the confusion occurred in the first place, which is, prior to that, I was the chief security officer for Orbitz for about six years.

And like most sane, rational, thinking people, you can only be a CISO for so long before you have to decide on another career option. So I went out to co-found Kenna.

But what I'm going to talk to you today about is a little bit of both. One, specifically the internal DevOps practices that we're doing at Kenna and how we are doing these things internally, but how we're doing them securely, and where you're going to deal with typical objections from a security organization, or a former CISO such as myself, and why those objections are, for the most part, bunk.

So, as we talked about, there's going to be a few things that I'm going to go through here in my 25 minutes or so.

One is the security value of the automation that is typically plugged into in a DevOps process. Another is: what are the top concerns from a security perspective when implementing or automating a lot of these controls? And then applying these SecDevOps principles internally and dogfooding it ourselves at Kenna.

And I'm actually going to walk through a bit of a use case.

I was talking to Josh Corman here earlier, and he told me very kindly that I should stop saying SecDevOps and start saying Rugged DevOps. I'm down with that. I can do that. In fact, check it out if you haven't: the Rugged Manifesto.

The important part here is I want to show how security gets injected into this process, and how it works and actually makes security work much better, much more efficiently, just like everything else that we're starting to see in the org on the DevOps side.

And dealing with a lot of the objections will end up also showing you ways to actually make your audit process go a little bit more smoothly as well. I know that there was a talk in here, I think maybe two sessions ago, with a guy from Amazon Web Services who had a lot of... I did get a preview of some of the stuff that he was talking about, and dealing with some of the evidence that you go through when you're going through a particular security audit, and using some of the things that you end up having as artifacts within a DevOps process as evidence within that audit.

So why are we doing this? Or what sort of security value are we getting out of it in the process?

A really quick, short shout-out. I know some of you have probably seen this before. I actually got to attend this and talked to Neal, who at the time was over at Twitter. He's no longer there. But they had an amazing talk, and it is available online if you haven't seen this, on putting your robots to work and doing a lot of security automation, specifically in that case, mostly around application security within a DevOps.

This has probably been repeated at every DevOps talk, and certainly probably a lot of the talks here as well, but I really want to talk about the security benefits of all of this.

I've come from an organization... In fact, at Orbitz, when I started at Orbitz, we were pretty lean. We weren't calling anything DevOps way back then, but we had something called application support, which largely actually resembled a lot of what DevOps is today.

And we did a lot of frequent deploys. Not nearly as many as many of the organizations are doing today. Not nearly as many as we're doing internally at Kenna today. But we were doing quite a bit.

And what I noticed the difference is, when working with some of our partners and things like that, these folks that had long, drawn-out waterfall processes, and it's not even specific to waterfall per se, but the ability to create small, frequent changes where you can pinpoint exactly what happened and what broke this thing so that you can actually go out and fix it and fix it much faster than having to... what we ended up having to do was a whole lot of rolling forward and sometimes falling back as well.

A lot of people know about, I'm sure, again, this is nothing new in terms of a DevOps conference talking about a lot of the Netflix things, but I wanted to make sure that, for those of you who haven't seen it, there's a lot of security tools that they've actually open sourced that are extremely useful.

Outside of Netflix, there's something, if you haven't seen it, from James Wickett called Gauntlt on this application security side. And being able to plug in all of your different security toolsets into something like a DevOps process is fantastic as well.

And of course, just like this benefits ops, this also benefits security, and being able to integrate a lot of your security tests, your security tools, into your continuous integration environment, your continuous deployment environment, and being able to quickly and easily understand where things are breaking from a security perspective.

So, I promise I'm not going to call it SuckDevOps. We'll call it Rugged DevOps at Kenna, or what every CISO should know when going into a DevOps process.

We touched on this already. Small, frequent changes are huge. Tons of wins here. I put up a tweet from I Am Devloper, which is absolutely true. Ten lines of code, this is regarding peer code reviews.

So part of the process as we're going through, we're doing pull requests. We'll have somebody, multiple peers, come in and take a look at that code. One of the things that they're looking for are security issues.

But if I've got 10 lines of code, they're going to find 10 issues. Five hundred lines of code? Eh, looks fine.

The bigger the change, the more monumental the task, they're not going to do it. It's like, taking my CISO hat on, the days of generating these massive security reports and dumping them on somebody's desk and expecting anything to happen other than that getting filed in the trash.

So small, frequent changes help security as well.

Always be deploying. This is a graph of just our internal deployments at Kenna, how many we do on a daily basis. The red are where we had particular failures in the build. The greens, there were no failures. The more we deployed, the smaller, of course, the changes were, and the more successful we were as well.

So, static security requirements. One of the biggest pains that I still deal with even when I'm at Kenna, and it's more on the audit side or the due diligence side or things like that, is you get these long security questionnaires. Everybody's gone through them. You're looking for various artifacts to prove that you do what you say you do.

You're looking for policies that are often static, out of date. But more importantly, the evidence and the artifacts themselves become out of date because they are a Word doc or an Excel spreadsheet or some PDF that's sitting on a shared drive somewhere, that the last time anybody looked at that was the last time the auditor was on site.

So much like the talk from Amazon Web Services earlier, there's a lot you can do with various artifacts and the toolsets that you're already using to satisfy the auditor that's actually going to be, one, certainly a lot less overhead for you. But it's also going to make verification for the auditor much easier.

Because I know if this is in your current build process, if I know this is already going on in your continuous integration environment, I can see that process going through, then I know it's happened because it's happened from the robots. It's not a person who said, "Yes, we do this, and here's a Word doc that says I do that."

We've open sourced a number of different cookbooks ourselves from Chef, but pick your toolset: Puppet, Ansible, whatever it be, for managing configs on the security side.

We've got an open source ModSecurity cookbook. We've got one for iptables. We've got one for Nessus, which is a vulnerability scanner, one for encrypted volumes, Nmap, Duo two-factor authentication. We've actually got a couple of new ones as well that we're working on that I'll add to the deck, and we'll certainly get that out in the resources section of the presentation as well.

And then testing all of this and pulling it into our own continuous integration environment. So using things like, in our case, Brakeman to do the static analysis and breaking it up into certain pieces of code, because you're not going to do a complete static analysis check of your entire base as you're checking into master.

But if you can focus in on the key security components of your application, and you've got something like a static analysis tool that's hooked right into your continuous integration and continuous deployment, you can automatically test all of these things.

We happen to use CircleCI, but pick your CI of choice.

And then using ChatOps to actually not only understand what's going on from an ops perspective, but what's going on from a security perspective as well. We use Slack internally, and we've hooked in a number of different security tools directly into that, both for the build process.

So if I'm starting to see vulnerabilities that are coming up, as an example, either through my static analysis tools or even through some of the dynamic application scanning that we do, or we are seeing anomalies in things like our file integrity monitoring, all of this stuff gets alerted back up into a tool that everybody in the company is already using, in this case Slack.

But again, pick your chat of choice. There's a number of integrations out there for these things. We've hooked in the ELK Stack into there, PagerDuty, and Sensu as well.

So this is kind of what our testing process looks like. I mentioned we're using Brakeman for static analysis. We use dynamic application scanning in the development environment as things get pushed out.

But more importantly, we're focusing in on the new pieces of code that have changed and then hooking that in because, again, the longer your tests take, the less likely that they're going to actually occur, or somebody's going to complain about it and we're going to yank that out.

We do our own internal assessments. We obviously have our own penetration testing services and things like that that go on and hook in bug bounty programs.

And we'll actually tie in the results of our bug bounty program. So when findings get found through our various bug bounties, that data will come in. We'll actually pull that in via our API into our own instance that we're dogfooding, of Kenna in this case.

And then we can hook that into whatever we're using to track this, in this case Pivotal Tracker. But imagine you're using your bug tracking system or something like that. We have a Kanban approach to things like this, and we can actually start to prioritize the security bugs that are found, both not only by our toolsets, but as well as by the humans as well.

This is one thing that I wanted to touch on because I think it's important. There's a lot of objections that come through, and I've seen it, I've lived it, I've been on the other side of it.

How can you possibly move this fast? How can you possibly deploy this much stuff and still be secure? There's a lot, and especially: how can you comply with all of these various regulatory requirements?

One of the things that I proposed and has worked out well for us, we're using Chef and using cookbooks not only as evidence, but also as documentation of policy. So rather than, again, having this static document that's sitting somewhere on a shared drive that nobody's touching, I can be assured that if this is a cookbook that gets checked into my code base and is being used to deploy all of my various production instances of my servers and applications, then I know that I can track exactly where that's gone, and I know that is actually being lived up to.

Again, the ELK Stack. We talked a bit about dogfooding some of this. We actually use GitHub and Code Climate to actually check all of our source code, not only for security issues, but also for various performance issues and different things that are there.

I put a link in here as a little bit of an extra credit. There is an open source project that does a lot, goes way beyond some of the stuff that we do, that will really enable some of the compliance activities within your organization, all through continuous integration as well. So definitely check that one out.

I'm going to blaze through this in order to meet the time here, but APIs are definitely your friend. Describing APIs turn out to be great artifacts as well, and then using that to actually integrate all of your security toolsets in order to prioritize these issues as they're popping up, and knowing when things are going on, whether it be in the build, whether it be in the operations pieces, is fantastic.

And the number one thing that I seem to get over and over again when I'm getting hit on the head by an auditor is, "But what about segregation of duties?"

I'm here to tell you that this actually works much better than the old process or the old way of doing segregation of duties.

So in a typical, more of a waterfall process, I've got the devs checking in the code. The ops team is going to do the deploys to the various environments. It's going to go to QA. I'm going to do all of my QA testing in there.

If everything is passing or getting to a point where it's ready to go to the next gate, it's going to go to staging. I might do some performance testing and other things in there.

But the important part is that the ops team is separate from the development team, and there's no possible way that a developer could ever deploy evil code to production, because I got my ops guy there who's actually going to be the one deploying and pulling the trigger.

But how many times have I been in a large organization or a small organization where the ops guy is looking through the source code and going, "Yep, looks like we got a backdoor here"? I've never seen it. Maybe it's happened. But that's not an effective control for segregation of duties.

And I'm here to tell you that pull requests are a great way to do segregation of duties. It's not that you want to say, "Well, this role and this role have to be separate, and I need to have a developer develop, and I need to have an ops person make sure that they're the cop in the organization and make sure that nothing bad goes in."

It's that I need multiple eyes on it. And I need to make sure that there isn't one person who could be a bad seed and do something like a backdoor within my environment, and that's where pull requests come in.

So being able to check in the code, pull in the pull request, have multiple people giving a thumbs up on that code, both from a development perspective but from a security perspective as well, that is a great way to get segregation of duties. And so far, it's worked out well in our audits as well.

So we've got a little experiment that we've built out. And it's one of the things that we had constantly come in contact with, and obviously in our business, this is a pretty important one, but we're always being told by the auditor that you must scan. You must do vulnerability scanning, which is great, and you must scan everything before it goes into production.

I mentioned we do dynamic application scanning. We do network and host scanning, of course. We do static analysis on all of this.

But hooking all of that into your continuous integration and deployments means that's going to be really slow. And forget about deploying often and creating small changes that are constantly going out the door if you are scanning, especially on the dynamic side or even, in a large case, the static side. You're not going to be able to get all of that out into production.

So do I really need a scanner, or do I need to understand what are the vulnerabilities ultimately that are going to be in my own environment?

What we've done is started to build out an open source tool, and we're actually going to open source this at the end of the week. It's built completely on automation. It's made of the composable ingredients.

Here are our requirements. It's not going to be intrusive. It's not going to prevent or slow down our builds. It needs to be near real time. And probably the most important thing is it needs to take about 3.14 days to create.

So the ingredients that we used, in our case, it was Chef. But again, pick your config management of choice.

The issue that we wanted to get was, I wanted a reliable source of what is every piece of software, operating system, and package that we're running in a particular environment, in order for me to then correlate that back and say where we might be vulnerable.

So gathering all of this information, we use Chef plus this tool that we're open sourcing called Tattle. Again, you can use Opsmatic, Logstash, Splunk, you pick your monitoring service. We use a lot of Logstash personally. Package managers could be Dockerfiles as well.

But this needs to be ridiculously simple, and we need an API that sits in front of it.

So in this case what you're seeing here is actually just a quick command that's going to tell me what is running, what are my reported versions of software that are running for a particular application or infrastructure. So I can quickly say Tattle, update software, and then whatever flags that I want to throw on there, and that's going to report back.

That's important because then what I can do is I've got this Common Platform Enumeration, or CPE, fingerprint of every piece of software and every package that I'm running on a particular box or boxes.

And I'm going to then correlate that back and say, "Okay, National Vulnerability Database, I have a list of CPEs here. Tell me if I've got any vulnerabilities associated with these CVEs." And I'm going to get that back in an XML format really quickly.

So obviously, got to parse a little XML because I'm dealing with NVD at this point. But now suddenly I know here are all the list of Common Vulnerabilities and Exposures, or CVEs, that I have in my environment really quickly, really simply. I didn't have to scan at all, mostly because I know what's running in my environment, which is the most important thing, is know your assets. Once you know your assets, then you know your vulnerabilities as well.

So what did we do? I won't go too much into this, but we're obviously dogfooding this ourselves before open sourcing it. We use it a lot internally, and then we hit our own API to actually load that data in, prioritize it, and then push out things.

And voilà, we have cake and a little bit of icing on that cake.

We started to pull in zero-day feeds as well, because I know everything about my assets. I want to know what are the vulnerabilities that I can fix, what are the vulnerabilities I can't fix, what should I do about those things that I can't fix?

And of course, there's some assumptions on here. One is the reliability of CPEs, we found, can sometimes not be perfect. Especially we will pull in things from NVD and we'll sometimes see Microsoft Windows described, or the wording used in it, one way versus another way. That makes it a little bit difficult to code around, but we figured out a few things to do there.

So to summarize, we're going to change this slide to DevOps and InfoSec equals Rugged DevOps. Right, Josh? Wherever Josh may be.

But the important point is, when I think about all of the issues that I had when I was a CISO at Orbitz, and before that when I was at Bank of America and a few others, I've spent, up until now, my entire career on the security practitioner side. And there was always a whole lot of pushback from ops. There was even more pushback from dev.

And I was not exactly the most popular guy in the room or the company.

But if the security folks can figure out ways to embrace it by actually fitting into the process, you can actually end up with a much more secure process and move faster doing it, which is obviously a win for all of the parties involved here.

So I mentioned there's a number of resources that I mentioned here within the presentation. I have them listed up here. Watch for Tattle on GitHub, which is where we're going to be open sourcing that tool.

I'm going to add an additional resource that we discussed. But with that, I have one minute and nine seconds for questions.

Thank you very much.