John Melton

[00:00:04.000] My name is John Melton. That's my Twitter handle, if you would like to reach out and ask me questions about all this. Later on, I'll talk a little bit about some tools. They'll be accessible on GitHub if you are interested.

[00:00:17.280] I do have to take a flight pretty soon after this talk, but I really deeply enjoy talking about all this. So if you'd like to chat, I'm more than open to that. I'll hang out for a few minutes. But don't think less of me if I hustle off so I can catch a plane.

[00:00:36.200] I am the AppSec lead at a company that Oracle bought called NetSuite. We are an ERP/CRM type company. Those are the stats: about 5,000 employees, about 1,200 developers that I'm responsible for.

[00:00:52.720] I have done dev and security work. I've gone back and forth. This talk is about security and moving it into DevOps environments. But I've done this at a few different companies, and on the side, I do quite a bit of stuff with OWASP. Everybody know OWASP? Awesome. This is the right crowd. Fantastic.

[00:01:14.180] I'm the project lead for AppSensor at OWASP, and the tool that I'll talk about today is Manna, but I've been with OWASP since the very beginning.

[00:01:24.960] And then the standard boilerplate. This is about a 30-minute talk. I think the standard suggestion is to cover 20 to 30 slides. This deck is 100, so you are not going to pick up everything off these slides. Please go grab them and look at them. I'm not going to talk about every point. I'm going to try to hit the highlights and hopefully get you thinking.

[00:01:46.670] Thank you so much for being here. There's really interesting talks going on right now, some I'd probably rather be at than my own, so thanks for showing up.

[00:01:54.520] This is going to be an honest retrospective. It's about two years of work that I've packed into this one deck, and I'm going to be very upfront and honest about what worked and what didn't work. Again, I think I've heard Allspaw's quote of, "Context is everything," quoted a few times at this conference. This is my situation. It's not necessarily going to be yours, but you should be able to pull some immediate applicability and be able to move forward.

[00:02:21.940] And then I'm going to hopefully give you some bullet points of what I would try to do if I had it to do over again, which actually I do, because I'm doing it at the parent company of where I started.

[00:02:33.500] When Gene asked me to come give this talk, I went to the website and said, "What is this conference?" Because I'd honestly not heard of it and had a bit of imposter syndrome. So I decided to go to the story approach because you all are very smart, but you can't really tell me what my story is. I thought that was the only valid thing that I could really talk about that would actually work. So we're going to take a story approach, and we're going to do story time.

[00:03:01.540] My goal is to essentially be this guy for you. If you don't remember this video, this is the guy who was teaching kids about gun safety and shot himself in the foot in the process. I'm going to try to take this approach of, I've messed up and I'm going to tell you how I've messed up, and hopefully you can avoid those lessons.

[00:03:17.420] Again, context matters. Depending on your perspective, depending on your environment, you're going to get different things out of this talk, or some of them you can throw away off the top. Hopefully, some of them will be useful.

[00:03:29.260] This may be the most important slide to give you context. I was working in an environment of a startup that was still very startup-ish. It was about 15 years old at the time, but it still has the startup feel. There were no security people, so this is the environment I got dropped into.

[00:03:45.440] The reason that I got dropped in was because that middle area there had acquired this company on the left. The middle company is still technically a startup, but they'd been around about 20 years, and they had much more of a traditional feel than the one on the left. They had a full security team, but part of the agreement was, you're not going to impose your security processes on our environment. Culture was important, so we wanted to work through this.

[00:04:14.140] They brought me in to deal with microservices and a DevOps shop because I'd been in that environment. These were the stats, and my job was basically to come in and get stuff done, and I was the one person there. These were my goals when I came in. I was going to update all of these things, essentially prep for compliance because they were starting to expand into other markets and they had to deal with things they hadn't done before, secure everything, and basically do it in a DevOps-compatible way.

[00:04:41.660] The first day I thought, "I've got this. I've done this at companies before. It's going to be easy." I came in, drank from the fire hose. So be humble when you go into a new environment. Don't think that you've got everything worked out.

[00:04:55.640] The first week was interesting because I'm an engineer, so I came in and I started looking through code, and I started seeing things that were not perfect. Then I started looking through process documents and the lack of process documents sometimes, and some of the stuff that had been around for a while. I felt bad for myself at the end of the week. I was wondering about what I was supposed to do.

[00:05:40.000] Then I came to this realization that everything is okay. I came into a business that's operating, that's making money. They're being proactive about hiring a security person, so they want to make things better. The business has run without me there. So I have to figure out how to help the business be safer, but continue to run and continue to make money, right? Because if they're not making money, I don't have a paycheck.

[00:06:07.112] My first quarter, there's eight quarters, so hang with me. I'm going to move pretty quickly through these. My first quarter, these were my proposed things that I was going to tackle: just update training and get to know people. Everything was working great. I made a decent dent in the training. I found the right people, which I think is critical, to be talking to the right people. That was one of the more useful things that I did in the first one. The first quarter was kind of a half quarter, so I didn't get a whole lot done.

[00:06:38.812] Second quarter, it's like, I know tech. I'm going to get this tech stuff fixed. I'll have it knocked out six months in, and then I can move to some other job, right?

[00:06:50.492] I had come from a company that built static analyzers. Everybody knows SAST? I should have asked this at the beginning. Anybody responsible for security at all? Sorry, it is super bright. There's like six hands. Awesome.

[00:07:06.612] I came from a SAST product company before I took this job, so I knew SAST. It's like, I built one of these things before. Installing one and running it should be pretty easy. And then dependency analysis was another big thing. I think a lot of the vendors out here would like to talk to you about this problem, but it is a big issue, and it's one that the security community had kind of honed in on at this point in time. So I was going to solve those problems.

[00:07:30.832] Static analysis went okay. I did a pretty decent job. A lot of that was owing to the fact that I talked to the developers and said, "This is what I want to do. Tell me where to put it." Right? I got nowhere on dynamic analysis, even though the tool is great, I just didn't have time. And then we did pretty good on the dependency analysis. We use an open source tool called Dependency-Check.

[00:07:54.312] I'm happy to talk to people about commercial tools after this and what we use and what we like and my opinions. I don't talk about that during the talk at all. But there's a few open source tools that I will mention. I will point out from a security perspective, there's huge amounts of low-hanging fruit here for dependency analysis. So if you're not doing dependency analysis, this is a really, really good place to start, and there's some easy wins there, and you should definitely look at that.

[00:08:22.392] Updating training, I actually did worse because I made negative progress because I found out there was more to do than I actually thought I had to do. The number one thing I wanted to point out here was, tools might not be the right place to start, even though that's where I started, that was my comfort zone. That may not be the right place to start.

[00:08:41.332] The number one thing that I have learned over the last two years is an application inventory. Who has an application inventory? You know all the applications that your company owns. Wow, that is remarkable. I've never had more than one hand go up on that, and I think that was a liar. Application inventories are notoriously difficult to get in the first place and very, very hard to get right. Whether you're talking about a static inventory, an operational inventory, gluing that data together is really, really difficult. So if you're not thinking about this problem, definitely a place to look.

[00:09:19.592] Q3, the main thing that I wanted to talk about here was credential storage and champions. Credential storage was an issue. We have hard-coded passwords, we have hard-coded keys, all that kind of thing directly in our source code or in some configuration files, environment settings, those types of things. We wanted to solve that. I like HashiCorp Vault here. It's a really useful tool for solving this type of problem. I'll point out how we did it later and give you some blog posts so you can go read lots of data.

[00:09:44.852] The champions program, if you're not familiar with this, it could be called lots of different things, but the security community tends to call it champions. Essentially what it is, is each Scrum team has at least one person on the team that's more responsible for security. That gives the security team this window into what's going on in the dev environments, gives the developer team responsibility over security things so they can control and manage and move quickly. This follows the model of you set the requirements, right?

[00:10:16.212] One of the worst things for these teams when you go into any organization is to say, "Ye shall do it this way." That's not particularly helpful because each team's going to have their own needs. So if you say, "You have to solve this class of problems, choose how you do it. If you want to use our tooling from the security team, great, we'll help work with you and get it set up. Otherwise, as long as you meet the requirement, we're there." That's what champions really does.

[00:10:41.432] It also gives people an opportunity to grow their career. So if you want to expand in security, this is a great way to do it.

[00:10:50.712] One thing I didn't mention here on dynamic analysis: I had punted on this because I couldn't get it done. We had quite a lot of Selenium testing and scripts and that kind of thing that were already being done with our QA team, and through this champions program, I found somebody who was really interested in security, and I said, "Hey, can you go take a look at this problem? Like, I would really like to solve this problem." And he said, "Sure." A quarter later, he came back with, he had learned this tool that we wanted to use, which is ZAP. It's an open source tool from OWASP for security testing, for dynamic analysis. He had learned that, he had integrated it, he had written a bunch of tooling, and he had it queued up, and it was running on our nightly builds for a particular application.

[00:11:31.832] So I did zero to make that happen, but through the security champions program, we got this going pretty quickly. That was really helpful.

[00:11:41.302] I'm going to make this point a little bit more emphatically later, but leverage open source where possible. Contributing back is a big thing for us. Also, I was struggling with how to build these programs, but I have learned over and over again that people on GitHub, Twitter, wherever you are, those environments, if you go in and say, "Hey, I'm interested. You seem like a really smart person. Can you teach me about whatever?" people really like to talk about themselves, so they will communicate back with you and give you lots of information, tools. I've had a number of people give me access to internal tools just by asking the question and we had this problem to solve.

[00:12:22.452] Q4. Metrics, this was a tire fire, and then threat modeling and tracking attack surface. So let's talk about threat modeling. Who in here does threat modeling architecture design reviews for security? That's about what I expected. Awesome. Two hands for those that didn't see that.

[00:12:41.792] Threat modeling is essentially helping developers think about security when they're building an application, or when they're designing an application, maybe is a better way to phrase it. Everybody talks about shifting left to solve more problems earlier. We know this time and again from all the research: bugs are much cheaper to fix in the design phase than in development, test, prod. Same thing holds for security. So we should look at these issues there.

[00:13:08.992] Tracking attack surface, this was another part and parcel to our application inventory. We didn't understand, because we had so many microservices, exactly what those things were exposing, and we couldn't keep up. There's one person tracking a couple hundred developers, and that's basically the model for security. In the general open market, you have about one security person to every 200 developers, depending on the data set that you believe. I don't know about you, but I can't code review for 200 people. That's not my job. I can't do that all day, every day. So the scale was an issue. We had no way to figure that out.

[00:13:48.272] For threat modeling, the thing that I would recommend is starting small. We chose the stereotypical STRIDE/DREAD model, but we did a super lightweight version of that. What we wanted was a diagram of what you were building and the data flows and that kind of thing, and then we wanted you to come up with a list of threats. We gave you a starter list of threats, and importantly, when you think about threat modeling, you can go down a rabbit trail really quickly, and there's this whole huge list of threats that you have to worry about.

[00:14:24.112] Most of those are canned. If you talk about the OWASP Top 10, most of those are going to be canned depending on your environment. So we don't ask people, are you worried about XSS? If you have a web app, we know you're worried about XSS. We ask people more, if you think about maybe fraud detection's a better place to start, think about how your application, the business logic of your application, can be broken or abused in an interesting way. A helpful thing here is abuse cases in addition to use cases. That was a good way for us to get people started thinking about that.

[00:14:53.632] Tracking attack surface, this was really helpful for us. What we did was, and I'll point you to the tool later, we leveraged the same thing that frameworks do when they start up. We're using a set of frameworks. When they start up, they have to know the routes that they're exposing because that's kind of their job. We just hooked into that and said, "Okay. Every time you spit out some endpoints that are accessible, we want to keep track of that." We really did it as a list, and then we diff those files over time. You might have to go out and buy the diff tool in order to support this, but pretty simple and straightforward. It's not perfect, but it gives us pretty good indication.

[00:15:43.032] You're doing good. We're halfway done with the basic set. So we're going to keep chugging along.

[00:15:53.272] We got acquired at this point in time, and the things that we started looking at now were containers, runtime intelligence, and particularly the app deployment tool. The things I learned here, the app deployment tool, I'll give you links to, and I'm not going to talk too much about that. But essentially, we were able to build quite a lot of safety into the app deployment tool.

[00:16:16.792] From a security perspective, particularly if you have a team of DevOps who is looking to optimize everything, everybody's looking to push down. Put more controls, put more things into my platform so that I, as the developer, don't have to worry about it. That's the reason that we have sidecars. That's the reason we have these proxies that are doing this work for us. That's the reason service meshes are a thing now. We want to push things into middleware, we want to push things into proxies. This is just a natural thing. We want to have less and less responsibility on the application itself. The same thing is true of security. If we can push all of that down into the deployment tool, it's much, much better.

[00:16:54.692] For containers, there's a, I think I've got a link to it in here. It's aging now, but surprisingly, a lot of the information's still relevant. It was written by the NCC Group, Aaron Grattafiori. So if you're worried about securing containers, that's a good paper to go read. It's like a Bible. It's about 130 pages. Really good. We followed that, and I was lucky because we were doing greenfield containers. I recognize some of y'all are not. But there's a lot of primitives in containers that gave us a really easy path, and as long as I figured out the few bumps along the way, we were able to set some really good defaults for containers, and that gave us a huge boost on security.

[00:17:37.572] Runtime intelligence, the idea here, you can go look up the AppSensor project. That's what that's all about. Essentially, the idea is that you can build fraud detection, for lack of a better term, into your applications themselves and build something called self-defending applications. Applications can track what's happening to them, and if you have a situation where somebody's misusing your application, if you keep track of that, you can now start detecting attackers over time and not specifically attacks. You can start looking for terrible behavior going on in your applications.

[00:18:17.732] From a runtime perspective, AppSec and OpSec is an area that's ripe for exploration and renovation. These are a couple of blog posts about the tooling that we built for operational deployment and how we baked security in. Please go read those.

[00:18:34.852] The next thing that I wanted to point out was core security libraries. We moved from a model of trying to tell everybody, "Hey, this is broken, this is broken, this is broken, this is broken." If anybody's familiar with DAST or SAST or IAST or RASP, any of that set of tools that come out of the security products community, most of them the way they work is you build an application, you get done with it, and we throw everything we have at it, and then we will hand you a PDF that's 7,000 pages long that says, "Here's all the broke stuff. Go fix it." That's not helpful to anybody, and hopefully we all agree on that.

[00:19:08.372] So we took the other route. We said, okay, here's the one paved path, the one right way to do this, and started small, and we gave them, this is the right way to do it. Then we said, if you deviate now from our policy of this is the right way to do it, you may not be insecure, but you're now a policy deviation. By giving people this paved path, it helped with new recruits. They didn't have to guess anymore. Like, okay, I need an ORM framework. Which one is it? Well, it's this one. That's the only one that we use and that we support, so go to that one.

[00:19:41.732] Some people from Netflix might disagree with this on the freedom and responsibility. I know their security team well, so they would argue this point, but it's been really helpful for us. We did well with this. One helpful thing there is to go in and write custom rules matching things to make sure that people are actually doing this properly.

[00:20:02.232] From my perspective, if you're in security work, you should stop worrying about killing bugs and start worrying about killing bug classes. That's where the interesting engineering work is. There's a great paper at the end of this talk from Google where they talk about they made a couple of bug classes obsolete at Google just by spending a man-year's worth of engineering effort. When you think about the amount of engineering effort going in at Google, a man-year is trivial. That's a rounding error. They were able to solve some of these problems in interesting ways. Good paper to read.

[00:20:37.472] A couple of final things. Immutable infrastructure. This was a big win for us by saying you're not going to undo something. We're not going to roll things back. We're just going to roll over top of them. We push forward. We make sure the infrastructure is not tweakable. If you want to get new code out there, you commit a change to the repository and it runs through the same process. This was really helpful for us. Kept people out of tinkering in production and causing snowflakes.

[00:21:07.292] The other thing is fast and slow checks. We split our pipeline from we want to block the build in some cases, but in very few cases. We put the things that were super fast checks, which a friend of mine refers to as the Grep Master 5000, which is the super fast things that you can check and get off the list, and what I refer to as you must be this high enough to ride. So as long as you can pass the basics, and there's lots of things that fit in this bucket if you just start thinking about them, then you can split off your find-the-pile-of-bugs-at-the-end in a separate asynchronous process. That was really helpful for us.

[00:21:50.052] Configuration management, infrastructure management tools turned out to be really powerful for security, particularly because it now gives us an environment that we can interrogate to find out the infrastructure state.

[00:22:07.552] The last one we're going to talk about is the application portfolio. We stink at this, just being honest. It's a really hard problem, particularly when you're at a company that has evolved over time and you have lots of different... We have all sorts of different VCS systems, we have different CI systems, we have different ways. We've got 37 Excel spreadsheets that track this stuff in different ways. We have different clouds that we're deploying to. We have data centers. There's stuff all over the place, and it's really hard to glue all that stuff into anything. There's no vendors giving us a ton of help here. So this is a really hard problem. If any of you are very smart and want to disrupt something, there you go. That's my best suggestion.

[00:22:53.312] How did we do? Compliance was obscenely time-consuming. We did a pretty good job on that. We got a bunch of checkboxes from different groups. We were able to embed. We did a really good job, I think, with the champions program. That was pretty helpful. They did way more work than I did. 100 people doing an hour of work a day is way more than I can accomplish in a week, or an hour of work a week is way more than I can accomplish in a week. Spreading that out and giving people a path to do things that they're interested in is really helpful.

[00:23:33.612] What can I offer to you to go home with today? This is a lot. I'm not going to touch on all of these things. I've talked through some of them, and I've got a lot of points and seven minutes left. So I'm going to try to hit the highlights. But please download this and look at it.

[00:23:55.252] I would say if I was starting over, or somebody was starting in a position doing the same thing, I would say focus on champions and say no rarely, because it's not particularly helpful to come in and be the historical... Who in here has a poor opinion of historical security teams? Okay. The rest of you are liars.

[00:24:18.512] Saying no is not particularly helpful. That shows you how immature security is and that we, as a field, don't really understand that the business has to operate. Saying no rarely also helps when you do have to say no, people trust you that you're not making something up, that it's actually an issue.

[00:24:38.292] The other thing I wanted to say was, if you talk to people, I found this more and more, people are more interested because news stories are coming out. If you talk to people just about general security topics, day-to-day life, how do I safely do online banking? Okay, that's trivial for people in security, but it's really helpful. It helps you find friends. It helps you find people that are interested in security. There's a site there, securityplanner.org, which is really helpful. So if your non-IT related friends and family need a place to go, that's a really, really great site that can help them.

[00:25:16.612] For process, to me, this is all about building relationships with other teams. Security doesn't tend to be great about this, but making yourself accessible, building relationships with other teams, and then understanding the SDLC of the individual group that you're integrating with, knowing what touchpoints you can hit, what touchpoints you can't, where you're going to be blocking, where you're going to cause problems, and then understand the business impact of that particular application. Because you may have a great team who's really, really interested in security, and at the end of the day, their app just really doesn't matter to the business. Well, okay, we have to work with the apps that matter to the business.

[00:25:58.332] From technology's perspective, these are, I think, things that I talked about quite a bit. But the two things that I would point out here is to focus on the big things. There are things that matter more than others. Cloud is a big enabler. It gives you a programmatically accessible environment. When you can tell an environment to change, and you can ask an environment about its state, that's game-changing from security's perspective.

[00:26:26.052] The other thing is to support your primary tech stacks well. Again, with Netflix, their model being freedom and responsibility, yes, anybody can use anything, but they do have a few tech stacks that they support really, really well. If you start with those, I happen to know from their security team, if you start with those tech stacks, most of the stuff happens for you. It's just automatic. If you go off the paved path, you can do that. You have the freedom to, but now you have the responsibility to find your own tools to meet all those other security criteria. So it actually turns out to be easier to work with those.

[00:27:00.872] Lots and lots and lots of homework, so please download these links. They're really, really great. That second link there is the Google talk that I talked about. The first one, if you are interested in doing security program management, Ryan McGeehan's been around for a long time. He's got a great series there.

[00:27:18.852] Let's try to get to the end here. Everybody knows this, I believe. Really old thing that's gotten resurrected, where people talk about the things that they fundamentally believe as core truths. So I wouldn't put this in that bucket, but these are things that I think out of this talk that hopefully you take away with you. Five things.

[00:27:39.712] If you're a security person and you can't code, you should learn how to code, or you should hire people on your team that know how to code. This is endemic in the security profession, and it's a problem. This is the reason that I only hire developers. I don't hire security people anymore. We know enough about security. Developers need to know how to code. If you're going to make an impact, you need to know how to code.

[00:28:01.232] We obviously don't scale effectively, so you have to build a champions program and build your security capabilities to be self-service. The champions program helps you with the people problems, and self-service helps you with the technology stack.

[00:28:13.592] We should look at more detection and response. This is something that obviously all of you in this room know. But security really only talks about prevention. They don't talk about, okay, I put this control in place. How do I make sure that that control is working? So we need to get there.

[00:28:27.472] Obviously, some of these things matter more than others. If you put in CI/CD automation, it matters way more than if you put in a grep check for some API call. So we can make a bigger difference, and we should focus on those. And you can't protect what you don't know. You should build an app inventory.

[00:28:46.652] This is my personal belief that you should be contributing back to open source. I'm not going to ask, but I believe all of you are using open source based on the talks that I've heard earlier. Most of us are probably in organizations or personally can give back. There's lots of ways to give back to open source. You can write stories. Stories are super helpful. If I had more stories about what didn't work, I would have done some things differently. That would be really helpful. Code is great. Documentation is great. Anything you can give, even mentoring is a huge thing.

[00:29:19.032] There's this project, I won't belabor the point, but essentially it's a security scanner that works on GitHub that is open source that you can go pull. Security scanner that works on GitHub that has exactly one rule, which is not great, I grant you, but it does something. It finds this one really weird, esoteric static analysis bug. But the point is that if you think about this one bug, I don't know how many times this thing has happened across all of GitHub, but if you could fix one problem across all of GitHub, how useful would that be? So if you start thinking about how do you scale out your efforts, really helpful.

[00:29:58.192] There's lots of different ways to contribute. Again, I believe that y'all can do it. It's open source. That's your homework.

[00:30:06.512] Yeah. "Hard to Say Goodbye" is what that's from, by the way. That's it. And since I'm from the South, Charlotte, where wrestling's a big thing, I don't think we have time for questions, but I can't not put that slide in. It's in all my decks now.

[00:30:23.172] Again, I'm going to scoot out of here fairly quickly, but I have about 15 or 20 minutes if anybody has questions. If not, just reach out to me, please. I enjoy talking about all this stuff. So thank you.