John Willis

So, "You Build It, You Secure It," maybe an introduction to DevSecOps. We'll see.

I'm John Willis, ThoughtWorks Group. I've done a lot of stuff. This is a 50-minute presentation that I've got to turn into 30 minutes, so go find me.

I was at Chef early, got to sell a company to Dell, sold a company to Docker. I've been part of this organization for quite a while. I was the only American at the first DevOpsDays. I got to be involved in The DevOps Handbook, and in a labor of love with my partner in crime, Damon Edwards.

By the way, he doesn't want to brag about it, but they got funding yesterday for Rundeck. But don't tell him I said yeah, because he actually doesn't want to brag. He told me, "No, John, don't go ahead and brag about it." He's my best friend.

All right, so we'll have a little fun. I'm going to blame this on John Rauser from his keynote, but I wasn't going to do this because I don't have a whole lot of time. But I was at Alibaba Cloud. It was a keynote, big deal. And for some reason, I made this horrible... I don't know. And those crazy fellows over at Sensu decided they'd have a little fun with me. They didn't let up.

There we go. And now they find other posts of me that they meme, and I'm like, okay, yeah. All right. And then this, and I thought, okay, all right. It's over. Enough is enough, folks.

And then what happens yesterday? Thank you, John.

All right. So, a little bit of seriousness. How many people have heard this phrase? Quite a few, right?

So it was actually in 2006. They were interviewing Werner Vogels, the CTO, and still the CTO of Amazon. He was saying, at Amazon, remember this is pre-coining the name DevOps, "At Amazon, we do things differently. Our developers don't throw it over the wall." He said a long list of things, and he said, "We have this thing where you build it, you run it," meaning that it's a service. Everybody owns the operationalization of the service.

Great. DevOps started, and it was great because it fit exactly what we're talking about. Early days, we talked about developers wearing pagers, ownership as the whole thing, collaboration, dev and ops working together.

So we got to Agile, right? This is my short, brief history of the world. Agile was awesome. It did all these things. It got the developers. But I've been an operations person my whole career, and it's like, what are those guys doing over there?

And all of a sudden, bang. They're like, "Hey, we got all this stuff." I'm like, okay. They didn't invite operations to the party. That was the thing. There was all this stuff about growing fast, the Agile Manifesto, and it was like, my perspective, wrong or right, was: boy, I wish we would've known about what you were doing.

And so that was really the birth of, hey, DevOps. John Allspaw did the 10 Deployments of Flickr at Velocity in 2009. It was a seminal event. Gene talks about this all the time. I joke, I was in the room. People were throwing up in the back of the room: "You can't do that. That's horrible. How could you do 10 productions a day?"

So about two years ago, I was asked, this doesn't happen a lot, it's the way I dress, whatever, to go to a high-level executive in a bank to tell them what DevOps was. And I was like, "Okay, let me do this."

So I put 15 slides together, and it was like the green room thing. I'll never be on Letterman or Jimmy Fallon. But there was this green room thing, and they reviewed my slides, and they're like, "You got five minutes."

There's a picture I use, I've gotten rid of it now, but in another presentation where the kid with the spaghetti all over his face, and my highfalutin, I'm the DevOps guy. I'm like, I'm an impostor. I couldn't explain DevOps in five minutes. I couldn't.

The joke now is if somebody busts in the door and says, "Jamie Dimon needs you right now. You've got to tell him about DevOps," I'll be like, "Oh, I can't go."

So I spent a year thinking about why I'm an impostor, and I'm still an impostor. I could never go in and do the three-minute Jamie Dimon pitch. I would fail miserably. He'd probably laugh and have a good time, but this is the best I came up with.

This is my definition. Like it, love it, doesn't matter, it's mine: DevOps is a set of practices and patterns that turn human capital into high-performing organizations.

In other words, it's not about the technology, stupid. And I promise you, I'm going to get to security.

And there's my son. I don't know if you've ever been to the Horseshoe. It's really cool. I used to have a corny presentation, "Take the Risk." It's a 4,000-foot drop. It's the most insane thing ever.

So we have The DevOps Handbook. You've heard it over and over. Continuous delivery, all the things. Everything here has been talked about. This is a great conference. It is. Yes? Come on, give me a yeah! Hell yeah. Come on, man.

All right, so with culture. Again, let's get the behavior right.

One of the things that we learned early in, like I said, I was at the first DevOpsDays in Ghent. It felt like a renaissance. There were less than 50 people there. Eight months later, Damon and I and a couple other people decided to try to run one in the US. There were 300 people in Mountain View.

I am pretty old. I felt like my whole career was like, really, I'm not doing any good. I did Tivoli, I did this, I did... I just felt like, when are we going to get to what we really preach? And I saw these young people talking about a different way, and it was insane.

And then we did some experimentation. We went out, we talked, we implemented. I was at Chef at the time. I was practicing what I preach. And the empirical data, or the evidence to me, was that there was something about this new way that included behavior, speed, and resilience. And I knew that, and the early conversations were all about this.

In fact, Courtney, go back and watch 2014 and '15 of Courtney at Nordstrom, and some of the things that she did, where she changed the whole way they thought. There are lots of organizations that have done this.

One of the things the DevOps study, Puppet Labs, IT Revolution, showed academically rigorous data that proved that this was bullshit. This was the iron triangle, the memory muscle that we've had for years, which was pick two. Or even worse, the reason DevOps started was: dev wants to go fast, ops needs to slow down.

And so out of all of our learnings, what we found is that there's a new triangle. I love that. I would say it's behavior, but let's just put it as DevOps. It's actually generative culture, non-pathological. It's blameless. It embraces failure. You get to go faster and be more resilient, and the data shows this. In fact, the faster you go, the more resilient you get.

So it's put to bed. And if you don't believe me, read the study. Read the paper by Jez and Nicole. Not white paper, academically submitted paper.

So how did we get here? This was the original Wikipedia, the automated pipeline. It's about building gates. We check in code, we automate as much as we can, we automate the pipeline, we build a lot of resilience by... Gene talks about the first way, second way, and third way.

The first way is about getting through the pipeline as fast, shortening lead time. The second way is telemetry and amplified feedback loops. Well, you amplify feedback loops by gating things. It goes here, red, go back. It goes here, it goes here, it goes here. It hits automated testing, go back.

And as this happens over time, and I'm still talking about software, I promise you I'll get to security, you are creating this, whether you want to talk about antifragility or you are creating this resilience. But you're creating a cultural immersion of speed and resilience.

So now I've got to go to Toyota. I won't say the D word, I promise. A couple people got that one.

There are some great stories. Toyota Kata by Mike Rother is an amazing book. And so there's a story in there, well, a couple stories.

One is the Andon cord. I imagine most people know about it, but I'm going to repeat it. It was a rope in Toyota where somebody could, at some point, depending on the timing, pull the rope and stop the production of a car line.

What was phenomenal about that culture, and why they destroyed the American car companies, there are a lot of things in it, Rother does the best job of explaining it, is the first thing that would happen is the floor manager would come over. Before he or she even knew what it was, they said, "Thank you, because you created a learning opportunity." The fact that you were willing to. So think about that.

Then there's another story in the book where there's a plant in Japan that averages 1,000 Andon cord pulls a shift, and they go down to 800. So the Western culture are like, "Yeah, 200 less defects."

No. Plant manager: "We got a problem. We are learning 200 times less a day." And even better than that, now it's an opportunity to tighten our tolerance. Let's add more of those red things.

One more story. I love Toyota stories.

So Toyota builds their first plant in Kentucky, their first all-in, we're going to do it by itself. Not the NUMMI story. After that. At one point, they're producing 2,200 cars a day. And some analyst, this is the folklorist version, who knows what the real... But the folklorist version is that an automobile analyst is like, "How do you build 2,200 cars a day?"

"Very simple. You pull the Andon cord 5,000 times a day."

Oh, by the way, this is four years ago. Google is that Kentucky plant. And all the other stuff is cool, and we could geek out about that. But four years ago, 75 million tests a day. Wow.

Last year, 150 million automated tests a day. That's pulling the Andon cord 5,000 times a day. That's how you build resilience.

And by the way, I heard somebody... I always steal great things from speakers. If I know the speaker, I give attribution. If I don't know, I just steal it. I mean, I can't remember everybody's name.

When's the last time you called Google for support? Right? There's something there, right? They're pretty good at infrastructure.

So we had DevOps. We got like, "Oh, we made a mistake. Agile, we didn't invite the ops people. Okay, we got that figured out." And this is a James Wickett steal. See, I do give attribution.

Here's my dramatic: "We forgot to invite security! We did the same thing over again. Oh, you're all idiots." Take my hat back.

So, in summary: Agile took us from months to days to deliver software. DevOps took us months to days to deploy software. But security is the bottleneck, and really good timing to pay attention right now.

I'll hold off on the Equifax thing, but you've got to have an Equifax in a DevOps presentation, especially if it's security. And you know what? I feel bad because Brandon Holcomb couldn't be here this year, and he's been here the last two years as a speaker. He works at Equifax. He's done amazing things there.

So as the meta points, it's 30 times cheaper to fix a security defect in dev than in prod. I have all the reference points for this. On average, a breach costs $5 million. We know from the DevOps study that high-performing organizations include security in the software delivery pipeline.

And by the way, those 15 lines of code in Node.js that you wrote, there could be a million lines of code behind it. One study says that 80, 90% of our modern applications use open source. And by the way, they're alive. I used to love at Chef, when you start watching the build, you're like, "Oh my God, who wrote that gem?" Kids.

Nothing against Chef, like Puppet, they all, Ruby. Hey. I don't like Ruby. I don't know.

So, a couple of people. James, I think you were involved in it, but Josh Corman is the first person who comes to mind for this, the Rugged Manifesto:

I recognize that my code will be used in ways I cannot anticipate, in ways it's not designed. It was longer than I expected. And I recognize that my code will be attacked by talented and persistent adversaries.

Here's the thing. If you ever get to hang out with Shannon Lietz, do it. Pay to hang out. She runs, I think it's a 35-person red team at Intuit. How many banks have a 35-person red team? She is the shit on security, and I mean that as a compliment.

She talks about adversaries, and she explodes my brain, because you know the bad guys... I'm not a security person. I'm diving in to be able to be a prophet to maybe help. But she talks about the adversaries. They are working hard, and DevOps has got nothing on the adversaries in collaboration.

The dark web. Now I'll ask you: how many companies, we probably have 200, 100 companies in this room, how many of you have collaborated about adversarial risks? And do you even know what your adversarial data is per hour? Who's attacking you? How often? And the bad guys are way smarter than you all are on collaborating when it comes to security.

And please come up to me and tell me I'm full of crap if I'm wrong, but I could imagine there's a lot of enterprises in here where everyone's like, "Hey, let's share notes." Oh, yeah.

So, things happen. Gene is obviously a connector. He runs into Josh Corman. I get to hang out with Josh Corman. We did a presentation together a couple of years ago here. And Josh was driving, in my mind, the rugged... And obviously, anybody Gene bumps into explodes. He's a force multiplier.

So we had this Rugged, and things are happening. We're vulnerability scanning. Cool. James Wickett wrote something called Gauntlt, sort of behavior-driven development. There's been a fair amount of tools there.

Cool. And I'm not going to pick at Rugged. I'm just going to tell you that Rugged was not interesting to me, because it seemed like I'll do something here, I'll do something there.

When I showed up at RSA this year, and I started talking to people, talk about DevSecOps, the first thing I noticed was they were talking about a holistic systems approach to security in the supply chain. And I didn't see that in Rugged.

An analogy I would make is if I went to a DevOps shop today, and they went, "Oh, we're doing pretty good DevOps. Can you come in, John, and see how we're doing it?" They'd say, "Well, we don't do anything in source control. And that Jenkins thing just throws error messages all the time. But we do do behavior-driven development, and we do do..." And you'd be like, "What the hell?"

But that's the way security looks at almost every company that I visit. Like, "Do we do this? We do this." "What do you do here?" "We don't really do it here."

So, the first thing I was involved with was a discussion on implementing DevOps in a regulated environment where security was top. And I won't go into detail because this is the extreme case.

But are you doing threat modeling in design? I'm going to give you a much simpler example here. But security requirements, are you doing static code analysis in your development phase, in your CI? Are you doing dynamic pen testing? All those things.

All right. Anybody know what this is? It's a really unfair question.

Struts 2. Who said that? Fuck. I like that guy, even before he said that.

Yeah, I'm not here to make fun of Equifax. I really am... I think that's a counterfactual. I think us looking back and saying, "Oh my God." The question is, I'll lose time, but what's your true north as an organization? What do you say that you do?

So I know a company that gives out loans. I may have the dates wrong, but this Struts 2 came out on March 7th, and the patch was two days later, but the signature wasn't out for a little bit. And this loan company was, on their web pages, "We put people in houses." This is our true north.

And so the woman who ran AppSec was another insanely cool person that I've gotten to hang out with, knew that this was a catastrophe on that Friday. Because she could have put the patch in and anything moving forward, but she knew that there was all stuff out there.

Where is it? Is it in WebSphere? Is it in some device? I don't know. Because we're not really good at knowing that, I've got to tell you.

But she went ahead and went to the senior executives and said, "If we're really this on our webpage, then the only solution I can tell you that we have to do right now is shut down the loan application." She pulled the Andon cord. And her manager, on Friday...

I don't know that industry too well, but I think a lot of loans get on the weekend. That's when I always buy my houses.

That sounded... I blame it on my wife. Every time I sell a company, new house. I'm like, "Ah, damn. Got to stop selling companies." This is fun, right?

So they shut it down. There's one thing for the AppSec person to say, "Hey, I'm calling bullshit." But it's another thing for the leadership to say, "Okay."

So she wrote a script like this. This isn't the script, because this is the Struts 2. And she hit... And you think, "Oh, just a simple script." All she did is read `/etc/passwd`. And she hit every URL in the company over the weekend, and it took time, and there were people screaming at her. Just imagine the chaos.

By Sunday night, she'd identified every place that this parsing routine, which is a routine in Struts 2, and mitigated it up on Friday. So it's not the technology.

And so they're thinking, "Okay." I think the Rugged, like I'm not, "Okay, have fun, you guys." But then I started seeing this holistic approach, and I'm like, I like to think like this. And I don't have to know security that well. I just get to find out who Shannon Lietz is, and this woman, and I learn really fast.

I'm just saying this is the DevOps thing, and you can draw any type of boxes that you want. This is just my simple... Because I'm just trying to make a box point. The products, I just picked one. I purposely didn't pick Chef because I'm an investor in Chef.

So you've got Eclipse, you've got GitHub, you've got Jenkins, you've got Selenium, you've got Ansible, and you somehow get to prod or stage. Right? That's what we do, DevOps.

So all I'm asking is, I want you to think about every box from a security perspective. Do your developers know the security implications of what that AppSec woman did? Do you have training? Don't just send them to OWASP.

I mean, I don't want to offend anybody who has worked on the website of OWASP, but it sucks. Your first pull-down, you want to basically start hitting yourself with sticks or something. That's not the place where you send the developers.

What you do is you translate that stuff to your own wiki. Speak in their language about the importance of the stuff that's... There's an OWASP Top 10. This loan company has a GitHub repo of all the right ways to code to mitigate the OWASP Top 10.

The developers show up at the story mapping. They're part of the requirements. They have IDEs. They have plug-ins in Eclipse that scan for vulnerabilities. This is real simple stuff.

There's a ton of products. You can do this whole thing open source, too, by the way. There's FindBugs, so the minute it gets committed, you basically can search there. In the build, you could go nuts, right? Sonatype, JFrog, all the things. You've got Veracode. You've got a lot of things.

And so the point is, if you think about Gene's three ways, the second way is amplify feedback loops. We could also call it shift left. But you cannot shift left if all the boxes aren't secured.

So if you don't... How can you? Oh, man, there's nothing there. I guess I can't shift left. Right, because what are we trying to do? We're trying to get anything we can understand down here as early as we can here. And we need to have that holistic approach, and I beg of you to make sure that you have something in every box.

And the security people, instead of telling them or saying, "Oh, here they come," they're the people who need to come and do this with you.

I've seen some great examples. General rule: the security people are the only people who make the CloudFormation or Terraform templates. Interesting. That's it. They create them all. They create the cookbooks. Interesting. That's a nice little head-fake thing. They get added bonus. They get to learn a little about your world. They catch things you don't.

Security wiki. Don't have a bug system for security and a bug system for your software. A bug is a bug is a bug is a bug. But if you think a security bug is different from a software bug, even saying that sounds silly.

Visualize things, have Git repos, have wikis, create sandboxes so the developers can like, "Oh my goodness."

Another thing that I find, this is really interesting. You go talk to somebody and they have a pretty secure mindset about AppSec and the delivery of behind-the-firewall services and library scanning, and some are better than others. Some are holistic, some hit some of the really good buttons, like don't let Struts 2 get through.

But then you go talk to another team and they're like the cloud folk, and then you ask them about configuration management and like, "Why? Why would we keep that in GitHub?"

And by the way, the NSA breach, see, I've learned a lot. The NSA breach was basically, they got in through an external-facing Amazon VPC. They hit the old Struts 2. They were able to get into a default-installed Jenkins on the inside one. This is a kill chain.

On the inside one, once they were able to run a script under a default-installed Jenkins, they basically created an Amazon account, scanned S3, and they got all of NSA. And NSA was a little bit lazy, I guess, because they left a lot of stuff laying around in S3 buckets, and that's how all that data got down there. It's a real kill chain.

So, configuration manage. Somebody might go out to Stack Exchange and do a cut and paste of a VPC definition. Like, it worked. Yeah, but those two, those are actually two ones that you didn't know.

I'm going a little fast because I want to end with a real meta point. These slides will be available and all that good stuff.

And then we can talk about containers. You need to think about your policy containers. So containers can be way more secure if you flip the narrative. Don't inherit in the wild. Create your own base images, XYZ base. Run some type of product to make sure that any container running is already a policy-based container.

Somebody goes, "Oh my God, containers. Docker is so dangerous, John." I'm like, "Why?"

"Because you can pull any image off the internet and run it in production."

I'm like, "You're an idiot if you do that."

There's this thing called scratch. You say, "From scratch," then you say, "XYZ base," and then you say, "XYZ Java," and then you run some type of policy thing. Twistlock does a pretty good job of that. You can do this open source project.

Again, I want to get to my meta. Vulnerability scanning, port restrictions. Scan for secrets. That looks like a password. I'm going to read it. I'm going to kill it. You didn't tokenize it.

And by the way, serverless, they don't need us. They really don't need DevOps. I've been to two serverless keynotes and like, "John, do we need DevOps for serverless?" I'm not going to say anymore. All the things are relevant. There's more configurations where you can make a mistake.

All right, so here's all the things and the things and the things. Google me. I have a whole presentation on DevOps Kaizen that I've been working on with Damon for quite a while. I think we've got a pretty good model of how you can think about improving an environment and thinking about it. I don't want to be sales-pitchy, but I want to get to my meta.

I've only got two minutes. So what did I say earlier? I was at a cybersecurity conference two weeks ago, and there was an anonymous hacker who gave the story of the hacker. They had a hoodie, all this stuff, it was on video.

And he was talking about, out of the five breaches that he's done in banks, four of them, he got in because somebody held the door open for him. As soon as he gets in, phew. Four of them he got in by holding the door.

And at one point he said, "We were idiots." This guy had never worked in the enterprise, right? How many times have you tried to let the door slam and gotten yelled at? And if you're doing that for 10 years, I see the smiles, right? You give up.

Diane Vaughan calls it the normalization of deviance. So you just give up. You're like, "I'm tired of getting yelled at." So he said, "Your people are your worst enemy."

And I'm about to present that afternoon with this slide. Like, wait a minute. No. Our people are our greatest asset. What's wrong here?

I'm going to have to steal a minute.

I don't know if you've read Bill Bryson. Bill Bryson, he's hilarious, but he has this thing where he talks about the history of everything in science. And he says there's a moment in time when two sailors are killing the last two dodo birds on the planet for fun. And at the same time, Newton is finishing Principia.

And he says in it, to me, it's the greatest one line in any book I've ever read. It's about in the middle. He said, "You would be hard-pressed, I would submit, to find a better pair of occurrences that illustrate the divine and felonious nature of human beings."

And so here's the thing I'm going to do with 15 seconds left. You could buy a billion dollars' worth of perimeter security. You could buy every product on Earth.

If I told you that that hacker told it, that he got in the place because he tailgated. Imagine the day in your corporation. I'm there. I've been working there 10 years. It is raining. It is freezing. Susie is pregnant, and she's got four boxes, and she's a step behind you, right? And you let the door slam.

And then you wait, because surely you're going to help her, but you had to let the door slam. And you know you're going to get, "Oh, man, she's going to kill me."

And she looks up, she badges in, and she says, "Thank you. You just made us a stronger, safer company."

And then you help her get the boxes to her thing.

I'm done. Thank you.