James Wickett

Hey, hey, everybody. Thanks for coming to this talk. We're going to be talking about AI and security. It's one of these things everybody is talking about, and I'm going to try to convince you of one core key premise. I won't try to spoil that just yet of what that is.

Really happy to be with all the DevOps leaders here in Vegas. I run a company called Dry Run Security. We've been around for about two years. We're a startup tackling secure code reviews for developers — but also for the benefit of security people to understand what the heck is even going on in our system at any point.

I'm a LinkedIn Learning author with my good friend Ernest Mueller. I live down in Austin, Texas. If you're ever in Austin, look us up — we have good meetups and a good DevOps Days crowd there. You can find me on X as @wickett.

01The core premise

Here's the core premise I want to implant in your mind, and see if we believe it's true or could be true:

The cloud was to ops as AI will be to security.

From the jump — who would say, yeah, I think that's true? Raise your hand. Okay, no takers. This is good. Couple yeses. You don't want to give in on the first question from the speaker, right.

But I think we have to go back and ask: what does that mean? Let's unpack that. What was the cloud to ops?

02What was the cloud to ops?

25 years ago, when we were thinking about Agile — Agile needed ops. We always needed ops. But it took about a decade until ops got to that level of readiness to participate with Agile and delivering software at speed and at scale. We also needed a computing revolution to happen.

At that time, devs and security — back then they didn't really get along. We know today that's not true, right? Anybody believer in that? Yeah, right.

There's a really interesting quote. I remember the early days of DevOps. We were sitting around — some of the people in this room: Willis, Damon — and there was always this question of "what is DevOps?" DevOps is culture. We were like, we're not going to define it. It's not going to be a thing. Then it was like "DevOps is Chef or Puppet" or whatever your flavor was.

But Tom Limoncelli — really, really smart, lot smarter than I am — has a quote in his book on IT systems management for cloud that posits: this whole DevOps thing, we really didn't create it. It was inevitable. It was the inevitable result that we needed to do operations in a distributed computing and cloud environment.

I think Tom's onto something there because it really took this mixture of all these things — Agile, infrastructure-as-code, virtualization, the cloud — all preexisting conditions that had to be there for DevOps to be born.

In other words: we didn't choose the DevOps life. The DevOps life chose us. (This is Ernest and me in studio reshooting the DevOps course for LinkedIn Learning. That really is the pinnacle of comedy in the whole deck. If that one didn't hit, I'm sorry — that's really where we're going.)

03What was ops in 2009?

- A lot of time-series data, a lot of event data coming in. - Devs outnumbered ops, like a 10-to-1 ratio. - Complex needs — network, storage, compute. - Ops suffered from limited developer experience. My coming into the office on Saturday was racking and stacking servers and laying networking cable. We wrote scripts, did some Bash and some Perl. But we weren't really part of the main dev experience.

Many people have talked about the first DevOps Days and how it happened. Patrick's actually here — I snapped a quick photo. I loved that time because sometimes John Willis would just parachute into your organization and teach you, "Here's how you do some Chef. Here's how you do infrastructure as code. Here's how you run WordPress using that." It was like — oh, this is meaningful. We can actually do something different here.

04Security is next? My naive thought

I remember at that time thinking — security is next. Security's going to get this. Devs got it with Agile. Ops kind of got the Agile vibe and went into DevOps. Security — it's going to happen, totally.

Some people were talking about it. There's kind of the Rugged stuff, the DevSecOps thing. But I don't think I really appreciated — and I realized how naive I was to think that — because they're different job responsibilities. They have different sets of data. Ops and security, they're not the same. They're not interchangeable, really.

05What's the current reality for security?

Security has all that same data that ops has, but they also have documents, reports, tools being run. They have to figure out compliance. They have all this extra stuff that you don't really think about from an ops perspective.

They're more understaffed — the 100/10/1 ratio: 100 developers, 10 ops, 1 security person. They have to fulfill complex job roles — scanning, audits, all that kind of stuff. They're surrounded by a bunch of data, dealing with all the CI/CD pipelines, lots of different languages. Even your really smart AppSec people are experts in one or two languages — not all 10, not all the things you've acquired.

So we have a bit of an impedance mismatch.

Steve Bellovin wrote the original book on firewalls in the late nineties. He has a book called Thinking Security. In his opener he says: "Clearly something is wrong with our industry, with InfoSec, with security at large. We're protecting the wrong things, and we're hurting productivity in the process." Breaches keep happening, they keep getting worse, and we're slowing everything down — the organization is not able to move as fast as it wants to.

06DevSecOps and "shift left" didn't fix it

Okay, cool — let's do the same thing. Let's have a DevSecOps event. Whose job is it? We never really figured that one out.

Then we thought: okay, we'll just shift everything left — that's going to solve it. But the developers who had everything shifted left had to deal with all the penalties of the shift-left. New gates, new complexity that developers didn't like. (Did I hear a clap? Yeah, sometimes that's the feeling — "those security people…")

The complexity continues to grow. The backlog gets worse. Developers are trying to decode all the output. There are all these tools that were written for a different audience at a different time in a different space, and now developers are stuck dealing with that. The pipeline gets all cluttered up.

AppSec is not easy — it takes at least 10 OWASPs… (a "10 OWASP top 10" joke). We have a high noise-to-signal ratio. Lots of tools emitting a bunch of stuff. You still need expert analysis to figure out what's even going on. Lots of things fly even past the OWASP Top 10.

You're dealing with a lot of documents, a lot of specificity, detail. You're taking your most valuable resource and stuffing them into "write a bunch of rules and patterns." Even the coolest, best security tools on the market are still stuck writing, tuning, dealing with that output.

We have other programs in security too — security training. We'll put everybody through security training, it's going to go great. Then it goes in a slightly different direction and becomes "we have to do this for compliance."

(Sad story: a friend sent me a photo of him watching a LinkedIn Learning video by me and Ernest "for compliance." I was like — well, thanks for watching.)

In the end, no matter how we build the programs, they end up in the same pattern — because we're missing the context of what's actually going on inside our applications. We don't know who's writing it, what the app is doing, what's important, what's critical. Are we thinking about how developers are interacting with secure code training? Are we making the code training relevant to the code they're actually working on? The brittleness of the code in certain areas?

07Contextual Security Analysis

I've been working on this idea of how we think about this in a more holistic way. I call this approach Contextual Security Analysis. I spoke about it some at this conference last year.

The idea: gather all the different pieces of context that's available as a developer is actually writing code, and make contextually aware assertions about what's going on. Think about all the stuff in your environment you can make assertions on: - Commits and PRs, authors - Code paths and which functions they touch - What dependencies they use - What your security tools are finding - Past problem areas — Jira tickets, bug output

I think of it in a SLIDE framework: - Surface — what the change affects, the language - Language - Intent — of the person making the change, and why is that important - Detections — what your tools are pulling - Environment — purpose of the app, how it's serving the organization, how it's being used functionally

So you're thinking about security in the context of the actual usage.

08What does AI do well?

Stuff AI does well: transforming, summarizing, extracting data, and generating (generative AI). It's really good at the top two or three; it's getting better at the bottom.

I said: the cloud was to ops as AI will be to security. Cloud totally changed how operations functioned, organized, built, and delivered software. I think AI is doing the same for security.

09CSPM/ASPM: the solution we deserve, but not the one we wanted

If you want to take a picture of this and put it on Twitter and give me the hate mail — with all the Wiz stuff going on, this is your perfect opportunity:

CSPM (Cloud Security Posture Management) or ASPM (Application Security Posture Management) solutions — they're the solution we deserve, but they're not the solution we ever really wanted. No one sat around being like, "Man, in five years I really hope I have something that gathers all that data together and prioritizes it for me." That is not what we wanted — we just wanted it to work. But because we built systems that didn't really work, now we're stuck dealing with this mountain of data and trying to come up with ways to organize it.

I'm not saying they're wrong, I'm not saying they're bad, I'm not saying you shouldn't buy one. I'm just meaning — no one wakes up thinking that's what we wanted to do with our industry.

The other premise of "shift left" is: okay, we're going to take all the scan results and load them over to the developers. Thank you, thank you.

10What does life with AI look like in InfoSec?

With these two in mind, what does life with AI look like in InfoSec?

- Stop the security burden for engineering — so they're not dealing with all that busywork and a shoveled pile of results into their bug backlog trying to decipher it all. - Security themselves should have less toil in their life — anybody in the room old enough to be part of the early DevOps stuff knows this was a key function of what we were talking about: operational toil was a real struggle. Breakout sessions were basically just people expressing this kind of pain to each other. - Insight for security into what's even going on in engineering — that is a real problem. No one even really knows what's happening in the security group. Sage nods in the crowd — great.

11Natural language as the detector — the encryption example

We take our most valuable resource — security people — and we shove them into writing pattern-based rules.

Example: I want to know "is this code change modifying encryption for any sort of sensitive data?" I'd write: what are all my decrypt functions in Python? Write up all the patterns. What are our sensitive data? Write up social security numbers and regex for a whole bunch of other stuff. Jam it in my pipeline. Then I'd have to go do that for Go and Java and all the languages.

What if you used natural language to do this? "Is this PR changing the encryption for sensitive data?" So stuff like stripping out a `get_key` call for an encryption function in Python and replacing it with a hard-coded value like 1,2,3,4 — your team needs to be able to see something like, "yeah, you did make that change, here's that change clearly explained." This isn't something you could have done with pattern-based rules. This change was not a predictable pattern.

The other thing we've been excited about: leveraging LLMs to pick out, of the hundreds or even thousands of changes happening in an organization in a month, week, or day — highlighting the really risky ones. They may not even have any of those OWASP patterns.

12James Berthoty: GPT-4 vs popular SaaS tool — 5/5 vs 1/5

There's a funny video where James Berthoty takes GPT-4 and a popular SaaS tool and runs through five code examples. Guess which one wins?

- GPT-4: 5 out of 5 code problems found - The SaaS tool: 1 out of 5

GPT-4 found all the code problems — almost untrained model — just a little bit of prompt engineering. After James released that video, he launched the Laia (Application Security Tester) on GitHub — you can plug in your own OpenAI key. Look at some of his prompts to get an idea: "You are an application security expert, skilled in explaining complex programming…"

13Prompt engineering patterns

I'm a big fan of the CoStar prompting method. If you haven't seen this, look into it as you're building out prompts. CoStar: - Context - Output - Specificity and Style - Task and tone - Assumption and audience - Requirements and Response

People have won AI challenges with this kind of model. It helps when you're building something out for your team if you want to wire in some LLMs to do that analysis.

Other tools worth checking out: - Jason Haddix's cybersecurity bot ("SEC GPT" in the OpenAI marketplace) — you can chat with it for a bit of threat modeling, and harness it as a pen tester. - STRIDE GPT — more in the threat-model camp. - Daniel Miessler's Fabric — has awesome prompts baked inside it (120+ and growing). Allows you to leverage how they're building prompts, extracting data, summarizing data. He's doing it for everything from academic papers to summarizing pull requests. Example prompt: "You are an expert in summarizing pull requests to a given coding project." It has a Summary section, a Pull Request section, and example output. Good project to check out if you're new to prompting.

14Revisiting the premise

The cloud was to ops as AI will be to security. How are we doing on votes? Any more agreement with that premise? A couple takers, a couple extras. (Steven, you can vote twice — I love that.)

We're seeing more and more come out: academic research. Some of it I'm a little less convinced about. Academic research is just like anything — some good, some bad. But if you use Fabric, you can figure out if it's good or not and score it.

We at least see the topic being explored: - LLMs hacking websites - LLMs autonomously exploiting one-day vulnerabilities - (Also linked the controversy where people disagree with this and some of their models/methods)

There's another AI security research paper that's even more promising.

15Why this works for security: security's data shape

What are security's data documents? - All the policies we write - All the procedures and controls - Architectural decisions recorded inside our systems - All the tools we run (security is good about having a lot of tools — those tools have to get processed, with extra outputs we have to figure out) - Threat models — sometimes built dynamically, sometimes evolving as we make changes - Industry standards and outside frameworks (PCI, whatever you're under)

If that was all they had, okay — not exactly aligned to ops. But then they also have all the same stuff ops has had: - Any change to the previous categories - Any code changes that happen - Any infrastructure changes - Any industry news — now we have to respond to dealing with whatever the latest greatest flavor of problems are

This is why I think the cloud was to ops as AI will be to security — because there is plenty of workload that's very well suited for AI or LLM-type approaches. That's why we're going to see this sea change where a lot of security tools are moving that way.

A lot of them are like, "yeah, hey, we do AI" — there's some AI baked in on all security tools, and we've talked about AI as an industry for a long time. But at this time it really does feel different. It really feels like we are making that progressive leap forward in using and leveraging what LLMs are really good for to solve some of these problems that are really discrete and important for security in particular.

16Close

If you thought my DevOps paintings were amusing, you can use the DevOps Painter GPT inside the OpenAI platform. If you didn't think they were amusing, that's okay too.

If you want to know anything about contextual security analysis, I have a Contextual Security Analysis Guide available on our website. I'd love for you to stay in touch. I'll try to mill around the conference. Thanks for your time today.