Zane Lackey

Quick background on myself. I actually started out in pen testing, so my background's all on the security side of things. If you ever had a pen test done from iSEC Partners or NCC Group or anything like that, that's where I started out.

As a funny story between bullet one and bullet two, in the very, very early days of Etsy, I actually did their first two pen tests. When John Allspaw and Chad Dickerson had first gone in there, I went in and did their pen test, and I handed them this report that was like, "This is a tire fire." They had just joined and were looking like, "Hey, from the security perspective, we're burning down this whole engineering side and rebuilding it from the ground up. Let us know where to focus from the security side."

Then about eight months later, we had a great conversation like, "Hey, you should come over and be our first CISO. Build and run the security program here," in the very, very early days of what then became called DevOps. They got to do probably one of the most rewarding things a CTO has ever gotten to do from a security perspective, which is on my first day, hand me back the security pen test report that I had written and said, "Ha ha, sucker, your problem now."

So it was a lot of karma that definitely I had coming. That was a fun experience. But yeah, went over to Etsy, got to build and run the security program from scratch in the very early days there.

The more that we talked to our peers, the more that we saw every organization on the planet, really regardless of size, going through this DevOps journey, right? Going through the journey of DevOps, the journey to cloud, digital transformation, whatever we want to call it.

So we kept getting questions from our peers of, "Hey, we see you guys have done kind of new ways of defending your web applications, your APIs, your microservices. Is there a product that we can buy to do that? Because the only thing we see are legacy vendors." And we, like idiots, said no forever until finally it got through our head that, hey, after we've heard this a million times from our peers, maybe we should actually leave and take our lessons learned and turn that into a product.

Flash forward to today, that's Signal Sciences. So now we defend kind of a bunch of leaders across verticals: financial services, healthcare, technology, e-com, retail, you name it. They can drop us in to defend their web apps, APIs.

Anyway, this is not a vendor pitch talk. That's the last you're going to hear about that. I would bore myself to tears talking about vendor stuff here.

This talk is what I wish I could've sat myself down on day one of going through this shift, and if I had 20 minutes, what I wish I could've told myself. The painful lessons learned that seem very obvious in retrospect, that are not obvious until you stick your finger in the light socket a bunch of times and learn these lessons.

We've only got about 20 minutes or so, so it's going to be high-level, but I'm more than happy to follow up with any... If any of this is interesting, we can deep dive on any of this.

To sum it, I'm going to give you the spoiler ending right now upfront, which is that security shifts from being this fundamental gatekeeper, like this outsourced gatekeeper, to focusing on enabling teams to be secure by default. This is the only way we scale, right?

Security as it's been done in a waterfall and kind of data center-centric model for the last 20 years, it doesn't scale, right? And the dirty secret of what we all see is that if security doesn't adapt, it just gets routed around, right? If it is slowing down the business, the business isn't going to stop functioning. It's that we're just all going to make this kind of DevOps shift without security, if it's not there and able to keep up.

So what has changed? Normally when I'm talking to security audiences, I spend a ton of time on this. I will not spend a ton of time here because I think this is all blatantly obvious to everyone in this room. But it's really that change happens orders of magnitude. It's not just one order, it's multiple orders of magnitude faster than it has in the past.

This was super apparent to me on my last day pen testing at iSEC Partners, where I was pen testing a really large US healthcare company. That last week, they deployed once every 18 months. I left there on a Friday, I took no time off because I'm a dummy, and started at Etsy on a Monday. Monday morning they sit me down, what, eight-plus years ago, and say, "Hey, we deploy to production 20 times a day. Figure out security. Go."

So I did the right thing, which is find the nearest bottle of security whiskey I could find and put that on my desk and get through that as fast as possible, and then started getting to work.

But the other side of things is that this decentralized ownership of deployment really changes the way that security can inject, right? Code used to go on this journey from development to QA, to staging, to security, to back to development, to back to QA, to back to development, to SysOps, to dev. It was like the Oregon Trail, right? It would ford the river of QA and get cholera in security somewhere, and then eventually come back and eventually get through there.

Which, by the way, even the half chuckle I got out of that one is better than, I used that in Moscow one time in a talk, and there are zero laughs about the Oregon Trail in there, by the way. So thank you. I won't quit my day job, though. Don't worry.

But from the security perspective, this really changes the way that we think about it. It changes the technical controls, and it changes the culture of how security interacts with the rest of the organization. So this talk's kind of a brief high level of that.

I did a full deep dive on the security controls, and I also did a full deep dive on the way that culture changes. So if you want either of those slide decks, just drop me an email. I'm happy to send those over to you.

But the biggest takeaway, from a high level, the one lesson I hope you leave with is that security can no longer be outsourced to as this kind of gatekeeper group. It's that security's fundamental role shifts from being this gatekeeper to focusing on enabling teams, whether it's a DevOps team or a development team or whatever, to being security self-sufficient. That's the existential shift that's going on here for security. It can only become successful if it bakes into the development and the DevOps process.

Security is at this inflection point in the way that IT procurement was probably six or seven years ago at the real start of the rise of cloud, which is that when you go to your IT procurement group and you'd say, "Hey, I need a dozen servers for this new project that we're starting," and they'd say, "Great, we'll get those to you in 14 months."

And you'd say, "Well, surprise, no you won't, because I'm just going to go put down a credit card and spin up 12 instances in five minutes and be good to go."

Security's at the same spot where it says, "Oh, you can't ship that to production. You can't do this project or anything like that. It's going to take eight months of review before you can go live."

What I really see is businesses say, "That's great. You figure out what you need to do, but we're going to be live in three weeks. So you do what you need to do, and we'll do the rest."

So security, the commonality I see in organizations making this jump really well is security focuses not on being this kind of isolated group of security experts that use tools only usable by themselves, but focuses on how can we embed as much security as possible in with the development teams and DevOps teams.

So if we don't adapt, I will give you a highly technical diagram of what security looks like in organizations where it doesn't shift, which is that the security train plows into the DevOps car while shouting about safety in and of itself, which turns out does not work that well.

So what are the new concepts that we should actually focus on from a security side, from a DevOps side? It's visibility and feedback, except these aren't new concepts at all.

Security loves to think that it is a completely unique skill set and completely unique everything. The reality is security is learning the same lessons that the DevOps groups had to learn five years ago at the start of the DevOps movement.

Performance monitoring, data analytics, A/B testing, all of these things that are kind of fundamental pillars of what make a DevOps journey successful, they're all about visibility and feedback, right? It's how do we get visibility of something that was previously a specialized skill set and bring this to our general development teams and DevOps teams?

Security is the exact same thing, right? Security is a subset of good engineering in the same way reliability is, and the same way performance is, and the same way every other facet of good engineering is. So these same lessons are really slowly shifting.

I will first give you an example from the old days, though. This is all real, by the way. I did not make these things up.

So this is how we used to have security visibility, and this is, for most organizations, really still how we have it today, which is you'd come in every day, and let's say you were an airline. Again, this was real in, I think it was '97. Yeah. You're an airline, you had your marketing website, and you assume that everything was fine because no one had told you there'd been a breach. You hadn't detected anything, anything like that.

Then one day you come in, and your marketing website has changed. In this case, they put your plane on fire, and they make the headline of the page, "So we killed a few people. Big deal."

Now, I am not in the marketing department, but I assume that that is not the sort of message you want to send to your customers if you are an airline.

But this really illustrates the larger point, which is we really haven't had visibility from the security perspective. We've always kind of invested in preventative controls, but we've never had any idea of what happens when they don't function, when those aren't actually deployed correctly, like what's going on there.

So we really have only had security visibility as a measure of from everything is fine or everything is on fire and there's a massive breach. Just like with performance issues, outages, all of this, we all know it's a million shades of gray in between, right? You don't go from zero to massive downtime. There's a whole lot of things that happen along the way, and we want visibility into these complex systems.

Security is the same way, and so how can we improve? I'll give you an example of which of these scales, right? The way that a lot of organizations start, and this is a good starting point, by the way. It's the right way to start, is we start thinking about, "Oh, our logs. Let's take a look at our logs. Let's see if we see weird things in our logs. Let's get visibility there."

It's a good starting point. It doesn't scale at all, right? Even at really small scale, this just becomes overwhelming. What you want to start thinking about is how do we get visibility in a way that we can surface it so that anyone in the organization can make use of this data and have this sort of context, right?

It's a lot easier to see something like this happening than it is to make sure you're watching the logs at exactly the right time for some sort of anomaly to happen in there, right? Anyone can read these graphs and say, "Wait, what? What just happened over here? Let's take a look at this."

So the real key out of this as well is surfacing security visibility for everyone, not just the security team. Again, it sounds super straightforward. This was a real hard lesson to actually learn, which is we started building a bunch of security visibility, and we brought it to the security team. What we learned is that that's great, but that doesn't scale. It doesn't help anyone.

So we started thinking about how do we provide that visibility outwards to the rest of the application teams and the DevOps teams, let them consume this data and self-serve that data. Because it's a huge parallel to how many folks use something like an AppDynamics or a New Relic or a Datadog, any sort of APM, right? And how many people have massive dedicated performance teams to go with that?

Yeah. Right? The beauty of these things was that you didn't need that anymore. You took a highly specialized skill set that needed a ton of dedicated people, and you brought that capability to the rest of the organization. This is the same real lesson from the security side, which is: how do we bring that sort of visibility to the rest of the organization and give them that so that they can really be empowered?

I'll give you a more tangible example of this, which is, this was an interesting one to learn as well, which is just a graph of HTTP 500 errors, right? Generic errors in a web app or an API or whatever.

What's fascinating is if you ask different groups inside the organization what this implies, you will get wildly different answers. It's really interesting.

So if you ask your development team what this is, it's, "Oh, yeah, we had a bad push and we either rolled back or we rolled forward, or we let the intern do their deploy and they broke everything," whatever.

You ask a DevOps team, and depending on the time of day, they might say, "Oh, this was the third time I got paged on call this week. I can't wait for Friday and to hand off a pager," or something there.

You ask your security team what this implies. They say, "Huh, that's kind of interesting. That might be somebody actually discovering a vulnerability and figuring out how to make it work against our system before they have a working exploit and working payload."

And if you ever get to ask your attackers what it was like to attack your system, more often than not, you'll hear, "Oh yeah, that was me discovering a live vulnerability in the system and figuring it out before I got a working exploit."

So the lesson there is really how do we bring security-relevant data along with operationally and development-relevant data so that when you look at that graph and you make all of these implicit assumptions, "Oh, that was my development team, that was my DevOps team, that was my whatever. Huh, wait, why is that actually happening at the same time that we're seeing a huge spike in attacks? That might not be the thing I assumed it was."

But anyone can read this data, right? That's the key, is you really want to be able to bring this in a way that anyone can make use of.

Then the other side of things is feedback. Now, before I threw this one in, I actually looked this up. Office Space is now 19 years old, which, A, I feel so old. And B, I worked at Etsy, so I think I get to say this technically qualifies as a vintage meme at this point. So we can now bring Office Space memes back into slide decks at this point. Congrats to us.

The two big lessons learned here are really the visibility side that I was talking about there, and the context, and the other side is feedback, right? How do we get feedback from everything that we're doing here to drive and make the process better?

How many folks do, whatever you call them in your organization, game days, right? Like game days, running tests and things like that, whether it's kind of the chaos engineering bit of doing a Chaos Monkey-style approach or walking into a data center and pulling a Cat5 cable and seeing if your resiliency is actually there.

Take those kind of lessons learned, maybe not that full extreme, but take those lessons learned and apply that to security. It's something that Aaron and James said really well in the previous talk, which is you don't want the first time that you're dealing with a security incident to be a real attack, right?

So the first time that if you're ever thinking about, where are the exits or where are the fire extinguishers, you don't want that to be when the building is completely on fire, right? You want to look at it and say, "Oh, there's a door. I know where I can go if there's an actual incident." It's the same thing from the security side.

The way that we've always done feedback in security is we basically haven't. We've done pen tests. And pen tests, we all basically, I say it as a former, maybe reformed pen tester, which is that we all typically do them annually. We do them a couple of people for a couple of weeks, and we always are stuck with this choice of, do we have our pen testers focus on one particular piece, or do we have them try to go super broad and get very shallow coverage over everything?

And the challenge is, either way, it doesn't work out well.

So there are things called bug bounties. I can give a quick background on this. The actual details aren't super important, which is that bug bounty-- How many folks have a bug bounty, actually? Let me just ask that, or know what they are.

Okay, a little bit. So bug bounties are, you can put out a policy for your web apps, your services, your infrastructure, whatever, and you can say, "Hey, security researchers. If you play by our rules, and you look at these sort of services that we published a bounty around, if you discover a security vulnerability, here's a way for you to report it to us, and we will reward you in some way."

That might be monetary, it might just be a thank you note, it might be a T-shirt, it can be whatever. But it's this kind of contract that you can put out that says, "Hey, we're all going to be good faith actors here," which is, if you find a vulnerability in our systems, fantastic. Report it to us and we'll handle it in a really professional and courteous way and reward you in some way.

This really comes out of the historical bits. I won't go too far into the dark ages of this, but in the old days, if you found a vulnerability in a service, you had a 50/50 odds of getting sued by that company. So security researchers, they had real backwards incentives, either to not report it or to just publicize it immediately and anonymously. Everybody lost out of the old system.

What's beautiful about these kind of bug bounties are that everybody can win. Researchers can report issues, we can all get them fixed as people who operate services, and everyone can kind of play above board. The beauty of that is it unlocks a whole different way to have a feedback loop from the security perspective.

Which is, if you have a bug bounty and you have pen tests, they don't replace each other, but they augment each other because bounties can give you kind of real-time feedback that, oh, an attacker just found this real issue over here, right? So we know that there's actually something important over here. It's not just a theoretical issue or something like that. There's something real here.

And we could actually treat that as a game day exercise and say, "Hey, what if this had been real? How would we know if they found something else? How would we know if they weren't reporting something else? What's going on here?"

A lot of security is doom and gloom. One of the final bits I'm going to give you is actually a success story out of this.

In super early days back at Etsy, we had just started investing a bunch in how do we get this visibility out there? How do we empower DevOps around bringing that security visibility to the teams?

And one day, and you can actually go to this link, it's still up there. It's awesome. It was written by one of our attackers, who turned out to actually be a really awesome security researcher, but we didn't know it at the time. They were poking around an API that we had, and they discovered a real vulnerability.

Because we had really started investing in that sort of visibility, we were actually able to detect one of our attackers discovering a vulnerability, and we pushed a fix before they could report it to us.

So they were still sitting there working on an exploit payload, and suddenly their attack stopped working. It was a really cool interaction because they wrote in and they're like, "Hey, so I noticed my vulnerability isn't working anymore. I can't imagine this is a coincidence. By the way, I've been testing from my home IP. Please don't sue me. We're all okay, right?"

And so we actually had a really fun back and forth with who turned out to be a really great security researcher. It was because we had really taken those ideas and started putting them into practice and started enabling that shift.

The reason that I wanted to share this is that I think what's so interesting about the shift to DevOps and this kind of broader shift is that security a lot of times will-- kind of legacy security organizations will think that that makes us less secure, right? Like, "Oh, we can't go to the cloud. We can't go to DevOps. We can't do this digital transformation because it's less secure."

I'm telling you as a security professional and a CISO who has lived through this, I thought the same thing in the early days. What I came out the other side really fundamentally understanding is the shift to DevOps actually makes us more secure. Because every development methodology, development infrastructure, whatever, it's always going to have vulnerabilities. But for the first time, we can actually react. Right?

How many folks have lived through... It's the final hand raise, I promise. How many folks have lived through the waterfall era and lived through waterfall shops? Yeah, we can all drink later at the bar from that.

But we all know the nightmare of it, right? When a critical issue comes in and we say, "Okay, how can we react? Well, get everybody in the room. Let's plan a six-week sprint around making an emergency patch to ship this thing out. Oh, guess what? It broke everything along the way. We need to get everybody back in the room for two more weeks."

It's a complete nightmare trying to ship emergency fixes. It's still the same vulnerabilities that we have in these systems. The beauty is as we go towards a DevOps sort of model, we can react quicker, right? If you need to do an emergency fix to something, it's just another deploy. If you have the ability to now deploy every week or every day or every hour, it's not a rush emergency fix, it's just another deploy. And so that's why it can actually make us safer.

But it only makes us safer if we actually have something to react to, right? It doesn't matter that we can react to outages if we have no way of detecting them, except our customers angrily calling us and saying the service is down.

We've really, as an industry, invested in how do we add performance monitoring? How do we have kind of reliability, visibility into these complex systems? We're at those same very early days, but from a security perspective, which is how do we get visibility into these systems?

All of these things actually increase our safety, but we have to have the foundational aspects that we've learned from other parts of the DevOps journey.

So thank you. You're awesome. Thanks for sticking around.

Q&A

So I deliberately held the talk a little bit short just so we have time for questions. So, by all means, please, any questions. Or some good heckling, either one. We can get another boo. Yeah. All right. Yeah.

Q: You talked about bug bounties in terms of visibility. What other examples or ideas... One of the thought processes I have for driving feedback or vulnerability reporting with that is, does security create tickets so that they get assigned tasks, put it in the backlog so they have a story to work on? Bring it directly to them instead of a report through their management. It's a static thing. It's in their face. Here it is. It's assigned to you. Get to it.

A: Yeah, totally. So the question was around how do we... And correct me where I'm wrong here, but it was kind of thoughts around how do we plug into the development process rather than security, whether it's a bug bounty or a pen test or a static analysis that generates a report and goes up the chain and down the chain and back down.

I think the answer is in the question, right? Which is that that old model, it doesn't scale. It doesn't survive the jump to DevOps, to cloud, to whatever. We really want to think about not only pen tests and bug bounties, but the Gauntlt stuff, the ChaoSlingr stuff, like all of this automation.

The real way to think about it from a 30,000-foot view is how do we have security approaches that plug straight into the development and DevOps tool chain, right? I want to get a Slack message when I'm seeing an attack against some service that's going on. I want to auto-file a Jira ticket when a bug is discovered or anything like that, right?

It can't be a static PDF that gets emailed to a VP and down to a director and over to a dev lead or anything like that. It just doesn't scale. So yeah. Please.

Q: I guess what I wonder is how do you get started in building a security DevOps culture in a company that just hasn't really worried about security? Or where it hasn't been--

A: So the question was how do you get started with a kind of security and DevOps culture in an organization where it hasn't really existed?

This is where I think a lot of people tend to fall into the trap of focusing on the technology first. The problem is not the technology. The problem is the culture and thinking about how does security plug in.

So often it's ignored because the only people-- Let's be brutally honest about security. For the most part, it has just gotten in the way and slowed things down. So a lot of organizations are very hesitant to bring that in in the early days because their only experience with it has been bad.

I think a lot of what makes for effective kind of security and DevOps culture is starting from the top and saying, "Hey, just like we're bringing in a new approach to the way we develop software and deliver software, we're bringing in a new approach to security. We're going to have to figure out some things along the way. Some of the choices aren't always going to be right at first, but we're going to iterate, we're going to adapt in the same way that we're thinking about all of the other DevOps process that we're bringing in."

Start from that messaging at the top and really saying, "Look, security's important to us, but it has to be able to be a part of this DevOps process. It can't just be that kind of slow gatekeeper that we're all used to."

That, I think, is the most important place to start. The technology can follow. You've got to set the cultural tone that security is not going to be this massive impediment, that it's actually going to kind of enable that shift. Does that make sense? Cool.

And then you get down into the fun stuff, all the technical bits there. But starting with the technical, I've seen that just walk into a wall every time if the culture isn't there at first.

Q: To change that culture, you have to hit the brakes differently. The best analogy I've heard is security's like the brakes on a car. How fast would you drive a car with no brakes? The brakes are what actually enable you to go fast, not slow. They do slow you down, but they're there to help you go faster.

A: That's-- And you've got to shift from it being what slows it to what accelerates it. Totally agree. Totally agree.

Q: I stole that, by the way. I stole it. I wish I was original.

Q: I'm just curious to get your perspective on sort of the $120 billion a year that's spent traditionally on perimeter security, web application firewall, endpoint, versus the $5 billion, which is arguably AppSec. So it's this very small category within this very big red ocean. Is there a perspective that you have at this point that suggests, I don't know, if some of the budget comes from perimeter to fundamentally building applications more securely, or the budget's new? I'm just wondering how does the world sort of reconcile that huge dichotomy?

A: Yeah. Absolutely. So the budget shifts for sure because the risk has shifted and the budget always kind of inherently lags behind that in security.

So if you look at, let's break through all of this and just from a real perspective, how does everyone get breached? If you go back 10 years ago, 15 years ago, it was the perimeter, right? You would go Nmap a network, you'd see all these extra services that were running. You'd compromise a server that was in a DMZ-ish environment, and it actually had a few VLANs that were set up wrong, and you'd move from there.

That's not how you compromise organizations anymore. The risk has shifted from perimeter and infrastructure to the web app level and to the endpoint, right? To the laptops and desktops and things like that.

And so the way in which our organizations are actually being compromised today is you either pop a web app or an API or a microservice, or you land on an endpoint via phishing or a client-side attack like that. Then either way you move through the environment to get to the data.

So the risk has really shifted, but the security spend has not shifted as fast. Where you still see we're still buying ludicrously expensive firewalls and IDSs and IPSs that don't actually mitigate any of the risk that we see today. That shift, it's coming. I see the leading organizations have already made the shift, but it's the start of the bell curve. It's not at the peak or anything like that yet.

Because the spend has to shift to keep up with the risk, right? The risk has completely shifted, but we're still buying firewalls and IDSs because we've bought them for 15 years and no one's really stepped back and thought, "Wait, do I actually need this anymore?"

So the real shift is defend your web applications, your APIs, that, and defend your endpoints. That's really where I've seen a bunch of the innovation really start to come out, and combine that with really incredibly, not basic in a bad way, basic in a good way, security controls, like Duo for two-factor auth and things like that.

Other ones, please. Cool. Thanks for-- oh, yep. The researcher who gets the bounty at the end. Yeah.

So actually, that one I've got one other super quick story for you. That's my favorite bounty story.

So that researcher, it was actually a moment we were really proud of, which they did that before we had a bounty. That's why we thought it was really an attacker, everything like that. What I was really proud of is when we launched a bounty, we went back and retroactively rewarded everybody that had reported something before we had the bounty. So they were one of them. We sent them a reward. Felt really good about that.

The story, one of my favorite bounty stories. So if you launch a bounty, it's a great process. I highly recommend it. There are companies today like HackerOne and Bugcrowd that help out with all of the really tough stuff that we had to do by hand way back when.

But no matter what, you're always going to get a few bad actors as part of that. Don't let it worry you. It happens to everybody. It's fine. But you'll invariably get like the extortion-attempt person who's like-- Okay, ours, we had a few. They're always made up. They never actually have the vulns that they say they do.

But we had one guy who wrote in, he's like, "I found a critical flaw in the Etsy mobile application. You need to pay me upfront, and then I will report it to you. Otherwise, I'm going to the press in 24 hours."

And we're like, "Okay. That's a great way to spend my Monday afternoon. That was definitely what I wanted to be doing. But no, we're not going to do it."

Honestly, we had a real soul-searching moment of like, "What if this is real? Maybe it's worth it to just do it." But we're like, "No. Listen, that's just going to invite more of this. We're going to stand firm on our policy, and we're going to write back."

And we wrote back and we said, "Nope. You need to play by the rules. You got to report it."

And they wrote back and said, "19 hours." I'm like, "Shit."

And so that kind of went on for a couple more hours. Then, I wouldn't be able to make this up if I tried. We sent back a follow-up and said, "Nope. Sorry. We're standing by it." And they follow up and say, out of the blue, go, "All right. Fine. My mom talked me out of doing a blog post."

I'm like, "Wait, what? If I could ground all of my attackers, the internet would be a much more amazing place, for sure."

So you will get some great experiences like that through a bug bounty also. They are not all doom and gloom, for sure. But yeah, you'll have some fun ones.

Q: Did they spill the beans? Did they have an actual--

A: No, they didn't have anything. It was totally made up. There was somebody else who tried to extort us for 20 grand because they were like, "Oh, we've got root on your production server."

And we're like, "No, you probably don't. Tell us anything about it."

And they're like, "Well, no."

Like, "Okay. Yeah, you don't."

You'll get a few of those. I think the most interesting thing running a bounty program, especially early on, I mean, there's only like half dozen of us who had them at the time, is how few bad actors you get. You would assume everyone would be super argumentative about the money, and you would assume that you'd have a ton of bad actors. Neither of those was true.

You can get really great results even without a monetary reward. You can just say, "We'll send you a T-shirt," or, "We'll put your name on a thank you page." And the amount of bad actors was a handful.

There were two times when people argued about money. Well, in both cases, it was when they didn't have the vulnerability that they thought they had. So there were the extortion ones that were just bad actors. But then there were a couple where they're like, "Okay, you rewarded me at this amount, but I think I should be rewarded at this amount."

And we're like, "Okay, well why?"

And they're like, "Oh, I have this severity of a bug."

And we're like, "Oh. No, you think you do, but actually, here's how your vulnerability works. It only impacts this set of customers in this way that are not even real customers in the first place. There's no data in anything there."

And so that's the only other time we've really saw people argue about the money, is they thought they had a more severe issue than they did, and we explained it to them, and they're like, "Oh, okay. Yeah, that makes sense."

So I recommend those things. I've done kind of a whole separate talk on bug bounties, so if you're curious about any of this sort of stuff, just drop me a note. I'm happy to deep dive with you on a bunch of this. We're all facing the same stuff at the end of the day.

So, thanks again. I appreciate it.