Joshua Corman

A hug, man. Hugs. We do hugs.

All right. Good morning.

Good morning. You guys ready to be awesome? You bet. Immutably awesome? Okay. I think he owes me a big jammie red for the Sarah Palin comment, but we'll see.

So this is the first thing that came to mind when I thought of immutable awesomeness, but this is more of a palate cleanser.

He gave me an intro. I'm Josh Corman. I'm the CTO of Sonatype, but I'm also the founder of two passions of mine: Rugged Software, which has led to the Rugged DevOps movement, thanks to Gene and others, which is really trying to create a culture that understands the awesome responsibility that comes with creating digital infrastructure, and how our increased dependence as a society on ones and zeros is becoming a matter of life and limb, national security, economic stability.

But also I deeply, passionately care about this other group called I Am The Cavalry, which is focused on public safety and human life and the Internet of Things, which is increasingly becoming a concern.

And John?

John Willis

Yeah. So I'm @botchagalupe on Twitter. Just been in this business for a long time. Been a core DevOps organizer, been part of the DevOps movement from early on, and more recently had a company called SocketPlane.

So I've done ops for many years, but had a company called SocketPlane where we sold it to Docker seven months ago, SDN for Docker. And then I'll tell you later more about how I think what we're talking about combines really well.

Joshua Corman

And as you can see, John is literally...

John Willis

I got toys.

Joshua Corman

...toe in the bag. Yeah. Okay.

So I like to think about motivations, especially in the DevOps culture. I think one of the most attractive things... I'm often asked, "Is DevOps a cultural thing, a toolchain thing, a process thing?" And of course the answer is yes. But to me, the greatest of these is the empathy. It starts with the empathy.

And I don't know how you're motivated, but I think you have to understand why we do what we do. And I think the reason we're both passionate about this is we think the patterns being discovered and uncovered and matured in this demographic here are transforming the world. It's transforming the way we interact with technology.

For me, I wake up every day and I want to save lives. I know it seems weird to do that through software, but as our dependence on connected technology is growing way faster than our ability to secure it, we're putting software into everything: our cars, our medical devices, our homes. And that software is very buggy and could put us in a very bad position.

But maybe if you're a developer, you care about being on time and on budget. Or if you're an ops person, you care about service availability. Or if you're a leader, you care about innovation and driving business.

So I've kind of figured out way too late in the security world that trying to put my motives onto you is a failing proposition. Any strategy that depends on a change in human nature is one that's going to fail.

I mean, what motivates you?

John Willis

So it's funny you ask that, Josh, because people tell me I'm very passionate about how I present and the things I present about. I do get incredibly passionate.

And then when I met Josh, I realized I'm passionate about operations infrastructure, which has been always fine for me. He's passionate about saving lives. And to me, I am such a fan of you because, again, I think what I do is pretty awesome and fun...

Joshua Corman

Yeah.

John Willis

...but what you do is amazing.

Joshua Corman

I think the reason that Gene's tribe of tribes is growing so well is because we're all passionate. We all bring complementary skills, and I think the way that we're going to save lives is because of the tools and the innovations that you bring and the others do here. So we've all had these ambitions and ideas. I think we're finally at a point in history where we can actually do some of them.

So this is how security people look at DevOps. I'd like to take credit for this, but I cannot. Some artist made this awesome picture, and then Pete Cheslock, the day we really bonded...

John Willis

That's right.

Joshua Corman

...added the little extra spice. But this is exactly how the security world looked at DevOps. I think Gene called it irresponsible, immoral, unethical. Why would anyone do this?

And as such, we knew that this was something we couldn't change, so many, many years back at the largest security conference in the world, we did a talk that said, "Security's dead." Right? This is the end of security as we know it. The way we've been doing it, aftermarket, late in the stage, bolted on, $80 billion, very ineffective.

I said to myself, maybe this is actually a good thing, right? Why would we fight to the death to maintain patterns and practices that don't really work?

And what we saw is, whether we wanted it or not, DevOps was so transformational in driving such business value, we had to get in front of it. Moreover, we had to be a third participant that was driving value with them. So we put our heads together and we said, "This is a really tough problem, but we can solve it."

Well, fast-forward, and I think before DevOps had really come into its own, we couldn't get it off the ground. So they had to expand the tribe with people like Jez, with people like Adrian, with people like John and Damon. And they really saw, to go even faster, to be even more productive, we keep butting up against things like compliance or security or Heartbleed or these other things. So maybe we need to expand the family.

And this was going really well until Jez realized maybe what he's getting into.

But the part that I struggled with when I first saw this is everybody knows this cliché. I just can't hear it the way it was intended. What he means, obviously, is that every company, regardless of what you do, is becoming a software company. What I see is software is not eating the world, it's infecting it. Right?

We're putting Bluetooth and software and connectivity onto every single thing in your life. It's spreading like a plague. And as a security guy who knows there's a certain defect rate per thousand lines of code, and we're putting 10 million lines of code or more into a car, this is not funny anymore.

And I'm spending a significant amount of time and energy with a significantly growing population trying to make sure that we don't have similar failure rates in the things that affect our families as we do in the things that affect our web apps.

So when I think about this, it probably became palpably real for you guys with the advent of the logo, right? Once you have a vulnerability with a logo, like Heartbleed. But for many of you, whether you care about security or not, this was unplanned, unscheduled work.

In fact, not only that, it was painful psychic context switching. You had to stop what you're doing. You're not going to be on time, on budget. You're going to be late for your commitments to the business, maybe late to market. Maybe you lose business to a competitor or time-to-market advantage.

And what's scary is this wasn't just an existence proof that open source has problems too. It's that as soon as there was a little blood in the water, there were 30 other vulnerabilities found in that one chunk of code alone. And it really brought to the surface that it's open season on open source, and that the shared value we get out of all this software supply chain comes with shared attack surface and shared risk.

And it wasn't just this one. We had Shellshock and Bash and all these other POODLE things with all these other logos. And when we started talking about this with DHS and the financial services guys, they almost looked at this as a too-big-to-fail moment.

The CIOs of the major banks said, "We can no longer size or manage our IT risk. We have no idea what kind of compound derivatives are lying beneath the surface in the code that we write and the code that we buy."

So the model we came up with was, maybe this is like the earthquake in Haiti. This 7.0 Richter scale killed 230,000 people. Everybody saw it on the news. But within six weeks, a much, much stronger earthquake happened in Chile that got almost no coverage whatsoever.

And one of the reasons for that is it only killed 279 people. So when all the people put their heads together and said, "Why was a stronger earthquake so much less devastating in death toll?" And the answer was building codes.

Chile had modern building codes. So it wasn't the presence of an earthquake, it wasn't the magnitude of an earthquake, it was: do we have building codes? And the aha moment was, we don't have building codes for building code.

And my first thought goes to, sorry about the Palinism, sorry. The first thought goes to Deming, right? Because you can't have an amazing... I don't know which one of us, that's probably why we go so well together. We both love Deming.

John Willis

We both love Deming. Love Deming. If you notice, my icon is Deming on my Twitter handle.

Joshua Corman

So I think we've done the best we can within the knowledge and the constraints that we've had, but it's not good enough.

And even if you're not as concerned as I am about public safety and human life or national security issues, look at the economic waste. My CEO did a poll for just the OpenSSL flaw, and every one of these bubbles is a batch of patches from a commercial vendor like an IBM, a Cisco, an HP.

And basically, the horizontal said 110 days later is the farthest to the right, and 110 products simultaneously patched. So that up and to the right, IBM, not picking on them, 110 products 110 days later, times every single one of those vendors to respond to one security vulnerability. And think of the lost GDP and productivity for all their customers who had to take that and deploy it and patch it.

So this is an incredibly wasteful thing, and Lean, we don't like waste, right?

So my favorite slide I've ever seen from Gene is the notion of visualizing technical debt. And just like all debt, it compounds.

Well, I'd like to say it's a metaphor, but here's an actual picture of the complexity of Maven Central, the largest open source repository of Java in the world. Now we have the privilege, and part of the reason I came here is we have a purview into who's consuming which projects, at which volumes, with which vulnerabilities, and are they ever taking the fixes or not?

And let me tell you, folks, this is a public health issue. We have horrible hygiene in our software supply chain. And these aren't even zero-days or exotic attacks from China. These are known vulnerabilities that have fixes available and no one's taking the fixes.

So at some point, that complexity, we can't keep going faster. We can't keep having awesome sauce through DevOps. We buckle under the weight of our own complexity, and we're going to start petering out. And many of you last year actually were expressing in your slides that we've done everything we can and we're starting to see diminishing returns.

So it's really not about just going faster. It's not about just doing more deploys per day. It's also, are we actually driving value?

And what that speed and velocity comes into tension with is things like quality, compliance, maintainability, sustainability. So it's not raw innovation we care about, it's net innovation.

And if you go back to Deming, and we keep borrowing and stealing from Deming, I'm simply suggesting that if Lean comes from Deming, and Agile comes from Deming, and DevOps comes from Deming, a fuller embrace of Deming includes supply chain principles.

And you'll note there's not a single security assertion on this slide, but this was a head-to-head comparison the book made on the Toyota Prius to the Chevy Volt. It was 61% the sticker price of the Chevy Volt. They sold 13 times as many units. They made half as much of their own car, so significantly less labor for Toyota than their head-to-head competitor.

But the most stunning part is they did it with a fraction of the suppliers. So with 125 suppliers, they made 10 times the use of each of them. No wonder it was a more successful car.

So could we put this into software? Could we use fewer and better suppliers or open source projects? Do we need 11 logging frameworks in healthcare.gov simultaneously? Can we use higher quality parts from those high-quality suppliers? In other words, only the freshest of ingredients, the most recent versions. And can we track which parts go where throughout manufacturing?

And I'm going to take the risk of actually showing a visualization of the cost of this poor supply chain hygiene.

Can we cut? All right.

So if software's eating the world, here's a piece of software. Okay?

Now, this piece of software is not monolithic. We don't write our own software anymore. While we weren't paying attention, 90% of a modern application is assembled from third-party parts.

Now, I asked several people throughout the conference, "How many individual third-party open source components do you have in an average product?" Most of you said about 50, and I think that's probably a good guess because you probably chose 50.

But our study of several thousand applications proves there's about 106 unique components in a modern application, because when you choose something like Struts, it chooses more downstream dependencies.

Now, after Heartbleed, we know that it's not going to be all green, healthy ingredients that go into our food. Some of them have vulnerabilities. The industry average is about 23% of those components we pick have a known vulnerability that was entirely avoidable. There was a bad version, and there's a good version sitting right next to it. We just didn't know or care to look.

This is elective risk, elective complexity. And even if you don't care about security, each one of these red circles represents the opportunity for unplanned, unscheduled work, lost productivity, and an opportunity to be more productive and more innovative.

Now, I wish that hackers were the only threat, but we also know about lawyers. So if you understand what happened with Cisco and the Linux Foundation and Free Software Foundation, they paid out several hundred million dollars in penalties for using GPL and copyleft. So these yellow ones are also, 8% of them have some sort of dangerous or restrictive license.

And I personally felt the pain when I was acquired by IBM. We had to kill two products because we were using a borderline license, and I had to kiss goodbye two babies of ours out of our portfolio that we had spent years working on, because it was just too hard to back out that dependency on that bad license.

So if this is the thing, what I want to call out is, let's say only 10% of those even manifest a year, right? So that's going to be two times a year I have to replace something like a Heartbleed or a bad Apache Struts 2, maybe $100 each.

So what we're talking about is not super hard at the individual app level. It's maybe two components out of 24, maybe 20 hours to fix it, 2,000 bucks, right? But here's the thing: none of you have an app. You're probably touching 10. Your division's probably touching 100.

And now what are we starting to add up to? We're looking at a quarter of a million dollars in wasted payroll, and we're looking more importantly at unplanned, unscheduled work that could have been delivering new features to your customers or new value to market.

And these are small numbers. I asked one of these presenters earlier this week, "How many are you working on?" And he said, "Josh, I have 2,500 apps."

And 106 each, we are talking about $6 million and 60,000 hours of unproductive work, right?

So if this is the food that we're serving at scale, many of you also have some sort of refrigerator you're storing your ingredients in. So let's say you've got some sort of code repository. It's got 300 projects in it. Each of them might have 15 versions of those 300 projects.

We actually had one person using 81 versions of Spring in production. Think of the maintainability, sustainability, quality, help desk calls, break fixes, lost revenue on e-commerce from maintaining 81 concurrent versions of a dead project.

And the thing is, of those ingredients in our refrigerator, most of them have gone spoiled or rotten. They're either buggy or less functional or have vulnerabilities. So let's say 70% of those are rotten.

So this one piece of food that we serve is sourced from a subset of those. And if we can take Deming's principles of fewer and better suppliers: do I need to support every logging framework, every web framework, every crypto library? Can I consolidate like Google has, down to one supported version of a library, maybe two, in production? And does that translate into savings?

And at least for one executive, he told his board, "Of the $1.5 billion we spend on software every year and 2,000 developers, if I start saying an invest list of projects and a divest list of projects, in one calendar year, we'll get a 15% boost in developer productivity."

By the end of the calendar year, with doing nothing more than an invest list of projects and a divest list of projects to get at the heart of the software supply chain, he got a 30% boost in developer productivity.

So not only is it the safe thing to do when software's eating the world, it's also the most economic thing to do.

So can we switch to slides, please?

And if you're wondering about, "Well, Josh, that only handles the known vulnerabilities," one of the data scientists at Ed Bellis's company did this stunning chart. Whoops.

In the Verizon Data Breach Investigations Report, of all the successful attacks last year, 98% of the successful exploits tracked to just 10 known defects, eight of which have had a patch available for more than 10 years.

So think of the math. I did this across the Fortune 1000. We could reclaim $1 trillion in lost productivity simply due to looking at the code hygiene of the open source that we're sourcing.

So hopefully you think this is awesome. I think it translates into three things: less unplanned, unscheduled work and break fixes; fewer service interruptions for ops, where you keep your five nines availability; and because stuff will always go wrong, that traceability and visibility translates into faster mean time to identify and repair.

You can either be six and a half weeks, like DHS took to identify Heartbleed, or six minutes, like insurance people took to look up, "Which of my applications have this issue?"

So the big question is: who likes chocolate? Anybody like chocolate?

All right. Spread it out. All right. All right, now. I can get over there.

Who likes peanut butter with their chocolate? I'm not going to throw this out, but I'll leave it on the stage and you can put your chocolate in the peanut butter.

John Willis

Let me just say something. So Josh is awesome, right? Unquestionably.

I saw this presentation in Austin. I am going to tell you about what I was thinking until I met him, and I knew there was something missing, but I was going to trudge on anyway. And when I met him, I chased him down, and I was so relentless that we both could have given our own presentations here.

And I said, "Josh..." And Josh was like, "Wait, I've got a big story to tell. I want to tell it." And I'm like, "Dude, we have to do this together. We have to split our time because this story together is going to be really powerful."

And so our metaphor is the chocolate and peanut butter. And Josh has his transition set of quotes, and then I'll tell you my story and why I'm so passionate about us both being up here.

Joshua Corman

So I think we bonded instantly when I was pointing out these Deming quotes and whatnot. And then he got up to give the next morning keynote and talked about how game-changing containers and microservices and data gravity were.

And then Nick Galbreath, who's not here but is amazing, he got up and said, "Who likes Josh's software supply chain?" Everyone seemed to. "And who likes John's immutable infrastructure?" And everyone seemed to.

And he goes, "Yeah, you can't have them both." And he essentially asserted that, like the laws of thermodynamics, operational pain can neither be created nor destroyed, just moved to someone else.

So while Docker and other containers give you the ability to liberate yourself from IT, it comes with opacity, that you don't know the provenance, origin, quality, or risk inherent in the code that went into building that container.

And we said, "Nah, we're going to get the best of both worlds."

John Willis

We can do it.

So I'm John Willis, Director of Ecosystem Development, Docker. As I said, I got there through an acquisition.

So the presentation I gave at that DevOpsDays was... How many people have read Diamond's Guns, Germs, and Steel? It's a great book, and I won't spend too much time on this, but he talked about how 10,000 years ago, certain parts of the world became haves and have-nots.

And I thought about this kind of convergence that's happening now. Very passionate about this, that there's this beautiful convergence between containerization, I happen to like Docker, that's my brand, microservices, and this concept of data gravity.

And I think this is the new Maxim machine gun, right? Where we walk in with weapons. I think the companies that adopt this... And just a short story on data gravity, because it might be confusing. It's an ideal where you move compute to data, where we've historically moved data to compute, right? There's a lot more on that subject covered by other people.

But in 2013, there were two blog articles that were pretty close to each other: one on Martin Fowler's blog called "Immutable Server," and then the one that really struck me, because at the time I was very big in infrastructure as code.

If you saw, I actually was early on at Chef. I did a lot of work with Puppet and Chef, and I saw this "Building with Legos" article, and it was really how Netflix was running immutable infrastructure. And everybody was up in arms because they basically said, "We're not using configuration management. We basically take AMIs, we bake them, we treat them like JARs, and we pull them, and we're immutable. We don't touch them when they're in the infrastructure."

A lot of good information there. I don't have enough time to go into the details, but this model of, when it hits the infrastructure, it is immutable. You don't change things. No CRUD in your infrastructure. It's always a replace and roll forward.

And then when I saw those articles, I remember this fascinating paper written by a true computer scientist in our business called Steve Traugott. He wrote this paper back in 2000, I think 2002, where he said why order matters.

And what he was saying in his paper is that at scale, the way you build systems and the order of which you build systems will actually matter, and it will matter in an OPEX way. Now, he didn't say that.

And what he described is three models of delivery. He said there was a divergence. Well, divergence is just unmanaged. So if today you're just checklist-building servers, you fit into that category.

Over the last 10 years, we've had incredible benefit of this concept of infrastructure as code. It was a convergence model actually really designed by a man named Mark Burgess with CFEngine, and Chef and Puppet and Ansible, and all these products have followed this model of convergence.

I won't say it's a problem. It's still a way of delivery. But convergence is a convergence, divergence, convergence, divergence. At any given time, you really aren't what Steve was saying, which was way ahead of his time, is that the best model in his paper, in his science and math, in his paper where he proves it out, is a congruent model.

And at the time, there really wasn't any consumable way to deliver that model. And when I discussed this with Mark Burgess, he said actually he kind of almost wrote this paper for, and I would say Docker, but certainly container infrastructure.

So when I got over to Docker, I became the resident DevOps person. "Hey, somebody, he's DevOps," right?

And right about the time I got there, Gartner had done this paper. We got an early release of it, and it was called "Assessing Docker and Containers: Five-Star Delivery Use Models." And it's actually an excellent paper. It was actually written by a release engineer who did release engineering for 10 years, really did an excellent job.

And so they asked me to write a blog article about this process. And just write about it, say, "Hey, how awesome it is." Well, I don't do that. I wrote what turned into a white paper, which is a three-part...

And because I'm sure everybody's read The Phoenix Project, right? Get out of this room if you haven't. Sorry. You can't be here.

Right, the Three Ways of DevOps, right? It is a clear way to think about what we do. So I wrote an article about why I think, and during the process, I coined a term, I'm pretty certain of this, called immutable delivery, and I'll talk about that.

And anybody who doesn't know the Three Ways, right, it's the left, the right, flow, the right-left flow, of basically amplifying feedback loops and, of course, continuous learning, kata, Mike Rother, Steven Spear, all those guys, great stuff to get feedback on.

But I do want to start with one thing. For the real true computer scientists in the room, right, you would be saying right now, "There's no such thing as an immutable infrastructure," and I agree. Things change. It's going to happen. So think of it as a true north.

Don't get tied up on, there's a lot of arguments on Twitter about, "Why do we call it immutable? That's horrible." And I just get over that and say, it's a model of thinking of how we want to deliver. It's a true north. All right?

And then I'll go back to how many people saw Damon's presentation yesterday, right? Oh, you missed a great presentation. You should watch it in the video.

But Damon has this picture in his blog. It goes back like five or six years. It's my favorite, to this day, single one picture that describes DevOps. It has everything we want to say and everything we've been hearing over the last couple of days, right?

We want to get from the aha to the ka-ching. We want to shorten the lead time that it takes to get there. We want to blast down the metaphorical wall that happens between dev and ops. And by the way, dev and ops is actually a metaphor unto itself. We don't need to rename it to anything. It's whatever to ever.

We are trying to remove the friction that allows us to deliver things fast, and then we must never forget the feedback loops and amplifying the feedback loops, right? So let's start with that.

So what was interesting is I thought I had read everything there was to read about Deming until I met Josh, and then he gave his presentation and he talked about this supply chain book. It was interesting. I had already written this article, and I called it "The Three Vs." And when I read the book, I realized there's a fourth V.

And so if you get the picture here, I'm starting to describe a model, I haven't clearly defined it yet, of delivering really fast, giving variety, a true velocity model, a lessened variability model, right? What Steve Traugott was talking about, order matters, and visibility, Guns, Germs, and Steel. I'll talk about that.

But when I read this book, I realized, oh my god, this is why I really wanted to do this presentation again, because I think the peanut butter and chocolate is this ability to move fast and have this incredible variability and decreased variance, if you will. And then if we apply supply chain to it, we have actually some magic in our industry.

So the variety is, I'm not going to spend a whole lot of time on this, is about the kind of Lean Startup model, which comes from Toyota. Things like minimum viable product, pivot. It's basically about choice and putting yourself in a position of choice.

The place that I think it gets really interesting when we talk about immutable delivery is really the flow, right? We've seen this flow hundreds of times, right? And if I break it down to the three pieces of flow where immutable delivery gets really interesting, imagine in a developer workstation where you can actually bring up a whole service stack.

Let's say that you've adopted microservices. Maybe it's six services. Four of them are other people's. You converge that in a virtual environment. You're running and testing what looks like a production environment. When you're done, you basically check in the binary artifact, the immutable artifact, into the flow, right?

And then at that point, it goes into a CI loop where it is not then built, and the order of the build matters. It's immutable. And then it gets tested in the integration flow.

What's interesting is on our podcast, we interview development managers, and they tell us constantly that now at the desktop level for development, their developers get mad when they can't rebuild a system in under two seconds, right? Imagine that, right?

And then you move that into an integration model. And I'll tell you right now, in web scale, I would almost say that running Jenkins as a Docker container is table stakes. Most of the build slaves, in fact, a Docker plugin runs a Docker-in-Docker for your build slaves.

I discuss this in the article, why that's so efficient. But what's even more efficient is this idea that when you have to rebase, and this is in the Gartner article, there's a great story early on in the Docker story about a company that wanted to boost scalable integration testing on a particular database.

So they had a Postgres table state that they actually wanted to test 1,000 times in the integration flow. And if you had to build that state of that table through an infrastructure-as-code, say, virtualization model, it might take four days. They were able to do this in, like, a few minutes.

IBM did a study about a year ago with Docker, and they were able to fire up 150 Apache servers in 36 seconds, okay?

So think about the scalability of test and the optionality of what you can do in testing when you have this velocity of rebasing, basing, rebasing the infrastructure in hundreds of ways.

And then it doesn't end there. And I'll be perfectly honest to you, there's not a whole lot of great in-production Docker stories, although we heard one earlier this week from Capital One. But in development, if we buy into, and I have, Jez Humble's books, Continuous Delivery book, right, for one, there's stories about ideas like blue-green deploys, canary deploys.

Imagine if a 30-node cluster needs to be deployed, or actually rebased out, basically, and you can do that in a matter of seconds. There's a high-frequency trading company in New York that actually rebuilds their infrastructure at 5:00 every day because they can, right?

And then I love variation. If you really want to go Deming, we can get in a hall and talk for hours of why variation was one of the core principles.

So here's the thing: Java lied. They said, "Build once, run anywhere." Except if you needed a runtime environment. Oh, and by the way, if it was going to go this platform and you had to convert the actual VMDK to an AMI to a... Think about all the variants there of even infrastructure-as-code build.

Well, the container model or the immutable model, whether it's Docker or rkt or whatever your favorite container infrastructure is, it doesn't lie. It basically is immutable all the way through the process, meaning that the binary artifacts that you create and test on your laptop, if it goes green through the pipeline, are the same bits and bytes.

And we heard yesterday, and if you've heard this often, Werner Vogels at Amazon says, "You build it, you own it." Imagine if you build it and you own it, and you know what the bits are across the whole stack. Right?

And this is why I thought about this idea of immutable delivery. And again, going back to Josh, this is brilliant, except you say, "John, that's a black box in operations." And I'm like, "Yeah, that was the one part I'm scared about." Right?

And what if we could get together and think about Sir Josh's amazing presentation? What if we could make the visibility of that... Again, if you're not getting excited, please leave now. Right? But if you're excited about this possibility...

Google does 2.2 billion containers a week. And you say, "Well, that's Google." Yelp does 15 containers a second. These are models that are changing the way... It's Guns, Germs, and Steel, folks.

But imagine if I could give you all that, and I could work with somebody like, or we could all work with somebody like Josh, and we could actually make this visible so those immutable black boxes, we know how to deal with them when we get Heartbleed and we can find them, and they're kind of bounded in the way they're delivered.

There's one final case study I want to talk about. It's at 2014 DockerCon, where... And so if you haven't heard about Gilt, it's an interesting startup that's been doing very good over the years. They basically are a poster child. They have great stories of how they went from monolithic architectures to microservices. Great stories there.

They also blatantly say, "We believe in immutable infrastructure," and they're a big Docker customer. But here's the thing, and the whole presentation is awesome, but if you're a junkie of this stuff like I am, at 28:04, he stops himself. He catches himself in his presentation. He has an aha moment and he says, "You know what's amazing about this? Is our developers now, when they do a check-in, they basically check in a couple of binary artifacts and one meta file. And then when they put that meta file, that meta file drives the behavior of those binary immutable artifacts through the pipeline all the way through."

And then he stops and says, "You know..." And you can tell he's doing this on the fly. He says, "You know, before this..." And they were considered a DevOps poster child before this. He said, "We had basically 10 years of releases, probably 100 repos, and well over 1,000 release engineering scripts made up of Ruby, Bash, all owned by different people. Some had left, some have died, some are Ops, some are in Dev."

And I call that the wasteland between Dev and Ops.

And this model, and if you listened to Topo the other day, this is the kind of model that companies like Gilt, Yelp, and now I'm about to say a bank is actually... And if you heard Topo talk about how they check in, they check in at the pull request. When they're evaluating the pull request, those are immutable delivery infrastructure.

Anyway, so winding up, we see some peanut butter and chocolate here. I hope you do, too. And we want you guys to add the bacon.