Jez Humble

I can categorically state that's untrue.

So I'm here today to talk to you about architecting for continuous delivery. And architecture: very popular topic, and very important, because it doesn't matter how much you pay the DevOps fairy to wave the magic wand and sprinkle the fairy dust. If your architecture doesn't fundamentally support continuous delivery, you're going nowhere.

So architecture and culture, for me, are two of the biggest obstacles to actually achieving high performance in continuous delivery and DevOps.

So when people think about architecture for DevOps, what do people think about? Anyone want to shout it out?

Microservices.

And if you're not careful, this is how it can work out.

Microservices is hard. There's a lot of complexity with it. And before we go there, microservices, there's definitely a lot of advantages to it. But I want to return to the first principles of how we build scalable, maintainable, reliable systems. So in order to understand that and the interactions between that and your process, let's talk about continuous delivery.

I'm going to give you my definition of continuous delivery. It is the ability to get changes, whether that's features or experiments or bug fixes, configuration changes, any kind of change, into production, or if you're not building web software, into the hands of users safely and quickly in a sustainable way.

So one of the things that drove continuous delivery was not wanting to be in data centers at the weekend. And one of the reasons we invented techniques like blue-green deployments was so that we could push out changes during normal business hours in a fully automated, push-button way whenever we wanted.

And so that's still one of my key metrics going into organizations. When you deploy: how many of you, when you deploy to production, it's in the evenings or weekends?

Okay, lots of you. So that's a sign that something's wrong. That means something's broken and it needs fixing, because anyone should be able to deploy during normal business hours.

And blue-green deployments, which you can find a blog entry on Martin Fowler's blog about how he did that, that was one of our earliest and actually very successful attempts to change that.

So that metric, number of person-hours outside of normal business hours you have to work when doing a deployment, your goal for that should be zero. And again, that's an architectural concern.

So in order to achieve this, there's two golden rules that you have to follow.

Number one, we want trunk, mainline, to always be deployable. So your software should always be in a deployable state.

And then the second golden rule, which is still the most controversial thing I say today, is that everyone should be checking into trunk on at least a daily basis if you're doing full-time software development. It's a bit different if you're building open-source software. But if you're on a full-time development team, everyone should be checking into trunk on at least a daily basis.

And these two things may appear to conflict. You'll be like, "Oh, but I can't do both of these things." You can. You just have to get developers to break down problems into small incremental steps they can check in on trunk and keep the software releasable.

And if you subordinate everything else to that constraint, and you just make that a rule, it's amazing what you can do. But it is very hard to get people to change their mindset on that, because developers like to go off on a branch and develop their feature with their headphones on and not talk to anyone. And then at the end, when they're done, they want to merge into trunk. "Look at the amazing thing I did. Isn't it amazing?" And then you break everyone else's thing because you refactored something, and now none of their code works, and they all hate you.

And by doing this, you force the developers to talk to each other a lot more. And the whole reason people become developers is so they don't have to talk to other people. And so this can be awkward, and it can cause problems. But these are the golden rules.

Once you've achieved this, or as you're achieving this, I should probably say, there's three ingredients you have to care about.

Number one is configuration management. It should be possible to provision new hardware into your production environment in a fully automated way, including PXE booting those boxes, installing the right version of the operating system, the middleware, installing the application, configuring all that stuff, configuring the router to send traffic to those boxes. That should be a fully automated process, ideally. Hard. Doable.

For developers, it should be possible to take a new person, add them to your team, have them start with a new box, and run a single command to check out everything they need to build and test and deploy the software, and run a single command to be able to deploy that to any environment they have access to.

Who is in that situation?

For whom does it take more than a week to get a new engineer productive on your team?

Okay, a few of you. So the next person you add to your team, their job is to write on a wiki all the steps that need to be done, and then get someone to help them start automating that, preferably the lead engineer. So this is the level of configuration management we're talking about. Again, hard, but achievable.

We then need to start practicing continuous integration, which is the practice of having everyone working off trunk and having a suite of comprehensive automated tests that run fast, that give you feedback within a few minutes as to whether you've broken the build, and then, crucially, fixing it when you break it.

So who in this room is working on a team that's doing continuous integration? Put your hands up. Hands up. Keep your hands up. Bit of exercise at the end of the day. Then there's beer later, so you can relax. Hands up.

Put your hands down unless everyone, all the engineers on your team, are checking into trunk, not feature branches, but trunk, on at least a daily basis. If that's not true, you put your hands down. Otherwise, you can keep them up.

Okay. Put your hands down unless, when the build goes red, it's almost always fixed within 10 minutes.

All right. So there's about five people in the room doing continuous integration. Big round of applause for those people. Whoo.

Continuous integration is super hard. It's not running Jenkins on your feature branches and then ignoring the build when it goes red. It's making sure that your software is always in a working state, and that you prioritize keeping it working over doing new work.

In order to do that, we need test automation, comprehensive test automation, and multiple different levels. And this is the painful and expensive bit, often.

Once we do that, we create a deployment pipeline.

A deployment pipeline: any change we make to the system, we check it into version control. That triggers unit tests. Those are not comprehensive. They run in a few minutes to give us some level of validation we haven't done anything really dumb. If they break, no one checks into version control unless they're fixing it. And then we fix the problem. That triggers unit tests again. Those pass.

That triggers automated acceptance tests, which typically will take of the order of thousands or tens of thousands of hours to run on any reasonably complex system. We'll run those in parallel on a massive grid, get our feedback within the order of tens of minutes, hopefully, or a couple of hours if it's a really complex system.

Again, if those break, we've prioritized fixing them straight away. We don't necessarily gate check-in on having them green, but we'll make sure that someone's working on fixing them.

Once you have builds that pass all the automated tests, they will go downstream to people doing exploratory testing, performance testing, UAT, integration testing, and so forth. But we don't want to waste those expensive people and resources against builds that are not known to be good based on the automated tests.

The whole point of the automated tests is to give you a high level of confidence that what you're working on is deployable, so that we don't bother integrating, testing something that's broken.

Once your software's gone through all those stages, it's ready to release to production. We should be able to push a button to release to production. And if we feel even slightly nervous doing that, what that means is that our tests and validations are not good enough, and we need to improve them and make them better.

And there's feedback cycles. If you have a broken functional acceptance test, that means you're missing a unit test. If you find a defect in your exploratory testing, you need to look at the functional test that you're missing or a unit test that you're missing. So there's feedback loops here.

The goal is every change results in a build, every build is a release candidate, and the job of the deployment pipeline is to prove that that release candidate is not releasable.

We can't prove complete correctness except in the context of formal methods, which most people don't use. But what we can prove is the existence of known defects so that we can fix them straight away while they're cheap to fix.

In order to be able to do this, in order to be able to validate our golden rules, there's architectural considerations that we have to follow.

We need our software to be testable. And what testable means is when someone says, "Works on my machine," that actually means something useful.

This is the whole purpose of technologies like Docker and Vagrant and so forth: to be able to replicate enough of your production environment on your developer machine that you can actually do testing that tells you something useful about your software, which gives you some level of confidence that it is actually deployable.

If you have to buy or provision a really expensive integration environment in order to get confidence that your software is releasable, that is an architectural problem that you need to fix. It's a symbol of something being wrong.

And, you know, technologies like Vagrant and Docker are part of the solution. But no amount of Docker is going to fix the fact that, in order to make your software work at all, you need to spin up a mainframe and an SAP instance and a bunch of other crap over here in order to get any feedback that what you're doing is useful.

So architecture: very important. We'll talk about that a little bit later.

And second, deployability. We want deployments to be a low-risk, push-button affair, not an enormous set of documents that have to be followed over the course of an entire weekend that are not up to date. Because it's humanly impossible for them to be up to date.

And making sure that you can make deployment a low-risk, push-button affair, you can't just take a complex, painful, manual process that's error-prone and automate it, because then what you end up with is a complex, painful, error-prone automated process, which is not actually much better in many ways.

So there's a piece around simplification of the architecture to enable continuous delivery. If you take an architecture that's very tightly coupled and complex and try and automate it, you're going to find it's extremely painful and difficult, and it's probably not going to work. And actually, you need to architect for this stuff from the time being in order to be able to achieve those goals.

So microservices are one way to achieve these goals. And there's particular implementations of microservices, like the Twelve-Factor App, that allow you to do this. But it's not the only way to achieve these goals. You don't have to use microservices.

If you do, there's a great book by Sam Newman called Building Microservices, which talks about it. But it's not the only way to achieve those goals.

My favorite quote about internet architecture comes from Jesse Robbins, who was Master of Disaster at Amazon, then was co-founder at Chef, and now he's founder of a company called OnBeep.

And back in, I think, 2007, on O'Reilly Radar, he wrote an article about operations as a competitive advantage where he said this one sentence, which encapsulates pretty much everything you need to know about building web-scale architecture:

"Operations at web scale is the ability to consistently create and deploy reliable software to an unreliable platform that scales horizontally."

So just in that one sentence is 90% of what you need to know in order to build scalable, resilient, distributed systems for the internet.

So we've got continuously create and deploy. That's continuous delivery, which wasn't invented then, but people were doing it. We just wrote it up.

And then you've got the reliable software, which again, you've got to architect for. And then you've got the fact that your platform is fundamentally unreliable. You're going to have network partitions. That's going to happen. And that causes you to have to make architectural trade-offs.

Most well-known of those is the trade-off between consistency and availability that CAP theorem tells us about in the event of network partitions. But you can't rely on the platform being reliable.

If you want your software to have an SLA that's better than the SLA of the underlying platform, you need to architect for that shit. You can't expect the platform to do that work for you, as we all know.

So how do we achieve this?

We've got to go back to some very fundamental principles about how to build software that we've known about since the '70s.

So componentization. Components, modules, services: these are all horribly abused terms. But fundamentally, I like Martin Fowler's definition: "Part of your system that could be swapped out for another implementation." So something behind an API is another way of thinking of that.

Why do we care about building componentized, modular software?

Firstly, we want to make the system more maintainable. So if you read the original papers about modularization, the best known of the original ones was by David Parnas, where he talks about two possible architectures for a very simple system. And then he runs a thought experiment about adding a feature, and he says, "Well, with this decomposition, to add this feature, we need to change both these pieces. With this decomposition, to add a new feature, we only need to change this one piece. Thus, this architecture is better because I don't need to touch as many parts of the system to add a feature."

So that's still a very important way to think about decomposing systems into components or services. The point is to make the system more maintainable, and there's well-known techniques like good encapsulation, low coupling between modules, things like the Law of Demeter, and solid engineering practices that enable this. Very well-known. Not sufficiently practiced.

The second benefit of this technique is to make the system easier to build and test. Instead of having to build everything all at one go, we can build just the part we've changed and test that against the other things that we've already built, and it's faster to get feedback on whether we've broken a thing.

And then finally, and perhaps most importantly, it enables collaboration at scale.

And to understand why your architecture enables collaboration at scale, we've got to turn to what's perhaps the most important law in architecture, which is Conway's Law, which tells us that organizations which design systems are constrained to produce designs which are copies of the communication structures of those organizations. Which is a bit of a mouthful, frankly.

Eric Raymond has a terrible computer science joke, which is, "If you have four teams building a compiler, you will end up with a four-pass compiler."

Gotta love architecture jokes.

The essential point that we're trying to make, I think, actually, is made really well by Rebecca Parsons, who's CTO of ThoughtWorks. She says, "If you don't want your product to look like your organization, change your organization or change your product."

I think that encapsulates very well what Conway's Law is about.

If you have a bunch of teams in different parts of the world, within those teams, you have very high communication bandwidth. Between the teams, you have relatively low communication bandwidth. And so if you need to add a feature, adding a feature requires high communication bandwidth.

How do you make sure that your organization is structured so that you don't require low-bandwidth communication channels in order to make things work?

And so the way that Amazon, for example, solved this is to have one team per service.

So a service, by the way, is an end-to-end implementation. So it includes all the different layers of whatever architecture you're using for that service, and different services can have different internal platforms that they're built on. We don't care.

But to add a feature, you're only ever adding a feature, in general, to one service, again, if you decompose your system correctly. And so that team has very high communication bandwidth. But other teams that depend on that, they don't need to know about what's happening inside. They only need to know what the API changes are.

So here's the key: you align your low-bandwidth communication channels, which are the inter-team communication channels, with your API boundaries. That's it.

And what you avoid, in particular, is splitting teams based on architectural layer, so having a web team and a middleware team and a database team. Because guess what? To add a feature, you need to change all of those things.

Or decomposing teams by function and having a dev team and a test team and an operations team.

The way to maximize the benefit of the different communication bandwidths that you have is to decompose the system so that adding a feature only affects one service, and then making sure that all the people required to build and deploy and maintain that service are co-located.

So don't organize by layer or by function if you want to move fast.

Then there's two ways, fundamentally, that we can combine these components or modules. And then it's not binary. They're kind of two ends of a spectrum, and in reality, you'll be kind of somewhere along that spectrum.

At one end, you can bind the components at runtime. So you have lots of tiny little services, microservices, that are bound together that you can independently deploy, effectively. So they're bound at runtime. When you deploy them, that's when they bind with the other services.

There's some trade-offs to working with this model. This picture, by the way, is what's now called, apparently, the Death Star.

So this is all the services in amazon.com and the connections between them. And any kind of sufficiently large organization will proudly display its Death Star as their architectural diagram so you can marvel at the amazing destructive power of their architectural decisions.

And one of the key practices you need to follow when you're doing this is your team is responsible for making sure that downstream teams don't break. If you're going to release a new version of your microservice, it's your problem if downstream breaks. It's not their problem.

And so you have to defend against that by using techniques like A/B deployments, good testing. API versioning is another way to do that. And you need to think about these things from the beginning.

So there is a bit more upfront thought involved in going with this model. API versioning is still an unsolved problem in some technologies, and it's something you need to think about if you're going to do this well.

And then also you have to think about monitoring. So if you have performance problems in a Death Star, one thing you can almost guarantee is that it's not one particular service. It's probably some weird emergent interaction between a bunch of different services that's causing your performance problems.

So if there's no way to look at an inbound request and then trace it through the various different services and actually look at where the latency is cropping up, you're dead. You can't do it.

So there are complexities involved with this model.

At the other end of the spectrum, what you can do is bind your components at build time. So you still have a componentized architecture, but you build one enormous binary at build time, and then you deploy that whole thing.

So this is the technique that Facebook use. They have an enormous binary that's, I think, hundreds of megabytes. And that was a couple of years ago. I don't know what they're doing now. But they just send that whole thing out to the cloud through a hacked-up version of... What's that? Oh, BitTorrent. They have a hacked-up version of BitTorrent they used to push that huge binary out to all the different servers.

Etsy have a monolithic architecture that they use PHP.

PHP is one of these technologies which I kind of love, because on the one hand, you could say it's monolithic. On the other hand, you could say it's the ultimate microservices architecture, because every PHP file is basically a microservice.

That's what I choose to believe.

If you are deploying a huge binary, you need to make sure that you find any problems caused by the interactions between the libraries and components you're consuming as fast as possible.

So I kind of used to say that Google was an example of a less microservices architecture. And then I talked to Randy, and he was like, "No, you're wrong." So fine. I'm wrong.

But one thing I will say is that Google has a countermeasure that allows you to make this work, which is they are very rigorous about continuous integration.

Anytime there's a change to any of the libraries in Google, there's a simple way to look at the properties you've impacted by the change in that library, and then it goes and builds those properties that are affected and runs an enormous number of tests against them: 10 million test suites per day.

And again, this is a couple of years old. More than 60 million individual test cases per day, more than 4,000 CI builds per day. It helps that Google has more servers than God.

And this is probably unattainable for mere mortals, but it shows you the level of effort they're willing to put in to making sure that anytime something breaks, you can detect it immediately.

And this is probably motivated by the fact they build everything off trunk. So anytime you change a module, it's not that some downstream component can choose to consume the latest version of your module when they want to. Oh, no, no. When we build a new version of some property, it's going to take trunk for all the upstream components and build off that.

So you need this level of coverage and rigor around the build process to make sure you're not going to break anything. And of course, this comes in very useful also when you're building monoliths as well: the ability to get rapid feedback. Have I broken anything downstream?

And then what goes with that in Google is, if I'm working on a team and I break someone downstream, they can revert my change from version control. That's a totally reasonable thing to do. And that level of collective code ownership, where if I break someone, they can just revert my change, that goes hand in hand with this because you can't allow people to just break downstream and go home, and then rock up the next day and be like, "Oh, I broke AdSense. Hmm." That doesn't really work.

So componentization is fundamentally what enables either of these strategies, or wherever you are on the spectrum between them. You still need to think about the basic architectural principles and how to enable testability and deployability in order to achieve our ultimate goal.

I want to turn a bit to the unreliable platform. There's another really great book I recommend by Mike Nygard called Release It!

And the key thing about this book is it tells you how to deal with the fact that your underlying platform is unreliable, and it gives you a bunch of architectural strategies to manage your ability to attain these architectural attributes when you have an unreliable platform.

Because guess what? If you develop an application that's not fundamentally scalable and secure and resilient, you can't pay the DevOps fairies any amount of money to rock up and sprinkle DevOps fairy dust on it, and scalable and reliable and security it. Doesn't work.

You have to build it in, and the developers have to care about it from the start, and they have to be working with the security people and the operations people to understand how production works in order to be able to do this.

And the best way to do that is to have them deploying to that environment as early as possible and running real production-like tests on it from early on, so they can understand whether their architecture is falling short while it's still cheap to change.

And in particular, orchestration. If you have to deploy all your stuff in an orchestrated way, where you have to deploy all these things in a particular order, that's a smell.

Any time someone says, "Well, my deployment tool must support orchestration," I'm like, "No, your architecture must support not orchestration." That's actually the answer.

So again, these are things we have to think very carefully about.

The problem is most of us are not there. Most of us have to move from A to B. The dirty secret is all the unicorns actually also move from A to B.

Randy Shoup has a great talk from Craft last year where he lists the three different iterations of the architecture of every major successful web company. And my little joke is that the unicorns are actually just horses with really good PR. And if you look inside these companies, things are often not as promised and are actually pretty ugly.

So Amazon's a great example. In 2001, Amazon had a monolithic architecture with a single DB, and they couldn't scale the DB anymore. They ran out of horizontal scale, and that motivated their re-architecture from a monolithic system to a service-oriented architecture.

And Jeff Bezos, a smart guy, amongst his other characteristics, leveraged this opportunity to achieve a bunch of other different goals, including building a platform.

And who's read Steve Yegge's platform rant?

Okay, a few of you. So if you haven't, it's mandatory reading for architects. You should all go and read Steve Yegge's platform rant, and he summarizes the memo that Bezos sent out to his entire technical staff when they embarked on this re-architecture.

"All teams, number one, will henceforth expose their data and functionality through service interfaces.

"Two, teams must communicate with each other through these interfaces.

"Three, there will be no other form of inter-process communication allowed. No direct linking. No direct reads of another team's data store. No shared memory model. No backdoors whatsoever. The only communication allowed is via service interface calls over the network.

"Number four, it doesn't matter what technology they use. HTTP, CORBA."

CORBA? Anyone using CORBA?

No, just checking.

"PubSub, custom protocols, doesn't matter. Bezos doesn't care.

"Five, all service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.

"Number six, anyone who doesn't do this will be fired."

I don't need to say anything.

He was very serious about this. He hired an ex-Army Ranger who'd been, I think, CTO of Walgreens or somewhere, Rick Dalzell, his name was, to go around the teams and find out if you were talking to some other team's database rather than their API. And if you were, he would shout at you a lot and try and have you fired.

And so people took this pretty seriously, which is actually really important, because if you're going to move to a service-oriented architecture, you better take the whole hardened interface thing seriously.

The other thing I want to bring attention to is this thing in point five about making the interfaces externalizable.

For me, the reason why Amazon Web Services was successful is because it was built from the ground up to make money and to be something that customers wanted to use.

I actually have Jesse Robbins on video telling me how he tried to shut down Amazon Web Services because they wanted to use Amazon servers to build AWS, and he was like, "No way. Get your hands off my servers," and then tried to have the whole thing shut down.

So they had to go and buy a data center, or buy racks in a data center in South Africa, where they were all based, to spin up the AWS thing. And it was an external-facing offering long before it was their internal platform for amazon.com. That only happened in 2010, 2011.

So that's actually really important. If you have to struggle and hustle and actually have customers love your stuff in order for it to be viable, that's very different from an internal IT kind of private cloud effort where you've got a monopoly, because guess what? Everyone has to use what IT builds.

And so you can make it as miserable to use as you like, because guess what? Everyone has to use it. And the result of that is that it's miserable to use.

And particularly where you have a private cloud where you still need to raise a ticket or send an email in order to actually get a resource, like, eh-eh, that's not a private cloud. That's some very expensive virtualization technology that has zero impact on the overall cycle times and outcomes of your development teams.

So the other rule they followed when they were making this transition, I love putting words into people's mouths, and I'm going to do this to Werner Vogels, who's CTO of Amazon. He says, "You build it, you run it." The teams that build the software must run it in production.

And this is not the only way to do it. You can have high-performing organizations with functional silos where the functional silos collaborate in the greater interests of customers in the organization. Doesn't always happen that way.

If it's not happening that way, this is one countermeasure that forces you to act in that way. So by having these co-located teams where all the functions were co-located, that's one way in which you can follow Conway's Law very explicitly, and this is pretty much what they did.

So that's a huge transition. It took them four years, 2001 to 2005. And then Netflix basically did the same thing in two years, just to be like...

Is Adrian here?

Go and ask Adrian about that. But he's like, "Yes, and we did it in two years."

I want to talk a bit about re-architecture. There's a pattern. We talked earlier on, if you were at the panel, about not doing big-bang architectures. There's a pattern for architecting incrementally called the Strangler application.

Who has seen Tomb Raider, the movie? Anyone? Okay. Important architectural lessons from Tomb Raider.

So this is a temple in Cambodia, in Angkor Wat, and there's a tree growing up here. And what happened is one day a bird came and pooped on the tree a little seed of a strangler fig, and the fig grew up and killed the tree, and all that was left was the strangler, and the tree inside was dead.

And this is how we get rid of horrible monoliths.

What we do is we say, when we build new functionality, we're not going to add it to the existing app. Instead, we're going to build new services around the app using solid engineering principles, componentization, good encapsulation, tight coupling.

Maybe we reorg the way we run those teams, make them cross-functional, potentially. But all new stuff gets built in that way.

We definitely don't go and rebuild all the old stuff in the new architecture, and especially not without actually going and doing user research and finding out how users use it.

Otherwise, what you do is you rebuild a system designed for people 10 years ago's business processes. And it's imperative, when you're rebuilding stuff, to go and look at the users, because they'll be going click, click, click. "Oh, and then I go to this other system. Click, click, click. And then I go back to this system. Click, click."

And you're like, "Why do you do that?"

And they're like, "Oh, because that's how it's designed, and it takes 10 minutes to do something that should take one minute."

And you're like, "Oh, that's really horrible. Why don't we actually fix that for you?"

And then you implement how the process should work, not how it works right now. Otherwise you're just perpetuating misery. So always go and do user research.

Build new stuff as it comes in, and then gradually over time, you strangle the old thing.

And the crucial thing is there is no end state for this. The idea that there's a perfect architecture for your system is false and a great source of misery in our industry.

The reality is that every stage of evolution of an organization requires a different architecture. An architecture that allows a startup to rapidly iterate through MVPs in search of a scalable business model is very different from the architecture that you require to take your scalable business model and scale it. And your architecture will necessarily change over time.

The idea of the to-be architecture diagram, which is three years out and will be invalidated when, after two years, the VP of engineering is fired and replaced by a new VP who creates a new three-year to-be diagram for the architecture, which will be invalidated after two years when that VP of engineering quits...

Let's just accept reality and accept that our organization and its architecture will always be evolving, and we should always be applying this pattern.

That's the main lesson I want you to take away from this: that your architecture will always be evolving, but that we should always follow the basic fundamental principles of componentization and making sure that our software is testable and it's deployable, whatever architecture we're using.

I want to end by announcing something I'm pretty excited about.

So you may have heard several times today Gene Kim and other people talking about the Puppet Labs DevOps Survey of Practice. I'm really excited to announce that we're trying to make that available for general use, so people who are facing the beginning of a DevOps journey and planning what they should address can get personalized benchmarks on how they're doing.

So we are actually putting together a survey that you can take personally to tell you how you're doing compared to the wider population.

If you're interested in this, follow the offer on the next page and sign up, and we'd love to get your feedback on that so we can build it out.

There's a whole bunch of stuff I'd be very happy to give you. I'm going to be around after this. I hope the rest of your conference is awesome.

Thank you very much for your time.