Cornelia Davis

Fair warning, this is a technical talk.

I know that we talk about DevOps being all about culture. I come from Pivotal, and I can tell you that we spend a lot of time working with our clients on the culture side as well. But it turns out that tech matters. Gene had asked me to talk about cloud native architectures because it does have a significant impact on the relative success of DevOps.

Without further ado, let's start with a quote. In London several months ago, when we had DevOps Enterprise Summit in London, Jez Humble, along with Nigel Kersten and Nicole Forsgren, stood up on stage and unveiled the State of DevOps Report in 2017. In that, one of the things that Jez said on stage in particular was, "Architectures matter."

I actually pulled this quote directly out of the State of DevOps Report, and it says here, I'll just read a couple of spots: "Loosely coupled architectures and teams are the strongest predictor of continuous delivery." So architecture matters with respect to continuous delivery. Down at the bottom, it talks about the benefits of those loosely coupled teams and services being higher throughput and higher quality and stability.

All of this DevOps stuff that we do, we're not doing it just for the fun of doing DevOps. It's to achieve these things that I've underlined here at the bottom: higher throughput, higher quality, and stability.

Now, another report that I pulled was from the 2017 State of the Cloud Report. I started with DevOps. Now I'm looking at the State of the Cloud Report. The punchline on this slide is 75% of enterprises are moving at least a significant portion, or some portion, of their workloads into the cloud.

When we talk about cloud, and we talk about DevOps, and the goals of DevOps in terms of throughput and reliability and stability, what we're doing now is saying, well, cloud native is about loosely coupled running in the cloud. That's kind of the definition, is we put those two things together.

Now, I like to point out that cloud is where you compute, and cloud native is how you compute. Today's talk is on cloud native architectures, so it really doesn't matter whether the cloud that we're talking about is up on some public provider or it's an internal private cloud that you have. I'm really talking about the how. I'm really talking about the architecture.

With that grounding said, let me introduce myself. My name is Cornelia Davis. I work for Pivotal. I'm the Senior Director of Technology there. I work in our product organization. For the last five or more years, I've had the tremendous pleasure of working with our very large clients. Our clients are very large enterprises, and I've been helping them innovate their way into a more efficient, continuous delivery-based approach.

Being a propeller head myself, I tend to work on the architectural side, although I spent quite a bit of time working on the cultural and the process side as well. I come from a background of development, so I wasn't in ops. But I'll tell you, after really only about a month of talking to the customers about this platform as a service five years ago, which was said to be all about the developer, about a month later, I realized that it wasn't all about dev. It was also, in fact, maybe even more so, about the long tail: the operations, keeping these things running in production for a long time.

I actually spent four or five weeks working with Tony Hansmann, who was running our Cloud Ops team. He's at the back of the room there. So I actually did do a bit of a tour of duty in ops and, quite frankly, worked with a lot of our customers on ops.

I've been working in web architectures for more than 10 years, probably closer to 15 years. Cloud native in particular, for about the last five years. Specifically, I've been working with the Cloud Foundry product and helping our customers embrace that.

As I said, I work for Pivotal. More recently, shameless plug here, I am writing a book on this subject. I'm writing a book with Manning Publications. It's already available in early access, so the first four or so chapters are available. I'm expecting chapter five to push in within about two or three weeks. I'm working on that right now. You're going to be seeing a lot of the content in this talk that's in that book.

Let me talk a little bit about Amazon Web Services. Here we have this technology company that hosts applications for thousands of major corporations, major corporations that you're all very familiar with.

But you know what? As it turns out, Amazon sometimes breaks, just like anything else sometimes breaks. When that breaks, and I studied a particular outage as a part of the research for my book. I studied an outage from September of 2015.

In September 2015, Amazon Web Services suffered an outage. That outage affected companies, again, these large corporations that are running their properties on there. It affected things like Airbnb and Nest and IMDb and many other companies, including Netflix. Netflix is kind of the hero of our story. We'll talk about that more in just a moment.

The outage lasted for more than five hours. But you know what? Here's what Netflix had to say. They said, quote, "Yeah, we experienced a brief availability blip."

Now, is it their definition of brief that's in question here? Do they consider five hours to be not that big of a deal? No. In fact, while the Amazon outage lasted for five or more hours, the Netflix outage lasted for minutes. Just a few minutes. It turns out that they did so because of, of course, people and process, but also the architecture of their applications.

Netflix are arguably the poster child of modern cloud native application architectures.

Let's talk about autonomy. I want to tie this back to DevOps. What we talk about in DevOps, of course, DevOps doesn't mean that developers are doing operations or vice versa. That would be a very strict way of trying to interpret those two words, putting them together. It's really about eliminating the friction between dev and ops.

Why? Because we want to have more efficient means of delivering value to our business and to our customers. It's about that continuous delivery.

In order to be able to do that and be agile, yet still resilient and have a very robust system, we have to have autonomy. How many people here have been around the industry as long as I have, where you remember those Gantt charts that filled up an entire wall where you had to coordinate many parties to be able to deliver something into production? I think most of us have lived in that world.

The only way to get better at releasing software in production and then keeping it running in production while you're constantly upgrading that and bringing new value to the customer is to have autonomy.

Now, we could spend literally several hours talking about the different axes along which you have autonomy. But I think we can all agree that if we start to break down our application architectures into components, that allows us autonomy around...

This will help you? Okay, it's going to help me. Should I use this? Oh, wow. That's much better. Are you going to turn off the lav then? Yes. Okay, excellent. I think I can manage. I don't have to type, so that's good.

Sorry, let me get back to my train of thought here.

Speaking of autonomy, if we break up our monolithic architecture into components, I think we all understand some of the things that are on the screen here. It allows us to scale our application more efficiently, which is going to allow us to deliver a much better operational experience to our operations team and, in turn, give a much better experience to our customers.

It allows us, from a developer productivity perspective, to scale teams in a very new and interesting way that allows people to be efficient very quickly. It allows us to have independent development and deployment cycles as well as experimentation cycles. Then very concretely, it allows us to have that resilience.

Netflix having the resilience of a few minutes of downtime while the underlying infrastructure had five hours of downtime, that is part of what you get from cloud native application architectures as well.

Now, this is essentially the agenda for the rest of the talk. You can see that there's a lot of content. I'm going to go through it very quickly. I am a teacher at heart, and so there's a part of me that wants to take each one of these and teach them to you very carefully. But even if we had 45 minutes, we wouldn't have enough time to teach each one of these patterns very carefully. So what I want to do instead is really just introduce these patterns to you and relate them back to the goals of higher throughput, more continuous delivery, more resilience, and so on.

I won't go through the list now. We'll go as we get along. There are two categories, if you look at that. I'm going to talk about the application tier, which we'll see in just a moment is stateless. Then I'm also going to talk about cloud native data. What does that mean?

Let's jump into the cloud native app. I am going to make the assumption here that we're breaking things up into pieces. Yes, we sometimes call those things microservices. We have our larger application. Our larger digital system is made up of a whole bunch of these smaller application components. So we've broken it up.

We've broken up those application components, and now what we're going to do, instead of trying to make any one of these components bigger, what we call scaling vertically, we're going to scale horizontally. That's a fundamental architectural pattern, and it's something that your developers need to understand.

They need to understand that they're not going to get higher throughput by increasing the size of the memory or so on. They have to understand that the solutions that they're building are going to be scaled by creating additional instances. So they have to think about breaking up the problem into those individual instances.

Now, an important part is, in this first picture, you can see that every one of the apps is scaled to the same level. They all have four instances. But really important is that you're going to have different amounts. Some are going to be scaled very wide. Some are going to be just a couple of instances. Who knows, maybe you might even have a singleton here and there.

As soon as you do that, as soon as you have multiple instances, it introduces something else that you maybe had before. Now, you've probably been scaling your applications at some level. You've been scaling out, and you've all had load balancers that have been in play.

Here's where things might be different, though. It's what's highlighted there in yellow. It's that that load balancing, that routing capability, has to be dynamic.

You cannot have somebody who says, "Hey, I see I need more capacity. I'm going to scale this out to double the instances, and now I've got to file a ticket to get somebody to put an entry in an F5 load balancer somewhere." That does not give you the autonomy that you need, and it's not going to give you the agility that we're aiming for with our DevOps practices.

So that has to be dynamic and be fully automated. A dynamic load balancing capacity.

Now, the other thing that you have to keep in mind is you'll notice there that the first router that I drew was on the outside of the application instance. Notice on the second router, what I've done is actually drawn that routing capability can, in fact, be embedded within a client of the application itself.

Client-side load balancing is something that we've been leveraging in cloud-native architectures to get just that extra little bit of performance. Because one of the complaints we sometimes hear is, "Well, if you break things up into components and I always have to go through a router, doesn't that introduce some additional latency?"

Sure, it does. One of the techniques that we use to reduce that, and by the way, in a lot of cases, that added latency is no problem. It's a great trade-off for the architectural flexibility to have that router in place. But in some cases, you might need to eke out just a little bit more performance, and so you can start to do things like client-side load balancing. I'll talk about that more in just a little bit.

Okay, so we know we're scaling out. The next thing that we need to understand is that those application instances need to be stateless.

Now, you can see here that I've drawn a router, and right now I have a single application instance. My user has gone to that application instance through the router and has logged in. Now they're accessing that app, and they have their valid user token.

Now, I have some event where I notice that my application instances are being overloaded, and so I need to add an additional application instance. As soon as I do that, and you probably can't see it in the tiny little print in the lower right-hand corner, if I'm storing that state locally, then that valid token that allows me to continue accessing that app only exists on the upper one.

So the next time that I go through this router and my traffic gets routed to the lower instance, my user gets an unauthorized. They scratch their head and they hit refresh. Oh, and now it's working again because the traffic went to the first instance. They go, "Oh, okay, cool. It was just some weird blip." Then they hit submit, and it goes to the second instance, and they're given an unauthorized.

This is a pretty extreme example. But the whole notion is that you don't store that state locally. By the way, one of the things that I want to say is make them completely stateless and do not use sticky sessions.

Sticky sessions, of course, is what's going to allow you to say, "Hey, if it's the same session ID, if it's the same user, keep sending them to the same instance." But of course, if you have a network partition and that instance goes away for even just a split second, then they get that unauthorized and that bad user experience again. So sticky sessions really don't belong in cloud-native architectures. It's a stopgap that we're using for these stateful applications that we've had in the past.

Now, the solution to that, of course, is to have some type of an external state store that all of those instances can leverage. The valid tokens are now stored in the state store, and I can go to any one of my instances on the application.

Now, I want to pause here for a moment. This is a super simple pattern that is extraordinarily powerful. Because in the olden days, 10 years ago, or maybe even five years ago, or maybe even a year ago, or maybe even now for some of your applications, what I did was I did a lot of capacity planning ahead of time. So I had to plan for my capacity, and then inevitably, I over-planned and I over-resourced.

But then maybe eventually I ran out of capacity on that over-provisioned resource, and now what do I do? I have to go through this very painful process to scale things out.

Capacity planning in a DevOps world shifts from a planning to a capacity management problem. This architectural pattern is incredibly crucial for enabling this shift from a planning to a respond type of mindset. Those are the types of patterns that we want to put in place in our DevOps world: stop planning everything ahead of time. Instead, set up my system so that I can react appropriately.

All right. The next pattern that I want to talk about is application configuration. Application configuration is something that also has to change as we move into this world that is much more agile and much more of a DevOps-style world.

It used to be, when we only deployed our applications every six months or eight months or 10 months or 12 months, that my properties, my application configuration, could be done in a property file. Sure, that property file was part of my checked-in code. Because I was only releasing every once in a while, I could treat that property file a little bit like a pet. Then my deployment was a little bit bespoke and a little bit carefully cared for.

What's happening now is you'll see, as we go through and look at some more of the other patterns, that because my application instances, we already saw one, are coming and going. I showed scaling out, but sometimes we scale in as well. Sometimes I'm going to have many different instances of those applications, and because of the changing topology, my application configuration may need to change a bit.

Even just knowing the other parts of the topology of the application might be something that I want to configure in, and we'll see some examples of those configurations.

Here's a pattern where, rather than having the actual property values, the application configuration, in a property file that is part of the deployable artifact, we want to externalize that.

Now, application configuration, I like to talk about it in two senses. There's really two categories of configuration data. There's system configuration data, which is the data that's around the application itself. It's the contextual information. What host am I running on? What IP address am I running on? What port am I listening on?

That's all system configuration information. That's not something that a human being is configuring into the application. That has to be done as a part of the platform.

Then, of course, there's the application configuration itself, which might be even something as simple as color palettes or some type of skinning of an application, or it might even be credentials that you're using for various components to be able to communicate with one another. That one's kind of on the hairy edge between system and application configuration.

The key is that the best practice that I recommend is that property files now, instead of carrying actual values, actually act as the specification of the properties that need to be injected into the application. Now things make their way into the property file either from the system environment, through environment variables, or through some type of an application configuration service.

That application configuration service, by the way, is designed to handle multiple instances of the application. You might have thousands of instances of the application, and that application configuration service will handle refreshing all of those on your own.

And of course, treat it just like source code.

Okay. Now, the next thing that I want to talk about is application life cycle. So let's take application configuration, what we just talked about. When I used to deploy my application every six months, I could do that configuration. I did the configuration at the time that I did the deployment. It was a very carefully controlled process. Application configuration was applied at deployment time.

Well, now in this cloud-native world, where my capacity is scaling in and out, my application instances are going away because they might be being upgraded or there might have been some kind of a failure, when do I do things like application configuration? Are there other events that I need to know about?

For example, if my application instance needs to know about the other application instances that are part of its cluster, then if one of those other application instances changes its IP address, how do I know about that?

These types of things are really part of the application life cycle. Your developers need to think about the application life cycle being far more changeable and more frequently changed, and they need to address those concerns and think about being able to do several things: absorb changes from the environment, not only at deployment time, but at other times. They also need to be responsible for projecting their own state out, because somebody else in the collection probably cares about that state.

You start to have responsibility in both directions.

Now, part of that application life cycle is, I just said that the applications are responsible for projecting out their state, and that's really important. But what happens if the application instance goes away before it has a chance to say, "I'm about to die"? It doesn't know that.

So there's another notion in application life cycle about health endpoints. All of your applications need to be able to have some type of a health endpoint that the system that they're running in can ping on a regular basis to see if that application instance is still running. So that if it's gone away, if it's no longer available, the system can correct itself.

Now, remember, Netflix is our hero of the story here. Those are the types of patterns that they put in place. When that AWS region went down, they automatically recognized. It wasn't that the region said, "Hey, I'm about to go down." The region went away, and Netflix had its health checks that were constantly pinging the applications, and within seconds, knew that the application instances, not just one or two, but all of the instances in an entire region were down.

So they knew immediately, "Ah, a region went down," and they had practiced, and they were able to stand up their capacity in another region. So, absolutely critical component. That was because they handled application life cycle events the right way.

Now, this is actually a picture. I spoke ahead of my slides. This is where I'm talking about broadcasting out those changes and also about health endpoints.

By the way, I see some of you taking photos, and that's great, and I love that. I am going to make these slides available on SlideShare. I'll post them later this afternoon. Also, the conference will have the slides. I just sent them to Jess this morning.

All right. Another thing that's really important is the notion of parallel deploys, versioned services and parallel deploys.

When we think about downtime and the causes of downtime over the last X number of years, I'm going to look at my clock here. I don't have the statistic off the top of my head, but a great number, more than 50% of the downtime that you experience in a data center, is due to some change that's being applied, an upgrade that's being applied, for example.

Which is why we tend to do upgrades between midnight and 4:00 a.m. on Saturdays, right? It's because they're risky and because we know that doing a deployment is risky.

Well, one of the things that makes a deployment risky in the olden days is that we replaced deployments. We took one deployment, and we completely replaced it with the next deployment. There's a great deal of risk associated with that because even though you have done a full test of this in a staging environment that is an exact replica of production, that's a fallacy.

There is no such thing as an exact replica of production, right? Which is why we're still doing our rollouts between 12:00 and 4:00 a.m.

So when we do that deployment, and we do that replacement, and something goes wrong, I'm now in a state where nothing's running. One of the critical patterns here is that you do parallel deployments, that you do canary-style or blue/green deployments where you have two versions, the new version and the old version of an application running at the same time.

You start to bring a little bit of traffic over to the new version while the old version is still handling most of the traffic. Of course, you have to have the practices in place that allow you to incrementally move more traffic over to version two, and if something goes wrong, very quickly pull all the traffic back to version one.

So versioned services is something that your application developers need to understand. They need to understand that their version two of the application doesn't rule the whole world, that there's a version one that is still living in that environment, and that they need to be aware of the fact that there's going to be two different versions working at the same time.

That's what my snazzy little animation shows.

Okay. Almost coming to the end of the application side, and I do want to spend a few minutes talking on the data side, is this notion of service discovery.

If you remember, a couple of slides ago, I didn't call it out too much, but I said that when environment changes, I was talking about the application life cycle, when my IP address changes, I need to broadcast that up so that there's some magic that happens so that all the interested parties can find out.

Well, there's a couple of different ways of thinking about that service discovery. Service discovery, by the way, is not the old UDDI stuff. This is, "Hey, I've got instances of a service that I know about, and where can I find those instances?" It's that level of service discovery.

There's a couple of flavors of magic that I'm going to talk about. One of them is your router. Remember I talked about that dynamic router earlier where I said, "Hey, I have a new IP address. I have to project that up so that the router can be updated without a ticket." Having the router be dynamically updated and having all of the consumers of that service go through the router is one way of doing service discovery.

Another way of doing service discovery is to have something that we call a service discovery server, where it captures those changes in the IP addresses, and then the clients, this is where you're doing client-side load balancing, they continue to use the old IP until they have a problem.

When they have a problem, they say, "Hmm, maybe the IP address changed. Let me go see if it has." Sure enough, the service discovery server says, "Yep, an IP address changed." They're a client-side load balancer and continue on.

All right. The last pattern that I want to talk about on the application side is, sometimes we jump right ahead, and we talk about circuit breakers. Who's heard of circuit breakers in the microservices world? Right. Great.

Why do we care about circuit breakers? To prevent one application from destroying the entire application. To keep one application from destroying the other application. Absolutely true.

But at the core of that, really what we care about, and the reason for that is an even more first principle, and that's the first principle of retries.

In a cloud-native architecture, and I think it shows there, the number one thing on the list of the fallacies of distributed computing is: the network is reliable.

It's not.

What happens is that in this cloud-native world where we've got lots of different components, if every one of those components expected the network to be reliable and expected the service that they're calling to be available when they needed it, then our systems would never work.

I've actually done the math, and you can go from five nines down to one nine with a tiny amount of network outage. So what we do in cloud native architectures is we employ retries.

This is just like when I'm accessing something in my browser and the page doesn't render, I hit stop and I hit retry. Unless, of course, I'm doing my purchase, then I don't. Then I wait.

But retries are an important part of cloud native architectures.

Well, what happens is that when you have a network partition, and ironically, that AWS outage that I talked about in September of 2015, that outage happened because of a retry storm. I won't go into the details, but what happened was they had a very short network partition, which caused a whole bunch of retries to get queued up, so that when the network came back up, the systems that they were retrying were overwhelmed with requests, and they couldn't handle that capacity anymore.

They didn't have a circuit breaker in place. So the circuit breaker is in place to keep these retry storms from taking down your system. The retry storm happens, but you keep it from taking down your system. Again, this is all about retries. If I don't hear back, I come back, and so that's what we use circuit breakers for.

Okay. I'm going to jump past this one because I only have three seconds left, and I just want to mention a couple of things about cloud native data.

Clearly, I know we started just a couple minutes late, so just bear with me, please. If you need to leave, by all means, I won't be offended. But I'm going to go maybe two or three minutes longer to just give you the overview of cloud native data.

In fact, on cloud native data, I believe my colleague Elizabeth is speaking on cloud native data. Not exactly? Not exactly. Okay, sorry. Different topic. Anyway. Elizabeth will be speaking in the keynotes tomorrow morning.

Let me carry on with cloud native data. If we go back a picture, I'm sorry it's a little bit dark here, but what we've got here is a whole bunch of independent microservices. So we're loosely coupled, right?

But check this out. All of those loosely coupled microservices are all tied to the single monolithic database. That is, by the way, a perfectly great way to start. Perfectly okay, because you need to start somewhere, and starting to break things up at the application tier is good. But this is definitely not cloud native data.

Let me just give you a high-level overview of a couple of the patterns that I want to talk about. Number one is what I'm calling data APIs. Instead of having dozens of microservices all tying to the database independently, introduce a services tier, introduce a data API tier that all of your microservices are going to communicate through to get to the database.

Now, it's still a shared database. There's still two little dark bubbles here that are tied to the database, which is really kind of an anti-pattern for cloud native data. But it's better. At least we have fewer of those. So the data APIs is one of them.

Now, I'll warn you that if you are having those data APIs just to act as a proxy, which is what we did back in the SOA SOAP days, we generated a whole bunch of SOAP-based web services that just acted as proxies. It didn't do us a whole lot of good.

What you want to do is you want to start making those very intelligent and allow them to have their own materialized view of the data that they're projecting out through the API. You can use some type of caching technology to do that.

Speaking of caching, I won't go through the details here, but this is a slide from Adrian Cockcroft of originally Netflix. Caching is pervasive through the entire cloud native architecture that they have. Caching plays a far more fundamental role than just performance in cloud native architectures.

If you remember, the versioned services that we talked about before applies equally well when we're talking about data APIs. The really great thing is when you do that, when you have multiple versions, it gives you an option to migrate your database schema in a much more flexible way. It allows you to loosely couple the data consumption from schema migrations, and there's a whole bunch of different ways that you can do that, and I won't talk through those.

Finally, the last two things that I want to talk about is, here I'm still showing a single database with a bunch of microservices. You might have heard that cloud native data, what we talk about, is every microservice gets its own database.

Now, this sounds great, and this allows us to do polyglot and those types of things, but it also establishes a challenge. If I've got now an application that calls two other microservices and they each have their own databases, I'm forced to do a client-side join here.

Now I have a request coming in. I have to do a request to another microservice, get back the response, do a request to another microservice, get back that response. So it starts to feel a little tightly coupled. But that is one of the things that you can do. Instead of doing joins in that monolithic database, you have to do them on the client side.

Now, here's the most interesting thing that I want to talk about with cloud native data. It's part of the early release. I just released a new chapter in the book, and the title of the chapter is "It's Not Just Request Response."

What I'm talking about in that chapter is if you turn the request-response model on its head, and you move toward an event-driven style, it changes a lot of things. It introduces a great deal of autonomy.

You'll notice that when I went from request-response to event-driven, I added another data store. So the new front-end API got its own data store, and now what we do is, when I have changes happening in one data store or another, I actually propagate them very proactively. It's an event-driven system.

So you'll see that another event's going to happen in the lower right, and I'm going to propagate that immediately to the database so that when the request comes from the consumer, I don't have to navigate the entire hierarchy of microservices. That's been done preemptively through the event-driven mechanism. That is a really key pattern in event-driven systems.

Now, you might have a challenge then of keeping things in sync. For example, if something happens in one database up here, I need to propagate that down here, and if something happens down here, I need to propagate it up here. You might be thinking, "Ah, well, I got you because that's now the complex network that you're expecting me to do with event-driven systems."

The answer is no. What we're doing in this particular case is, instead of having that hairball that I just showed in the previous diagram, we are moving into an event-driven model where we have a unified log in the middle. The unified log is the place that we collect all events, and now each one of the services that are tied to that unified log is responsible for materializing their own view from a sequence of events.

What is the open source name for that event log? I think I heard it. Kafka. Kafka is the thing that is most often associated with that unified log. By the way, that unified log then becomes the source of truth.

Of course, I'm realizing now, having gone six minutes or seven minutes over, that I had way too much content for 30 minutes. But I hope it was helpful. I appreciate you all staying for a while. It's lunchtime, so I'll stick around if there's any questions. There's some references in here as well, and I thank you so much for your attention.