Sam Guckenheimer

I'm Sam Guckenheimer. I joined Microsoft about 12 years ago now, in 2003.

I've talked and written about our journey from, in those days, being a waterfallian organization that did these very long, multi-year release cycles, to becoming agile and doing small batches and quick increments and all that kind of stuff. And we got pretty good at being this provider of commercial off-the-shelf software to deliver to customers on-prem.

Now I work in the area that is now called Cloud Services, but that includes what you might know as our Team Foundation Server, which is still a product we deliver to customers on-prem, as well as its sibling, Visual Studio Online, which is more or less the same thing as a cloud service.

So this is the chapter about moving from being the company that delivered this thing for customers to install themselves on-prem, to being the software-as-a-service company that lived in the cloud, ran from the cloud, delivers from the cloud, at the same time that we deliver the software for on-prem.

Now, if you roll the clock back five years, we had all this great recognition. We were up there in the analyst magic quadrants. We got all this great recognition, blah, blah, blah, for our on-premise products. And we were pretty proud of how well we'd adopted agile practices, paid down technical debt, and so forth. We said, "Oh, and of course we can make this a hosted service. No problem."

And we managed to make TFS, Team Foundation Server, cloud-hosted in a matter of months. But making it a cloud service has been a many-year journey. And I'll give you an instance of that.

About two years ago, November 2013, we announced that this hosted service, Visual Studio Online, was no longer in preview. And let's take a look at what happened.

The morning of the announcement in New York, we pulled all the feature flags, said, "Come on in, everyone." We quickly crossed the million-user mark and fell over for about nine hours, came back up, fell over again, came back up, fell over again. And we're talking about, here we are, a 24-by-7-by-365 service that doesn't do downtime.

So we were pretty embarrassed, and there were a lot of learnings from that. This is a talk about those learnings.

There are seven habits and technical practices that we've had to develop to become the SaaS provider, software-as-a-service provider, for Visual Studio Online, that's now three times bigger, and is continuing to provide both the online service, which we host through our Azure public cloud, and continue to ship Team Foundation Server. I'm going to go through these, and then I'll do a little demo of how we're productizing them for you to use.

So three of these are really, if you will, agile on steroids, or the next generation of agile.

The first is team autonomy and enterprise alignment: how we work in teams.

This should be no surprise. We work in small teams, feature crews. They work in team rooms. The folks in this team room are working on the same feature area. If they have their daily standup, they stand up. They don't need to go off someplace. Behind the camera are some small focus rooms, which hold four or five for a private discussion. But if you want to discuss something about this particular item in the backlog, you can just swivel your chair and talk to your neighbor. And if a third person joins in, that's not a bad thing, because you're all working on the same stuff.

It's not a cordoned-off area of 100-person space, because that turns into a library. It's a space for 10, 12 people, something like that, because you're all working on the same stuff.

We happen to work in three-week sprints. Empirically, this just seems to work best. We're highly distributed geographically. Three weeks seems to be a really good point at which to sync. We're heading into sprint 90. It means the same thing here in Seattle, in Raleigh, in Hyderabad, India, in Cambridge, England. And we talk about, "Now, what are you going to do in 91? What are you expecting from me in 92?" and so forth. And everyone just has the same sprint calendar. There's no discussion about it. Everyone knows what it means.

The sprint starts on a Monday, wherever you are, ends on a Friday, wherever you are. On the cloud service, deployment will be on the following Monday at our peak time, which happens to be, you can think of as Monday, East Coast U.S., so that we get traffic from Europe and East and West Coast. Because if something's going to go wrong, it's going to go wrong when you're busy. So that's when we want to see it.

Now, we take these agile practices and we try to trim down the overhead from them. So you have this whole issue of how do you involve stakeholders, how do you scale, blah, blah, blah.

So the sprint starts with a communication out in an email, which in turn goes on our team site, our SharePoint site. The email is hyperlinked to the items in our Visual Studio Online instance, and it says, "Here's what we're planning to do in this sprint, and here's the sprint goal for this feature crew." And it ends with an update to that email with a video added. The ideal length of the video is three minutes.

As a customer, you now can do these things in order to achieve this thing. So imagine across 30 crews, you have 30 of these with three-minute mails, and that is a representation of what was accomplished in the sprints. And people comment on them quite vigorously.

So all that's really lightweight. All that works well with geo-distribution. It means you don't need to worry about how do you get the stakeholders in the room, how do you deal with time zones, any of that stuff.

The second habit and practice that we carry forward from agile is how we manage technical debt.

Now, technical debt we think of very broadly. This is not just about static analysis warnings on your code. This is about how you manage your infrastructure so you don't get configuration drift. This is about your database. This is about anything that will throw up unplanned work or surprises that will get in the way of what you're planning to do.

This is really important to us because we are delivering both the service, which has these three-weekly increments, and the definition of done on the sprint now is gets deployed live across multiple data centers and is subjected to live usage. And then we deliver the on-prem update as well, where quarterly we roll up those updates that we've delivered into the service and package an on-prem update and deliver that to customers who install in their data centers, of course under their control.

We update the service. We're managing that. The customers who are running in their data centers manage what they do on their own update schedules. Okay? So we need to think about how both of these work.

This means that we've had to significantly change how we think about our test strategy, our test automation, indeed how we think about our practices for test and code going together.

In the past, we had very elaborate labs that did lots of integration testing, and we relied heavily on what we called L3 tests. That is, making sure everything worked together, worked really well together, and so forth. And we had, in theory, a lot of unit testing and would measure coverage at unit tests and all that kind of stuff. But if you looked at the investment levels, there was a lot of work on testing when everything came together.

We shifted that. Shifted it so far that we don't even have in the HR system anymore a distinction between developers and testers. They're all engineers. They work in a multidisciplinary feature crew. They're just engineers.

And we try to test as close to the code as possible, and try to build in services that are as autonomous as possible, communicating through well-defined interfaces, but testable independently, where the tests can run anywhere and the tests move with the code.

So the testing has been shifted radically to support this idea of being able to be resilient across time and not to build up debt, not to just work in this one configuration. And if tests are not reliable, that is, if you don't know that this will definitively find a problem and that red means red, then the test is thrown out and redone. And then the infrastructure for running tests is managed as a shared part of the engineering system.

The third thing we carried forward from agile was the idea of focusing on the flow of customer value. The difference now is that we can measure it, because we're running a service. We have telemetry that lets us tell what customers are really doing.

But the measurement isn't so obvious. Here's an example: SLA. Everyone has a service level agreement, it seems. We have one that says if our performance falls below three nines, we give you your money back, blah, blah, blah. And we thought that was a pretty easy thing to measure.

Well, if you just average performance and say, "Okay, we need to average to get to an SLA," you miss a lot of things. So if we take this particular date in September 2013, if you treat the SLA as averages, you would see this dotted black line up here.

If you take a slightly more nuanced approach and look at command execution time, so how many commands are actually running slow on the server, you see this blue line, which says, hmm, we had about an hour of non-conformant performance where we were really out of SLA then.

If you take this black line here and say, "How many minutes were customers actually waiting for something to respond beyond how long they should have waited?" So in other words, for each customer, you make the assessment: did that user have to wait too long for a response appropriate to that particular action?

And the actions are very different. There's something very different from, say, clicking to refresh a page, or running a load test, or running a build, or what have you. So this is fairly nuanced. But for each of those, you have to have a sense of what a reasonable expectation of response is.

Then you get this four-hour deviation on SLA, and you say, "Hmm, we need a much more sophisticated way of calculating this customer by customer," which is what we use now. So we beat ourselves up on the hardest measurement, and we describe the math and blog it and say, "So here's how we will measure and hold ourselves accountable to you for that."

Not obvious. Important learning. The blog, interestingly, is called, and this says something about Brian Harry, my boss, "Are Customers a Bag of Sand?" arguing against using averages.

It's an instance of what we call embracing the red. We focus on the outliers. So for this 99.9 thing, we say, "Okay, folks are getting good performance. Great. We're happy with them. Folks who are getting okay performance, we're happy with them." But everyone who's at the 95th percentile mark or the 99th or the 99.9th, we need to look at them. And we will proactively go after why they are getting slow performance and drill into that.

And in some cases, that means we'll actually call them, contact them and say, "Are you seeing something? And can we diagnose this with you?"

Another example, which I showed Gene and he loved, came from one of our service reliability engineers, where he was noticing, I don't expect you to read all this email, but here's the gist of it: noticing that a particular VM was running hot and slowing down, and noticed that it was because we had a noisy neighbor, not related to us, not one of our services, that was soaking up resources.

And out of that, realized two actions were necessary. One, we needed telemetry to detect this, so we could move our own process off onto a new VM. And two, the other service needed telemetry to detect that it was soaking up resources and behaving badly. And the gist of this whole email was to say, "Hey, I was digging around and trying to figure out why this was happening. Here's what we need to do," as an example of following the red.

Now, we also talk to customers. So the telemetry's great. You learn a lot about what's happening in a quantitative way with performance, behavior, availability, and so forth. But you need to complement that. You need to get in touch and say, "How's it going? What are you doing? Are there things you like, things you don't like, things you wish we were doing, concerns you have?" and so forth.

And you need to talk to non-customers. "Hey, I saw your usage went down since last month. Is there some reason for that?" And so we buddy up engineers who volunteer with customers who are using the system and talk to them. And that's a regular piece of the cadence.

So those were three practices that you can think of as second-decade agile: the teaming, scheduling, managing technical debt, the flow of customer value.

Let me talk about four that are really new with DevOps.

The first is managing the backlog with learning, or managing a hypothesis-driven backlog. This is what Eric Ries calls build, measure, learn. The idea is that you don't have requirement-like things in your product backlog, be they written as user stories or whatever, but you have hypotheses that need to be tested with experiments and validated, substantiated, or diminished based on data and turned into validated learning.

Here's an example. We had a sign-up experience for our service, Visual Studio Online. It looked like this screen. You were asked to sign up to create your team project, which you need before you do anything, and to fill this stuff in, and then you were given a bunch of other information around it.

We were not at all happy with the conversion rate on this. So we said, "Okay, we're going to make this a lot simpler. We're going to have you sign up, have you create your team project, and then we'll tell you about your other stuff."

And because we have two front doors, one through the web and one through the IDE, we'll try it on each. And we'll split our traffic to expose the new variation to a portion of the traffic. So we basically had a two-by-two matrix: the old experience with the two doors, the new experience with the two doors, that we could run.

And so on some of our traffic, we did this, and we got some interesting results.

Let me start actually on the right-hand side of this. The interesting thing here is that if you look on the right, you'll see this big bounce in the IDE experience, where we got on that cohort a 7x improvement in the new experience, comparing the new cohort to the old one.

On the left, we got about a 50% lift. Now, 50% isn't bad normally, but it sure didn't look that good relative to what was happening in the IDE. So we dug a little bit and realized that someone else was conducting a simultaneous experiment, driving a lot more traffic into the web. So effectively, they were generating a bunch of unqualified leads into the top of the funnel.

And we said, "Wait a minute. We don't need unqualified leads. Turn them off." And then we got the green bars, and we found that we got a 3.5x bump from the new experience on the web.

So this is a story about the value of both multiple cohorts and keeping experiments clean so they don't contrast. And it's a story about our not having coordinated well when we were doing that. Fortunately, we found the error.

So that's working with hypotheses and getting validated learning out. We did that with a portion of traffic. Of course, it worked, so we rolled it out broadly.

Now, we could do that because we were gathering evidence in production, which is the fifth habit and, of course, something new with DevOps.

We collect everything through telemetry. I just showed you an example of some business metrics where we were logging activities. We collect traces proactively. So if, for example, we see that some machines are running hot, we may need to debug. If something goes wrong, we will proactively collect data that may be used for subsequent debugging.

KPIs about performance, about funnel, and so forth. We collect perf counters about everything going on on the machines and environments. Dependent services, this is a simple example of a network ping mesh. So if we find that things are going much slower in Australia, we might check and say, "Hey, what's happening on the base network?" And if the fiber's down in Singapore, well, that could add 300 milliseconds. So you collect at all levels, understand back up what the effects might be.

Now, all of that works, that evidence works, because we have a production-first mindset, or as Satya Nadella, our CEO, calls it, a live site culture.

Satya started talking about this when he was at Bing, then Cloud and Enterprise, and now spreads this across the company. The idea is that site status is always the first priority. The site needs to be up, in our case, 24 by 7 by 365.

We have a global response team. In our case, we have people in five time zones, so they don't have to work eight-hour shifts before they do handoffs. They're responsible for being a first line of defense and for conducting weekly live site reviews on every incident. So every incident gets a postmortem. Of course, blameless. Of course, five whys.

There's also a monthly service review, which goes into business issues as well as technical issues. There's real-time communication around every LSI, live site incident. And everything that comes out of an LSI at root cause becomes a product backlog item.

So in other words, yes, we will remediate to get the service up as quickly as possible, and we will fix everything at root cause. And the root cause remediation needs to happen within one or two sprints, depending on how difficult that is, and it needs to happen with automation.

So an LSI looks like this in our VSO instance, Visual Studio Online instance. You can see the data that was automatically collected through our telemetry. You can see all the timings of things. How long did it take to escalate? How long did it take to communicate, to remediate, and so forth? All of this is tracked. All of the follow-up actions are tracked, and so forth.

We use simple tools for all the pieces of this, like just communicating with customers. You don't want to have to think about what do you need to do to keep customers informed, so there's simple templates that go out to let people know, "Hey, here's the status of the service." It goes out on Twitter, goes out on our service blog. Folks can come back to us on Twitter.

And then when we're done, we do a full retrospective of what happened at root cause and blog that. And that creates a sort of public record of accountability: here's what happened, here's what action we took, here's what actions we are taking so that we don't let this happen again.

Now, let me give you an example of some proactive action. One of the problems we found in these weekly LSI reviews was that we were spending a lot of time triaging alerts.

We had this tier-one layer of folk who were looking at alerts and deciding who's the DRI, designated responsible individual, i.e., developer, in the feature crew whom I need to contact. And our internal SLA is: you've got five minutes to be on the bridge during work hours and 15 minutes outside of working hours. So if it's 2:00 in the morning, you're supposed to be on the bridge within 15 minutes. The developer, you build it, you run it.

And the alerting was this long stream of alerts, and this poor guy in service delivery was looking at this and trying to figure out which one really counts. So we said, "This is not working very well." And we created a health model that took into account the code structure and the org structure so that it could identify which of those alerts was indeed the root cause and say, "Ah, the problem is in here. This looks like an APM problem there in the middle, and that is the feature crew, and that's the DRI, designated responsible individual, whom you need to contact."

And in fact, we'll automate that with an autodialer, so we don't even need this tier-one triage. In the month we put that in place, we had a 40x improvement in alert efficiency, and it's been going up from there.

Okay. So we took an automation solution to this problem of sifting through the alerts. Pure live site work.

Now, security. We've had a security lifecycle for a long time, and we used to think, "Boy, security's really important. We need to build a great castle wall, protect the perimeter, keep the bad guys out," all that stuff.

Now we realize that's part of it, but we need to assume that the wall's been breached. If you look at the big attacks that have made the news, Sony was spear phishing. Target was an HVAC contractor who tunneled through from an open port to get to the point-of-sale systems.

They're already inside. Protecting the perimeter doesn't do anything when they're already there. So you assume that you're breached. And what damage can they do once you're breached and they're inside?

So we run war games on ourselves where we basically have hackers come in and do mock attacks using privileged knowledge, but knowledge that someone very clever could get that is not out of line with the kind of stuff you see in some of the more sophisticated attacks. Using heuristics like old code is vulnerable code, scanning drives for clear-text files, looking at test data for passwords, things like that.

And those people then go in and they try to hack in, and we have defenders, part of whose job as the SREs is actually to detect and defend. And we harden the system based off those war games, which we do on a regular basis, and those in turn become more live site remediation items.

There are well over 1,000 secrets in VSO that we have learned to protect this way. We are constantly rotating them. We're keeping them in a secret store. We're constantly auditing for any possible spurious routes, dual-homed servers, whatever, and playing through the scenarios. What happens if someone tailgates in a building, someone inside forgets to log out, whatever.

The seventh practice, which is also very much DevOps, is managing infrastructure as a flexible resource. This is really about using the public cloud. So that allows us to deploy any time. It allows us to take advantage of resources to, for example, have a canary stamp, to run Go Big load tests when we need to, and to use techniques like feature flags, so we can separate development from exposure of new work.

We use our own release management capability to control the release of our product.

The way feature flags... How many of you know feature flags? About a third of the room. Okay. So the way feature flags work is that before you start new work, you register a feature flag, and it starts down. Unlike the old days when you created a branch and you said, "I'm going to do my work over here, and then three months later, I'll come back, and we'll have all this merge debt."

The feature flag basically says, "No one gets to see my new work unless I raise the flag for them." And it can control to the level of identity and feature. And you can raise it at runtime based on individual user, ring of users, whatever.

So new work happens behind the feature flag. It enables you to experiment. When we talk about things being in private preview, it means that the folks who get to see that capability are those who've opted in to the private preview. And then public preview means we've lifted the flags for everyone. You can do dark launches of dependent features and do progressive experimentation or refinement.

Scale units you can think of as sort of data centers within data centers. We're now at eight data centers, and this allows canarying. So we have the chart I showed you at the very beginning. We were at one. Got us to about a million users, and so it happened. We're now at eight. That one was in Chicago. We put one in front of it that we call SU-0, which is our canary. We work on that. That gets the first deployment of every update. It's heavily instrumented. We make sure everything's working there. And then there are rings of deployment that go from there out to subsequent data centers here, and then Europe and Australia and so forth.

Okay. So those are the seven habits that we've learned as a SaaS provider: team autonomy with enterprise alignment, managing the flow of customer value, managing technical debt, grooming the backlog with learning, gathering evidence in production, having a production-first mindset, and managing infrastructure as a flexible resource.

The kinds of metrics that we track are those related to the live site health. How long does it take to detect? How long does it take to mitigate? What are the prevention items that will cure this at root cause? We age everything. So how long has it taken us to do this? What's happening on customer support and so forth.

Velocity is really about the engineering flow. How fast can we get stuff out there? How long does it take us to build? How long to get to self-test? How long to deploy? How long to implement the learnings?

Engineering is really about, at this particular point in time, these are the valves that we used to monitor the flow. Like the bug cap is a notion that you can only have so many bugs open before you have to stop any new work. Keeps technical debt out. For example, if bug cap is five, there are 10 engineers on your feature crew, you hit 50 bugs, you stop new work.

Everything's aged. Tests, passes, coverage, and so forth are all monitored. And then usage metrics are all about growing the business and watching what customers are doing and making them more delighted.

Okay, so let me flip to a demo.

And I'm going to just explain it for a minute. So this is an example of productizing the kind of thing we do in learning. I said we deploy with our own build and release management. We develop with our own VSO, with Git underneath.

So I'm going to show you developing with Visual Studio on a Windows machine, committing with Git. That'll trigger a continuous integration with our build agent running on a Windows VM in Azure. That will package a Docker image, a Docker container, which will go up to Docker Hub, which of course we don't control.

And it will also take the artifacts and it will make database updates, because the database needs to be considered as part of the CI/CD pipeline. And we'll use SQL Azure for that, which has capabilities to package database changes in a so-called DACPAC.

And then we'll use our release manager to push those and pull the container from Docker Hub and install those on a Linux instance. And it will do that three times: once for a dev environment, once for a QA environment, which will be similar but we'll add load testing, and once for production environment. And you'll get the idea.

And this is the URL if you want to follow along with me: peopletracker.us. Okay.

And for the first two, there'll be a waiting for approval.

And just since I'm going to switch, I'm going to remind you of this, if you weren't here at the beginning, of this URL where we have a survey, if you missed this, for this session: aka.ms/does15. You'll be entered to win a... And we have 10 of these Jolt backup power supplies, so we know whether to keep coming, and you'll get a text to pick it up at the booth.

Okay, so here we go.

Let's see.

Okay, so there's a Kanban board in Visual Studio Online. We don't need to go there and talk about Kanban. This is the application I was talking about, peopletracker.us. If you want to follow along, you can click on the homepage, and it'll tell you what technologies are in here. There are multiple stages. I'll use dev. And if you click on people, you'll see a bunch of people, mostly test identities, but you can add your own if you want to do that.

And you'll see first name, last name. You can add yourself. And you'll be on the list.

Now, our product owner said, "Wait a minute, you need at least a middle name because that's not going to work for most people." So, in the great tradition of cooking shows, we did some coding to add middle name. It's going to take a schema change, so you have database changes here to push as well as website changes.

And you see here in Visual Studio, our favorite IDE. If I click on this, I need a... Let me make that larger for you. Hang on.

You see here I need a commit message for the push, so I'll say, okay, added middle name. Kerr Jean.

And then I commit and push. And when I make that push from the IDE and go back to the browser with Visual Studio Online, I'll be able to see here in my build tab that there's a Docker CI build, which I'm using. And there's a new one that's being queued. It's starting to run. It's going through and showing me all the steps that it's doing. It's building the solution right now.

If I were to edit that definition, you can watch it running through here. I wouldn't, of course, normally edit it while it's running.

If I were to edit that definition or create a new one... Come on.

I have the build steps which are running these things. They're pushing the Docker images. They're publishing the build drops. If I add a build step, you'll see I can add build steps for Android, for Java build runners like Ant or Gradle or what have you. I can do Xcode or Xamarin for iOS. Everything's there.

If I go back... I don't want to save changes. If I go back, that should be done by now. And indeed, it is. And there's a little stamp on it saying that there's a chained release. I can look at the tests that have been run with it. We don't have many in the interest of demo time.

But from the build, there was that push to Docker, and there was also the creation of a release. So the build was continuous integration. And now we have the start of a release. You can see here that release 28 has begun.

And if I look at Docker Hub, Docker Hub had build 345 previously. If I refresh this, you'll see that it's got 346 now as its most current, which is what release is pulling down.

And we'll take a look at this, and that will be pushing this release into dev, which is going on right now, and then QA and production. And you can tell by the blue that it's still going on in dev.

So let's go back. Still in progress. When that finishes, which should take a minute, it should turn green, and then you'll see the change on the website and the change in the data.

So that's an example of our using here Windows, Linux, Docker all together as part of a release management chain.

Okay. While this is going on, let me take questions.

Q&A

Q: Are you leveraging Docker for VSO?

A: Are we using Docker for VSO? Not yet.

VSO primarily uses Azure PaaS. So a lot of the things that would be benefits of containerization in terms of the lightweight footprint of Docker are already provided by the PaaS environment in Azure.

We will be going to, I think, a Dockerized build as an out-of-the-box thing, or build and release, in the next quarters. But in our core service, we've already got the footprint benefits that we would get for Docker because we are already the multi-tenant service on PaaS.

I'm not sure why that's taking so long.

There you go. So the People Tracker website's updated now, and you see that the changes have been made, including data changes. And the release will now move on to the next environment stages.

Other questions? Comments? Go ahead.

Q: Is there a rollback capability or anything if you were to push something out there and discover that it's something really bad, and you want to roll back to a previous release?

A: Is there a rollback capability? So there are... It depends how you set up the release. This one is set up with staged environments, so that you would check dev, QA, and so forth.

The big question is what happens at the database layer, usually. So yes, you can go back and say, "I want to push build N minus one."

In our service, we have the problem that we are managing customers' data. So every time we do a schema update, which is frequent, we have updated the schema, which is entwined with customers' data, and we don't really have an ability to roll back that tier. It's behind an API, but we can't do that.

If you are managing the data, and it's yours, then you could do effectively a flip to pull back the data tier. But that requires effectively the application or the service to have that make sense. And the release management service has these actions available to it. But it's something that you would use judiciously.

Sure. Anyone else? Go ahead.

Q: Is there an expected date for when will the release be out of preview mode?

A: When will release be out of preview mode? So the release online will be out of preview mode probably within a month. I don't have a date when it will be delivered in TFS. We're using it now internally on our deployments. So that will be coming soon.

Good. So once again, there is this survey URL that I would love you to fill out, please.

I'm just about out of time. I will be at office hours as part of the metrics group, Wednesday at 1:00 on the ground floor. And I will be hanging around the Microsoft booth during the exhibition hours much of the time, so you'll be able to catch me, and I'll be happy to hang around and talk to you whenever.

Okay? So thank you very much for coming.