Chris Lundy

So, what is Columbia Sportswear about?

Well, a lot of people don't know about Columbia, but I want to draw your attention to the bottom line there. We connect active people with their passions.

But we're more than that. We are a global vision of innovative brands with value design for consumer engagement. What does that mean? It means we're building the next generation digital platform that will bring Columbia Sportswear gear, tools, APIs, services, and data to your doorstep, whether you're a company or a consumer.

We are excited to be able to show you how we've taken the DevOps approach that so many of you have pioneered, have spent hours, days, months working on, and how we've tied that together to make the experience of a developer, as well as an organization, one that is successful. And we hope that you can take this away and go do the same thing in your organization.

So Columbia Sportswear is an innovative company, and we're looking forward to charting the course in what we're doing globally.

One of the things, Jessica, I'm happy that we're here. We've partnered heavily with Microsoft to help not only utilize their tools, but also to utilize their dev skills, as well as our skills, bring those together to help solve some of the things that Microsoft is trying to do, but also help us be a better company.

The one thing that we want you to note before you start on your adventure, or before you start bringing things together, focus on the value that it brings. Right?

So, Jessica, one of the core tenets, the architecture objectives that we have: day one developer productivity. And we're going to show you that in just a minute.

The question comes down to, can you afford to have your valuable engineers waiting two to three days, or two to three weeks, to get their identity, to get their system, and to be productive in code? That's a measure. We'll show you how to make it so that it can be done within 15 seconds.

Jessica DeVita

That's a pretty bold statement.

Chris Lundy

Yes.

Jessica DeVita

Are you gambling here? Are we betting?

Chris Lundy

I'm not betting, but I'm going to show what we have.

Jessica DeVita

All right. Okay. Awesome.

Chris Lundy

And I hope everybody sees it.

The next thing that we want to talk about is another principle, and that principle is zero downtime deployment. Columbia Sportswear is a global organization. That means we have applications that reach all around the globe.

So when we design in the cloud and we design with DevOps, as well as with continuous integration, we have to design with the fact that we cannot have any one of our regions down at any given time. So utilizing the Azure platform and PaaS, platform as a service, we're able to do that. To be able to deploy new code within a commit, through a build pipeline, to delivery to production, all within the same process. We're going to show how that works.

The question and the measure for you guys--

Jessica DeVita

Folks.

Chris Lundy

Or folks. Sorry about that.

Jessica DeVita

Thanks.

Chris Lundy

That's just a thing. Thank you for keeping me--

Jessica DeVita

Thank you.

Chris Lundy

Yeah, no problem.

Jessica DeVita

Yeah. There's so many. We want to be so inclusive in technology, right?

Chris Lundy

I know. Thank you.

Jessica DeVita

Let's start using words that reflect our diversity. Thank you.

Chris Lundy

Thank you. All right.

Jessica DeVita

I appreciate it. It's important.

Chris Lundy

I know. I appreciate her, too, so.

This is DevOps, right? We're talking about how do we make--

Jessica DeVita

This is about adapting.

Chris Lundy

It's all part of it. Yeah. That's it. Absolutely.

Okay. So, how much does it cost you to have a maintenance window? For instance, we have systems within Columbia Sportswear that have a 10- to 12-hour maintenance window every week.

What do we do with that? Well, that's down productivity. That's loss. It may be down during the midnight hours here in the U.S., but somebody in Malaysia or Singapore needs that system. So how do you manage that? It's a good measure.

Another one: managing DevOps through the database layers and actually injecting data into assets that you've deployed into Azure. We're going to talk about that.

So the measure is, how long does it take you to stand up a staging instance, make it productive, utilize it, test it, throw it away, and repeat that cycle? How long does that take? We have one environment right now, it takes weeks to do that. With this new approach that we're building and we're designing, we're going to show you, we can do that in a matter of minutes.

Another thing is to be able to reach out to your senior leadership, and that senior leadership needs to be able to see what's happening. So through VSTS, as well as our build pipeline, there are dashboards that we have deployed out in Power BI that you're able to utilize. We have Tableau, we have Cognos, we have a number of tools that help us with reporting, but Power BI really does reach into some of these native tools within the Microsoft stack.

And then finally, we wanted to have short, predictable sprints with actionable insights. So we want a product owner to come to us and say, "Hey, I want to revision this product service. How long will that take you?" And our answer could be, "Well, let's design it right now." And sit down with them, design it, put it into code, and deploy it right within that time window.

We've been able to do that utilizing open source tools, utilizing our pipeline, and it's pretty amazing to see the impact that it has on the business.

So one of the first things that we really want to talk about is design your approach. Enterprise architecture, system engineering, the development group all got together, sat down, and said, "Okay, what do we want this thing to do? What are some of the key major milestones that we want to have within the build-out of our DevOps pipelines, as well as our continuous integration and build cycles?"

And so we just laid it out. We said we want people to be able to clone the repo within a matter of minutes. We want them to commit and it go all the way to production. We want security and code quality checks. We want compliance checks, because we are a governed industry.

Jessica DeVita

And we've been given a really shining example from back in 2009 when John Allspaw and Etsy were doing these deploys every day. So you've got some aspirational goals, and I'm excited to see where you are with that today.

Chris Lundy

We do. Yeah, absolutely.

The other thing is change management. You still need it, but the difference is, it needs to change. Change management needs to change. Right? We heard that yesterday. We're going to continually talk about that.

You notice here that, I realize this is kind of difficult to read, but what we're saying is, as long as the checks are completed, why not push it to production? Why not let it go? Put other checks in to notify your product owner so they can push the proverbial button to say, "Yeah, go ahead. Release it." Because all of the checks are done, and it's a safe deployment.

The other thing we're going to show you is how we've crafted our DevOps pipeline, and really our development environment. What you see here are six key characteristics that every developer will have going into their coding process.

So this is it: don't be afraid. Try it, refine it--

Jessica DeVita

Yeah.

Chris Lundy

Measure it.

Jessica DeVita

If you were in the Disney talk this morning, we heard Jason talk about this need for collaboration, curiosity, and courage, and that really manifests in this. And I love that you have this sort of motto for your process. Thanks for sharing that.

Chris Lundy

You're welcome.

Okay, so enough of that. See the bit.ly link there? Go ahead and try to hit that link, and what you'll find is there's nothing there. But keep an eye on it. We're going to do some demos.

So the other thing I'm going to show you is, you see the Ryu command? So, you see this little guy right here? How many are familiar with Ryu from Street Fighter? Okay, yeah. You guys must've used E. Honda or something. Right? That's okay.

So Ryu is our chatbot. We use Hubot, and we're using that through Slack. What we've done is we've been able to take the identity process, and with that one command, get them onboarded not only to Okta, but to Azure AD, the on-premise AD, as well as Slack and Visual Studio Team Services.

Okay, so we've issued that command. We have a brand-new developer starting with us. Welcome, Scott.

Jessica DeVita

Hey, Scott. How are you doing?

Scott

I'm good. Thank you.

Jessica DeVita

Welcome to the team. Good. I'm glad you have your hoodie. That's representing right there.

Scott

Absolutely.

Jessica DeVita

And the headphones. You're all set.

Chris Lundy

Yep. And Scott could be here with us, and we're onboarding him. Think about your onboarding processes. But Scott could also be in another time zone that's completely opposite of yours, and they can communicate over Slack. It's important to think about the coordination costs and how you'll connect your developers together.

One of the things that we're doing at Columbia is hiring developers, hiring good engineers.

Jessica DeVita

We're hiring.

Chris Lundy

Hiring. So what that means is that we want their experience to be simple.

Jessica DeVita

So are we.

Chris Lundy

We are all. Exactly. Nice.

So, we want their experience to be really seamless, but we're also building a learning environment, as Scott mentioned yesterday.

So, Scott, you've received your Slack and your Okta notification here. Literally, that command, that Ryu command that I showed you, took 15 seconds to run. Okay? So go ahead and get connected through Slack there and Okta. Let's make sure you can connect into our environment.

Jessica DeVita

There was a time when it took weeks to get just even authentication, identity, linked into your Active Directory, et cetera.

Chris Lundy

Yes.

Jessica DeVita

That took quite some time, and now we're down to what?

Chris Lundy

Fifteen seconds. Not just the company that I work with now, Columbia Sportswear. Previous companies I worked at, two to three weeks was the norm to get the environment set up, to get their identity set up, all of that. But with this command, and because we've tooled it...

Now, don't get me wrong, it was not easy. We had a lot of trial, a lot of failures. Fail often, fail fast though, right? So that's okay.

Are you in? Okay, great. He's connected to Slack, so that means he is now enabled in our identity system. So he's got to do the customary "I agree," and then we'll make sure that he's on board there. Great. That's awesome.

Okay. So now what we want to do, just real quick, is before he issues his day one command to build a repo and have a website up in Azure within about 10 minutes, I want to show you how simple it is to get a new web app deployed up into Azure.

Are there any people in here that use Azure today? That's good. There's a number of Microsoft uses Azure, we hope, right?

There's a number of tools that are in the environment that allow us to not only integrate within the Azure space, but also to take the templates that are preconfigured, either use a vanilla one or change that to your specifications, and then save them. We're going to show that in a second.

So what I'm going to do is just create a new web app, and this web app is pretty simple. All it's going to do is create an S1 version of a web app, which is pretty basic, 1.5 gig of RAM, two core. Just going to call this `does17webapp`.

We're using subscription, obviously dev/test, but we're also using a predefined resource group. Resource groups are important, and the main reason why is because we can group like assets within a resource group and basically manage the cost and the use of those.

So we'll go ahead and create that. What it's doing is it's verifying that this is a unique name, and it should kick off.

I did purchase the Wi-Fi here, so if something happens in here with Wi-Fi, I'm going to blame it on the hotel DevOps group.

Jessica DeVita

I don't know if they'll be doing that. That's like a team, right? It's a whole separate team.

Chris Lundy

So, it's still validating. We'll let that validate for just a second.

What we're also going to do is talk about Visual Studio itself. If you're in the Visual Studio environment, it's very easy to build a web app, push publish, and it's up in Azure. But is that what you really want to do?

At Columbia, we said no. We realize the developers have autonomy to do their thing, and we appreciate that, but within this pipeline, we may have developers who are not on premises, or they may not be in our offices. So what we really want to do is give them the frameworks and the pipeline already prebuilt, everything within the boundaries of that pipeline, to be able to get their code to development and to production safely.

Okay, so what that means is that we had to tool to make sure that the developers could just get right to coding.

So let me show you here, this is just about finished. Okay, let's just go ahead and switch back. We're going to go back to--

Jessica DeVita

And while we're talking, and we use words like simplifying, and we hear words like abstractions, this is not to say that any of this stuff is simple. What we're doing is easing the onboarding. They're easing the onboarding for new people. That has a ton of benefits that ripple throughout the system.

But it still isn't simple. We're just presenting the right abstraction at the right time. But you can see how mindful they have been about the additions to the pipeline that add in the safety, this notion of how can a developer on day one actually make a meaningful change that delivers value to a customer, and do so safely, and come home at the end of that day and be able to really feel great.

It's not so much that the change that we're going to see happen here is that complex. It's just about seeing the pipeline end to end.

Chris Lundy

Yes, okay.

So now, Scott, I need you to issue a command into Ryu. It's `ryu new demo vsts repo`, and in quotes, `does-171114`.

Now, what's going to happen, go ahead and hit Enter, is he's issuing a command to Ryu to create a new repository in Visual Studio Team Services.

It does more than that, though. What it's doing is it's creating an automated pipeline. It's identifying that this is a repo that has a web app service associated to it. So it's going out to Artifactory. We have a number of open source tools we're going to talk about. It's going to Artifactory, grabbing the build cycle, the build process. It's also grabbing the ARM template from Azure, and it's going and creating that ARM template and renaming it based off of the naming convention here on this particular repository.

Now, this takes about, what, about two minutes or so, roughly, to get done. And what we've done is we said, okay, we want the developer to have everything they need from day one to start coding. So if you remember, he got provisioned within a matter of minutes. He created his account, he logged into his account, so now he's in the environment, and now he's just waiting for his repository.

It just doesn't have the repository, though. We have a distribution of Read the Docs. I'm sure some of you are familiar with that. It's a great tool. We use it extensively. We've decided to put Read the Docs distros in each of the repositories so that each of the developers has their own documentation standards.

Jessica DeVita

That's kept with the code, and it's important that you do that to help create that documentation culture and set up the behaviors that you want.

Chris Lundy

Yes, that's correct. It is kept with the code.

The next thing, so you notice it's finished. The next thing is that we have a distribution of SpecFlow. SpecFlow for ATDD, or behavior-driven development, feature-driven development, is there. So we're going to show you some of the things that we built here.

Here are the files that were created in our Visual Studio Team Services. Those are all of the PowerShell scripts needed to run the repository. We also have documentation, as well as a standard web app service.

Can you go over to the build? Pause for a second, please. Give me the time to... There you go.

Okay, so this is the build process, and you'll notice that it's starting to run. And what it's doing, again, is it's grabbing the templates, it's processing them. You notice that we had SonarQube in there. You want to open that? Thank you.

So, Scott, as you're coding, as you're making changes to your code, you're going to have to validate that you did a good job. You're going to have to validate that you didn't open any security holes. You're going to have to validate that you're meeting the OWASP standards, because SonarQube allows us to add those standards in the build process.

Okay, the next thing is, you'll see what's happening is it's deploying up to a dev instance in Azure. After it's finished with that, it tests that dev instance to make sure it's running properly. And then once that's finished, you notice there's a notice there that says it's going to run swap slots.

Is anybody familiar with swap slots in Azure? Not too many, and this is an awesome tool. The reason why is, you're familiar with A/B testing, blue-green testing. The way we're using it, and we're going to use it more extensively at Columbia, is to essentially create a staging and a production instance side by side. They have two instances running, A and B, on both instances of staging and production.

What happens is, when Visual Studio Team Services deploys up into staging blue, it creates staging green. As you make a change, it deploys to staging blue, and then when it's finished and tested, it swaps them, and the site never even knew it did it. So now you have continuous feature delivery within your staging environment.

But it goes beyond that, because what we're going to do with Visual Studio Team Services, with some of the open source tools that we have, we're going to not just swap within staging or production, but swap between staging and production, so that we have global resiliency on our tool set.

The next thing is, this is Read the Docs. So Scott, I'd like you to make a change to the title of Read the Docs.

So he's going to clone the repo. And he's listening to his music. He's rocking out to '80s--

Jessica DeVita

We have a policy. He can't have headphones on at work.

Chris Lundy

'Eighties hair metal. So he puts his hood covering on there, his hoodie.

Okay, so he's connected. He cloned the repo. Now he's going to go into the Read the Docs distribution example, make a Markdown file. You can change the title of that.

Yeah, let's just add DOES 2017 to that. Pretty simple with regards to using Nano, for those who like to use Nano. Whatever tool set you like to use.

Now he's going to commit that code, and then we're going to switch back to the build pipeline so we can see it running.

And as he's doing that, and we see it kick off, which we do, we're going to have Scott go back to the index file in the web app and make the same change to the web app, and we'll kick that as well.

And each time this is running, we're going through the same checks. It's going through code quality checks, security checks. It's going through unit tests. We're going to talk about the SpecFlow process in a minute. And each time that it runs, it has to follow and meet those checks.

So for instance, let's say somebody's coding and they introduce a bug into their environment that's critical, or into their code that's critical. SonarQube will check that, and if it sees a critical bug, it will stop the build. It will fail it, and then SonarQube tells the developer what to do.

Jessica DeVita

I like that notion of being able to, in the software expression, pull the Andon Cord. But we're having software do that when something doesn't adhere and meet the standards we need.

Chris Lundy

Yeah, correct.

Jessica DeVita

So it's really ideal the way you've thoughtfully put those stops in the pipeline in place.

Chris Lundy

Yep. Okay, so this is our website before he made the change. Pretty simple. Pretty easy. Static website sitting up in Azure. And that's it. We can just change that. Okay. Please do.

Jessica DeVita

We're 20 minutes into his first interaction with Hubot, which is allowing them to really standardize and be able to leverage APIs to talk to their infrastructure, talk to all of the moving parts. And Ryu's out there fighting the battles for him, getting him set up and running within just a few minutes here.

Chris Lundy

Yep. Okay, so the web app is up. Go ahead and show that. It should be updated now with the changes that you made. I don't think it's updated yet.

Jessica DeVita

Not quite?

Chris Lundy

Let's go to Read the Docs. Okay. So let's double-check Read the Docs. There you go. Read the Docs just updated with the last change.

This is nothing new. Build processes, continuous integration is nothing new. How we've put it together is new.

Jessica DeVita

Yes.

Chris Lundy

A lot of companies have struggled to tie all of these processes together. We leverage PowerShell, Chef, Terraform. What are some of the other tools that we use, Scott? Open source tools.

Scott

Grafana.

Chris Lundy

Hubot, Grafana, a number of tools to make this work.

Okay, so now what we're going to do is have a little interactive session here. Now that the web app is up in the repository, what I'm going to do is do something that every developer hates and every code quality person hates.

Thank you, Scott. Appreciate that.

Jessica DeVita

Thank you, Scott. Welcome to Columbia Sportswear.

Scott

Thank you.

Chris Lundy

All right. So what I'm going to do is I am going to change some code here, and essentially I'm going to add a new Twitter feed, and hopefully this will work. I'm going to commit that.

And what we would like you to do is to tweet something when this change commits. We'd like you to go to the website and--

Q&A

Q: I have a question. When you were making changes, for one class, you got them and you said, "Make the same change in the web app?"

A: Chris Lundy: Yeah. One of them is web app, the other one is Read the Docs. The documentation aspect.

A: Jessica DeVita: The documentation. We want to ship documentation in the same repo alongside with the code, and so the Ryu chatbot creates that behavior automatically. So the first change he did was to make the documentation line up with what he was going to code, and then he made the actual code change for the web app so that all of that gets shipped together and tied to the release.

A: Chris Lundy: Yep.

Q: So that's coming once program will ship--

A: Jessica DeVita: This is what's working for Columbia Sportswear. This is how they're using Microsoft Visual Studio Team Services with a whole bunch of open source software. It's not just for .NET. They're doing this with Java teams. You're doing this with .NET. You're doing it with a number of different tools.

A: Chris Lundy: A number of different tool sets, different teams. They have different skill sets, JavaScript, Java, .NET. They're learning.

Chris Lundy

Each of those environments, each of those teams needs to learn. And the reason they need to learn is because the tool set is one thing. The skill set and how to think about how things tie together, how the change that you make in code will affect security or will affect the performance of an application, because we talk about performance and load testing.

How many people can spin up a performance and load testing environment and then destroy it when it's done, take the measurements, make tweaks to the site and to the build process, and then do it again? That's not easy to do. But with our tool set and with what we're doing, the direction we're going, we're able to do that.

So here's what we're going to do. We're going to go into the web app here again, and hopefully it will show up. There it is.

Jessica DeVita

There we are.

Chris Lundy

And we're waiting for--

Wait. "Welcome, DevOps Enterprise change agents." Awesome.

Jessica DeVita

Love it.

Chris Lundy

There it is. So that was Scott's change there, and he's talking about you folks.

Let's just watch this process run. And what we want to do is when this comes up, we should have a tweet button. We want you to tweet something, and it has some hashtags already defined and some mentions.

Jessica DeVita

Thank you, marketing.

Chris Lundy

Yeah. Thank you. And also, we'll have some prizes to give away after the talk.

Jessica DeVita

The URL.

Chris Lundy

Oh, yeah. So let's--

Jessica DeVita

Yeah, if you could bring up that URL.

Chris Lundy

Switch over to that URL. That's right. There you go. Let me just put that up. So then you can use the Bitly URL, see if that works.

Jessica DeVita

Or you can type the long form. They both go to the same place.

Chris Lundy

You got that? I'm going to switch back. Why don't you give us a tweet here?

Jessica DeVita

Yeah, come on.

Chris Lundy

Make sure it comes back. The right one. Not yet. That's all right. We're waiting for that refresh.

Jessica DeVita

Yeah. Great.

So even though you talked about a lot of the need to design, spend a lot of time designing up front with the whiteboard, using really collaborative conversations, and then you built this, and you tried a bunch of different things that worked to varying degrees. You tried a number of tools that didn't work at all.

Chris Lundy

Oh, yeah.

Jessica DeVita

And this is the thing we want to encourage is just try some stuff.

I know that we have requirements about which cloud we can use and things like that, that we are not able to make those decisions necessarily. But please be encouraged to go out there and try some stuff, and look to this for inspiration.

You can't copy and paste this and have it work in your environment. Your unique environment is... The complexities just... You can't copy-paste. But let this inspire you.

Chris Lundy

Yeah. Okay, so as a good architect does, I broke the process.

Jessica DeVita

Nice.

Chris Lundy

I didn't code properly. So I'm going to have you switch. If you'd like, you can... Let me just zoom in real quick. There you go. That doesn't do that. So anyway, `does-171012a-webapp-dev.azurewebsites.net`. There we go.

Okay. So if you'd like to tweet, you can do that.

What we're going to do is talk about now how and what we used. Notice the first thing that we talked about, Read the Docs. Read the Docs distribution is there every time we create a new repository.

Here's the repository Scott just created. Take a look. What you see here is the documentation. This is the documentation that he had. This is important because what we're doing is we're positioning the analysts within the organization to go ahead of the developers, find the integrations that their product team is going to work on, and then document those ahead.

So what they're going to do is they set up and say, "Okay, here's my new integration. Here's the scope." Like the theme or epic within an Agile project.

Then what they do is they take that the next step, and then they go to requirements as code. How many people are familiar with SpecFlow and the Gherkin language? Good. Great. That's awesome.

What we've done is we've built that process into our build pipeline so that the analysts can go through and document scenarios.

So here's what Gherkin does and what SpecFlow does. We can create a scenario, and notice this scenario on the page here, and this is in Visual Studio for Mac. The scenario is the site presents a blank form. Given the main page, when the page loads, then a blank contact form is presented. Fairly easy.

Jessica DeVita

That's a really great framework to start thinking about how can you teach other people to think this way, think with this lens, right, of test-driven development, behavior-driven development. Given, when, then. Those are very action-oriented and help people understand what they need to test for and how to describe it.

Chris Lundy

Yeah, absolutely.

So then what happens is they take that. The analysts take that, and they extend it: given, when, then, and, or, but. They also put in these example tables. So if you want to predefine a test case with values, you can do that here.

And then notice what it does. Every single time you create a feature file, it creates the code behind. And if you take a look at this, it basically stubs out all of the words that you stated. Here's that table. Here is the process that the developer can go through to go and hook this particular test case up to their code.

When they do that, they can create an assertion that will test that scenario. Now, don't you think it's important to know what you're testing before you build it? Well, some people don't feel that's important, but what we've found is that when the analysts are documenting the scenarios first, we have a much more predictable delivery of code. So SpecFlow has been very important for us.

Another tool that the developers will use is, again, SonarQube. Here's a picture of that. Every single build that runs has to go through SonarQube.

And what you'll notice is here's Scott's web app. And the nice thing about this, this is open source. You can download it, you can deploy it. You have bugs. If the bugs are not critical, why not let it go to production?

The other thing this does is it tells you if there's an issue, what to do to change it, and then how much time it should take to fix that issue. And then on top of that, it even extends it further, and it tells you what the tech debt is if you don't fix it.

On top of that, you have dashboards that show you the vulnerabilities within specific code. One of the big things that we want to make sure of is every time code goes out, we don't have any vulnerabilities. So the tool will tell us what they are, and it will tell us how to fix them according to the standards. So that's another one--

Jessica DeVita

Very cool.

Chris Lundy

Of the tools that we're using.

Within the build process, we have all of our build profiles in Artifactory. So they're artifacts that get picked up through PowerShell and run. You can see all of them here.

The other thing that we have within the build process and within Artifactory are architectural patterns. So we've created a number of models that we're using within Visual Studio, that we're using within our new digital platform, our open data platform, is what we call it there at Columbia. And so if a developer wants to understand how to code something, they can reference right within their code base the architectural diagrams for that particular pattern.

We heard about circuit breaker. We heard about service bus. We have those patterns as well, and they can see which assets to utilize within their process, and then they just have to configure and code their changes. So that's another nice tool that has saved us a lot of time.

To give you an example of that, we were talking with a developer, we were whiteboarding essentially a new web app that was going to check the dead-letter queue within Azure, Azure Service Bus. And after we whiteboarded it, asked the question to the developer, "How long will it take you to build that?"

He said, "Well, if it was two or three months ago, it would take us about four weeks." That was kind of our standard answer.

I said, "Well, now that you have the pattern and you have this build pipeline, how long will it take you to build that?"

He said, "A day." And we believe him because he did it. He had a working prototype within a day, which is pretty awesome.

That is the power of putting these things together. We hope that you take these, we hope that you see what we're doing and extend that yourselves. You have open source tools as well, and we hope that you all do the same thing.

So let's talk about our release management. You remember the model. We had the model of change management.

Jessica DeVita

Needs to change.

Chris Lundy

It needs to change. And even our change manager came and told us the same thing: change management needs to change. It needs to be lighter, more nimble.

So we asked the question to one of our product owners, he deals with the product development within Columbia, "How fast do you want things to happen?"

He says, "As fast as you can make them."

Okay, so what does that mean? Every time I have a change, I want to be able to go to my developer, make the change, and commit it. When that's committed, I want it to go. I don't want to mess with it anymore.

Jessica DeVita

Chris, I'm reminded of a quote from Don't Reboot Me on Twitter, and he says, "You enabled me to do the DevOps. You just weren't ready for me to do the DevOps."

This question about how fast can we go, there's going to be time where people have to learn that they can trust your systems. And that just happens organically as you start these smaller projects and see where they take you, and you show value, and then it can grow from there.

Chris Lundy

Yeah.

So another thing that we want to bring up here, some of the patterns that we're using. And these are really why you design the way you do.

Configurable compliance for any region. If you're in Europe, if you're working in Europe, GDPR is a major concern by May of next year. Why is that important? Because May is when the regulations come in. They can take up to 10% of your revenue for one violation.

Who is subject to GDPR? Great. Yep. So very important point for us. We have to make sure that our systems, the data that we're using within those systems, is compliant. So we can use Azure, we can use this pipeline to help us with that. We can create custom pipelines per region, change the compliance, change some of the tool sets.

What was the one that you mentioned from Chef the other day?

Jessica DeVita

InSpec.

Chris Lundy

InSpec.

Jessica DeVita

We're seeing a lot of customers using a lot of the open source tooling. One of those is InSpec. It's a language for compliance. You could bring that in earlier.

Chris Lundy

Yeah, absolutely. That's very important for us because we don't want to get charged with a fine, right? So any region, having compliance and pipelines built for that region.

Secure, auditable, and traceable by design. So our security, InfoSec team, they told us, "Follow the OWASP principles. That's where you're going to start, but we need to know every single time a build goes out."

So what happens? They're part of the build process. We've embedded them into the Open Data Platform team to make sure that they're guiding us in the right direction, but also they're providing us feedback that we can insert and inject into that pipeline, make tweaks and necessary adjustments.

Jessica DeVita

And they're sharing their context, which you don't have because of your operations background or your unique skill set. And so I love that they're bringing that to the table.

Chris Lundy

Yes.

Another thing: shippable, automatable, repeatable, predictable. These are some key tenets of our process.

So shippable, as in you can go anywhere in the world. Automated, to do as much as we possibly can to remove the touch factor from it. It doesn't remove the thought factor.

SpecFlow, Read the Docs, making sure that patterns are within the code base, requires that the developers and the engineers think differently about how they're coding. This is just a natural progression.

As we mentioned, we're trying to become a learning organization where the engineers, the QA folks, the analysts are learning new skill sets and can contribute to this process overall.

The other thing it is doing is we've seen very predictable results. We know that if you go and grab a web app service or a web API from Artifactory, or even a database, that we're going to get a database. We know how much it costs because we've associated resource groups, cost centers, as well as subscriptions as tags.

Jessica DeVita

I'm glad you mentioned that too, because even as we think about moving from this notion of a project funding model to a product funding model, that is a transition that we talked about at the DevOps roundtables yesterday. It was a lively discussion.

So how will you get there? And I think some of the things you're doing around the tagging, and the fact that this is happening programmatically because they've thought about it in their design and embedded it into the process.

You can't expect behavior change if your systems don't support, because people won't believe you if your systems don't do that. So anyway.

Chris Lundy

Yes. No, that's a good point.

Jessica DeVita

I could go on and on.

Chris Lundy

No, that's a good point, because we have to be believable.

What we're trying to do is we create small processes that we can take to the leadership, we can take to the engineers, we can take to the analysts, the product owners, let them touch them, let them see them, get the feedback from them, and then make change. So those little incremental changes are buying our next project.

We don't work on a shoestring budget. We work on an efficient budget.

Jessica DeVita

What can you do with no budget, no people, and no time?

Chris Lundy

There you go. Yeah.

So in doing that, we have to be scalable.

Jessica DeVita

Scrappy.

Chris Lundy

Exactly. Extensible. We have to be elastic. So elastic as in, we are in the busiest time of our season, so we're scaling. The system will scale automatically out, and when it starts to go back, it will scale back in. And what do we do? Nothing. We watch, we monitor, we make sure it's working. That's what our pipeline will do for us.

Idempotent, got to throw that out there for the architects. Self-healing. These services that we're building, we can't be down. One minute of downtime costs us tens of thousands of dollars. Some companies, millions. We just can't do that.

And then transportable. If we decide tomorrow that we want to go to a different cloud, the services that we built, the processes that we built, should be able to be pointed to a different cloud service and put in that cloud service with minimal effort.

Jessica DeVita

Absolutely.

Chris Lundy

The second thing, again, we talked about day one onboarding, developer and analyst onboarding. So our chatbot, love the thing, it's working really well. But we have business analysts, we have business product owners that are using the chatbot to do things.

"What's the status of my service?" It goes out and it looks at all the statuses within the systems and returns a message and says, "Here. Here's your status."

Reducing the time from weeks to minutes. Also, our repositories are preconfigured with the frameworks and standards needed. And then security, code quality, and testing are integral to that pipeline.

So we have several tests that happen. When the code is written, SpecFlow is written, Gherkin is written, unit tests are automated through NUnit. SpecFlow allows us to integrate tightly with NUnit, but we go beyond that. So that's unit testing.

Those same scenarios that were written in Gherkin are going to be used, we're working on this now, within our automated testing framework. So the build process, when all of the unit tests pass, it will go and kick off this other process that will test functional testing.

Not just the server-to-server interaction or API-to-server interaction, but actually going out to system A and system B, throwing data at it, testing it as though a user were using it. The results come back into the build pipeline, and if they don't pass, that build doesn't continue. Goes back, they start over again.

Then we're extending that even further. Next year, we're bringing Dynamics 365 into our environment. It's a major project. And so as part of that, we want to be able to spin up a performance and load testing environment, do massive tests in Azure, and then destroy it. We don't want that super expensive system running up in Azure for no reason. Right? Long-running systems are detrimental to the health of your environment.

Jessica DeVita

I like how you're taking on more of the business-oriented tools and applying the DevOps to those. I love to see that. I love to see how you're doing that.

Chris Lundy

Yeah. So let's just recap here.

How long did it take us to build this process? Well, I'm going to say the initial process took us six weeks. Was it easy? No.

What did it take? It took four developers that were embedded in the Open Data Platform team, two weeks, two weeks, two weeks. Three sprints to make sure that they wired everything together.

We started with GitLab. We love GitLab. But GitLab didn't work with some of the tools that we have. It won't work with Dynamics 365. So guess what? We stood that entire instance up, had the pipeline running in the first two weeks, said, "This isn't going to work." We threw it away.

We went to Git. Did the same thing. Guess what? Didn't work. We have other tools in the environment that won't work with it. So we got rid of it.

Then reached out to the Visual Studio Team Services Group, Jessica and her group.

Jessica DeVita

Pinged me on Twitter.

Chris Lundy

Yeah. There you go.

Jessica DeVita

And DM.

Chris Lundy

On Twitter, literally on Twitter. "Hey, we need some help." She said, "Hey, use VSTS." Got VSTS in here.

Jessica DeVita

Actually, what I wanted to do is really talk for a second about the customer development work that I have been doing with you, and with bringing your teams and connecting them into the product team at Microsoft.

Each of you have an area, probably, of Microsoft technology, Azure, VSTS, that's troubling. Maybe you want to know how to integrate it with other tooling.

I'm not here to tell you which tools you should use. I'm sharing their story because it's really important that you know today that it's not just for .NET. But I'm not going to sell you a product. I'm not a salesperson.

I believe in what you're doing and how you're extending it. It's not just about the pipeline, just all of these things you've crafted around it.

This is it. How do you integrate with the tools you have, pick the new tools that you want, find systems that are flexible enough that you can look at the technology of the future, like Disney is telling us, and make decisions that enable you to pivot?

Your business needs are going to change in the middle of all this design work you're doing. Are you crafting something that's flexible enough to respond to the changes that are going to happen and that you may not be around for because you're off doing another cool thing, or you're backpacking in Hawaii for a year? I don't know.

This is about building systems that support the humans, and making their lives better, making that a better day for them. How do we make today better for you?

Chris Lundy

Yeah.

Jessica DeVita

Whether that's with one of these tools or a different one, it doesn't matter to me. I just want you to have the right things.

Chris Lundy

Yeah. So remember at the beginning, we said try it. Just don't be afraid to try it. Do it. Go in there, tool something, see if it works.

The other thing that you need to know about working within Azure, working within VSTS, and some of the tools that you may have in your environment: ingress is easy. Getting things into that environment is easy.

Jessica DeVita

Getting it out is not always so easy.

Chris Lundy

Keep it in mind.

Jessica DeVita

Right. Keep that in mind.

Chris Lundy

One of the things that we saw within VSTS, for instance, is though it worked for us, there are some assets like D365 that are not fully integrated yet. So that means we have to build tooling. We're partnering with Microsoft, as I mentioned earlier, to try to get those tools working within that.

So what do we need help with? I'm going to go to the bottom one first.

We are actively working on this automated database deployment with data injection on a global scale. So this isn't just about working within North America. We have many regions around the globe where we have to have data, and we have to have databases.

So leveraging Azure, we want to be able to deploy an asset through the pipeline and then follow that up with the data, inject the data into that database so that it's usable. Like I said, we want to go from having weeks or even months to prepare a staging environment to the click of a button or the deployment of a pipeline. And that is our goal, and I think we're going to get there.

Scott and his team, super smart, ready to do anything. They'll try something. They'll throw it away. We'll try something else, and we'll go until we're successful.

The other thing that we want to talk about is the ability to do automation within 365. If anybody has Dynamics 365, and they've cracked this open, and they're able to fully automate the different environments within that, we would love to talk to you because this is a thing for us, and we want to make sure that we're able to do this successfully. So looking forward to talking with somebody about that.

Jessica DeVita

And the magic that happens when you can come to a conference like this and collaborate across your industry and find peers doing things that you just never imagined. So just an incredible opportunity for us to be here, and thanks for taking three days out of your job and your life to be here at this conference and learn.

Chris Lundy

Yep. So what's next?

Jessica DeVita

Well, this has been really fun. I'm glad that you took the time to stay through the talk. I'm so glad to have shared the story of Columbia Sportswear.

I would ask for just another 30 seconds or a minute of your time to ask you and mention that there's a DevOps assessment that was designed by Microsoft in collaboration with DevOps Research and Assessment, the DORA folks, Dr. Nicole Forsgren. And if you go to devopsassessment.net, you can take that with your team, on your own, to sort of see where you are in your process. It can give you some action areas that you might want to do first.

Is it version control you need to do first? Great. That's where you are. Where you are has to be okay. Being at a zero of DevOps, if that's the score, that's okay today. Now you've got a couple of tools.

Take the DevOps assessment. There's a game in the DevOps app, and if you post your completion page to that, you get a bunch of points and come by the booth, and I will give you a T-shirt, if you're into that thing with the T-shirts. And I do have the lady sizes. I'm excited.

The other thing I want to call your attention to, and I mentioned this once, is that everything that they are doing, again, we use the word simple, I just caution you, these are all really complex systems that are constantly evolving and breaking. And we've designed for resilience. And we've used all the configuration management. We've done the DevOps. We've followed the patterns. We've been learning about them for years at this conference.

But things still break, and in spite of everything that he's done, there are still going to be outages and incidents. Think for a second about that last outage or incident for you and what that meant for you in your context.

And then take a moment to check out the talks from John Allspaw in the session. I'm not sure what his scheduled talk time is. Please go to that. Please go to Sidney Dekker's talk.

These guys are talking about the cognitive aspects of all that database design. What about all the cognitive work that we're doing in our heads to get ready to add something, run into the system, to scale something?

These guys are doing incredible research into these areas because we're talking about codes and pipelines, but what are we really talking about here? We're talking about safety.

We're talking about that day one productivity of knowing that you could push a "Hello World" app or a full product catalog to production and know that you are able to do that with some semblance of safety, in that they've thought about the security. They've brought those people into the design process. Don't do your DevOps in a silo. Spend time with the people.

And check out the report that John will talk about probably in his talk, which is located at stella.report, just like it sounds, http://stella.report.

If you're doing the DevOps and you want to get a pipeline going, please read that paper and listen to those talks so you can understand why you're doing it. Understand why the safety patterns that you have set up matter, why documentation matters.

Read that, share it. I think it'll enrich your conversations at work. I think it'll make you more empowered to actually make a difference in your job and the jobs of who knows how many hundreds of other people at your work.

You just don't know how many people you're helping. I love that about my job. I love that I can talk about tooling with you, but also talk to you about why we are doing the tooling. Yeah. Does that make sense?

Go try something. Go spin up a-- follow a tutorial. Try something new that maybe touches on some of these areas. Yeah.

Read the STELLA report, and if you'd like to talk to Chris and myself about anything at all, could be what we've talked about today or your unique issue or challenge, we're going to be over at the Microsoft booth in the expo hall, and we've got a big whiteboard, and we can talk about technology a little further.

Please stay in touch with us on Twitter. I'm @UberGeekGirl on Twitter, @EA_Innovation on Twitter, and @DevOpsColumbia. They are hiring, and so are we.

And if you prefer handwritten sentiments, if you know what that means, feel free. Yeah. Thank you very much for being with us today.

Chris Lundy

Thank you.

Jessica DeVita

Thanks, everyone.