Scott Nasello

Originally, the company was built on taking a problem and having an innovative solution. It's the same today, that we need to be focusing on innovation, being different from our competitors.

If you do your best today, maybe tomorrow you do better.

Maybe you're on the fence about DevOps. Perhaps your own journey has stalled a bit, or maybe you need the nudge that Gene mentioned, where we caught his talk about DevOps and high-performing organizations. I was blown away by his talk, and I left with an autographed copy of the book The Phoenix Project, and I devoured it when I got home. And I promptly bought 20 other copies and handed them out to everybody in the organization that would take a copy, thinking that was the way to get a DevOps journey started.

Little did I know.

And it's really surreal to be here talking to you today about our journey. Never in a million years did I think we'd be here. And if it wasn't for the examples and inspiration by folks like Target and CSG and KeyBank and all the rest that you're going to see this week, we probably wouldn't be here.

I'm here to tell you that if Columbia Sportswear can get a transformation like this started, so can you. In fact, I believe you have everything you already need to get started.

It's not about fancy tools. It's not about spending a lot of money. It's about doing the right things right and chopping wood and carrying water.

As Gene mentioned, we were very unlikely because we've bought our software for a number of years. We're a Microsoft stack, and we run a lot of VMware and a lot of N-tier business applications.

We didn't have a budget for DevOps. We didn't have a dedicated team. We didn't have an executive sponsor that folks like Key and others were talking about having. And we didn't have a mandate. But we knew that we needed to change ourselves, and we relied upon our culture for the cues in which how to do that.

As we saw in the video, you saw a company that is intensely interested in innovation. But what you didn't see is a company also that is a family business at its heart. We were founded almost 80 years ago, and one time, we almost ran out of money.

And so that really brings a financial conservatism to the organization, but also innovation. And these seemingly incompatible ideas, innovation and financial conservatism, don't seem like they match very well.

Unless you find a way to innovate without a budget. Unless you find a way to innovate without a blank check and without a dedicated team.

Here are the main actors in our story. This is the platforms engineering team. They represent the typical disciplines that you would find with any infrastructure team: a little bit of storage, servers, databases, orchestration, et cetera.

And what's interesting is they had to continue to support the day-to-day business: break-fix, maintenance, support, any of the projects that came out of the blue.

And so early on, I told the team, "You may not like what I have to tell you, but it's going to be true."

And the truth is that our industry is changing. Your role as IT pros is changing, and the business needs us to do a lot more than we're doing already.

After one of these sessions, one of the engineers pulled me to the side and said, "Hey, Scott, I feel like I'm uniquely qualified to work on deprecated technology in an enterprise organization."

So I said, "Let's do something about that. Let's find the win-win. Let's find the thing that's good for your career at the same time as moving the needle for Columbia Sportswear."

A little bit about me. I started my leadership journey about 20 years ago in the United States Army. So a happy belated Veterans Day to all of you out there.

And I mention it because a lot of the lessons that I learned from the mentors and the opportunities I had in the Army stick with me to this day and have really informed my approaches.

I think we can all come to agreement that change is hard in any organization, regardless of its size. And in our case, we had all those typical things that you would expect. But because we'd used commercial off-the-shelf technology for so long, we had a lot of residue that we needed to cut through.

And the fact is that engineering team had largely forgotten how to engineer. Whether it was all the undocumented, closed-source APIs that we used, whether it was the very large install manuals that had thousands of screenshots in them, we had become intensely single-threaded.

And if you read The Phoenix Project, you'll understand the character Brent and the story. And Brent is the single point of failure, but also the subject matter expert on the technology. And as we started, we really had Brents in almost every role in the team. So much so that we had a matrix of primary and alternate owners for every technology and business application.

You can imagine what the team thought when I said, "How about if we swap the primaries and alternates?"

Needless to say, we weren't ready for that at the time.

One of the engineers said, "Hey, Scott, we're not Etsy, Facebook, Flickr. Why in the world do we need this DevOps thing?"

And so I conceded the point that, yeah, we don't need to spin up hundreds of servers. That's not really our use case right now. But we fundamentally need to find a way to get better. We need to find a way to become more adaptable, more flexible, and reduce the friction for ourselves, our business partners, and the enterprise.

I love this quote: "You're either building a learning organization or you're losing to someone that is."

In our backyard, we have some very notable competitors, and even if it wasn't the right thing to do to build a learning organization, it's pragmatic if you want to be able to sustain your journey and be able to hire the talent you need to really drive your operations.

If you're not familiar with Turn the Ship Around!, it's the true story written by Captain David Marquet of his time leading the USS Santa Fe. But the story starts off with him spending about a year preparing to lead the USS Olympia. And evidently, submarines vary quite a bit. I'm not an expert on that. But he goes from being the most technically proficient leader in the organization to the least technically proficient leader when he all of a sudden gets re-diverted at the last second to the USS Santa Fe.

And this message really comes home to him when he's on the ship getting it ready to go for deployment, and he gives a command to the crew that is physically impossible in this particular submarine.

And in the after-action review, the crew said, "Well, you gave us this order."

And he said, "Well, why did you follow it if you knew you couldn't do it?"

And they said, "Because you told us so."

So the captain decides this is a dangerous combination, to have a crew that only follows orders and to have a captain that is relatively inexperienced on this particular platform.

The story ends with the same submarine that was worst in the sub fleet becoming the best in the sub fleet, all in the span of one year with the same crew.

How did they do that? Well, did the team really need to change some of the behaviors and mindsets and attitudes? Absolutely. But the critical catalyst was a new leadership idea and a new leader that was brought in.

This story resonates with me quite a bit as well. I'm relatively inexperienced to lead my organization, and so it forced me to stretch and grow in different ways than I would've done otherwise. It forced me to engage the team and really think about how to grow the team and grow each of the members of the team.

And whereas if you're an expert, what do you normally do when there's problems? You jump in, roll up your sleeves, and help the team to solve it. I had to let the team figure out how to solve the problems themselves and coach them along the way.

I believe that leaders own their learning systems within their organizations. Further, everyone in the organization is tall enough to ride the ride, meaning they're capable of growth, and they're worthy of the investment to help them to grow, and that teams are what we're really trying to build here.

We saw in the first two presentations squads and cross-functional teams, both from CSG and KeyBank. And as we started, our talent strategy was really grow the talent that we had before we tried to bring new talent in. Because if we didn't have a great system, they wouldn't stay anyway. After growing the talent, we really wanted to retain the talent, and the idea that happy talent attracts more talent.

In my time in the Army, we leveraged heavily on the Be-Know-Do model, and it breaks down a little bit like: be a servant leader. Be a role model. If you want your team to go outside their comfort zone, they have to see you as a leader going outside your comfort zone. They have to see you taking risks, and they have to see you learning. They have to see you opening up books and things like that.

Know yourself, your people, and your doctrine. And probably more importantly than knowing the rules, you need to know when to bend the rules in pursuit of some other principle.

One of the things that we did in 2016 in our support for learners, we refactored our year-end review process, and we took away all the stuff, all the artifacts that would normally be there, and we focused on two key ideas.

The first idea was learning. How have you learned and brought on external information? How have you shared that learning with others? And how have you learned from your peers?

The other idea was around performance. How have you worked as part of a small team? Remember, we had a lot of Brents. How have you worked as part of a small team to reduce friction for ourselves or business partners or the enterprise? How have you gone outside your comfort zone, and what mistake did you make? What did you learn from those mistakes, and how do you plan to apply in 2017?

As we started our journey, we spent a lot of time with the State of DevOps Report, and they talked about four critical behaviors for IT organizations. And of these, version control really caught our eye. It seemed like it was most foundational to where we might want to go, and it seemed like the easiest of the four. So we started with this idea.

We wanted to go from zero artifacts to 250 in our first year. And that may not sound like a lot, but when state of the art in your organization is, "I'm going to send a copy of my script to myself in email," or "I'm going to rename the file with a `.bak`," or "I'm going to snap a copy of the VM," none of those are version control, by the way.

That was the idea.

So by the end of the year, we had exceeded that, and we got from zero to over 500 artifacts. And one of the things that we did is, whenever someone wanted to make a change, we said, "Well, where's your artifact?"

And you may not have had an artifact to go along with your change, but at least we were training the organization that that was something we were always going to ask for. And we consistently asked for it: "Where's your artifact?"

So I'm going to take you through seven case studies in which, again, we didn't have this elaborate DevOps movement. We didn't have a fancy program. We brought DevOps practices and principles to things we were going to do anyway.

We were guided a lot by this idea of DevOps Kaizen that Damon talked about in 2015. So rather than go for a big bang approach, go for smaller approaches that you can actually use PDCA and other ideas from W. Edwards Deming to guide your effort. Learn along the way rather than go for a really risky effort.

So in our first example, we decided to take the new activity, which was version control, and apply it to a new context, which is server provisioning. Again, work that we were going to do anyway. Let's endeavor to do it in a different way so that we can get a better result.

One of my favorite conversations from this time was over the wall, hearing one engineer say, "You build servers like that?"

"Yeah, don't you?"

And these are two engineers that have sat next to each other for years.

So not until we tried to automate it did we realize that there was divergence in terms of practice.

And I will admit that we tried to use some of the turnkey automagic DevOps toolchains to build our servers. They didn't get us where we wanted to go, so we kept experimenting. And eventually, we got to PowerShell Desired State Configuration. As a Microsoft shop, that was a great fit.

And we had a strong preference for what you see is what you get. And the reason for that is because the infrastructure engineers didn't grow up as developers. A lot of abstractions would be confusing. A lot of abstractions would be disconcerting if you're trying to work through the problems.

So we started with the easy stuff first, greenfield, and we eventually moved on to N-tier packaged application environments. So whereas my team was working on orchestration, operating system, we had a business partner that was building recipes and cookbooks for their packaged app: Apache, Ant, Tomcat, et cetera.

One of the patterns that we introduced quite early was this idea of switching leads midway through. So you take the engineer that had built up a lot of the original tooling, and you move him off to the side, and you bring in another engineer because if this idea made any sense whatsoever, you should be able to take the artifacts that are in version control, and the next person should be able to own those, evolve them, and make them their own.

In terms of results for that last effort, we got well over 1,500 servers. We started with zero, over 1,500 in that first couple of years.

In our next example, this is a data center, if you don't recognize it. In our next example, we needed to move our data center. Rather than do the work the way we would normally do it, we decided to see if we could automate that.

So taking the idea of a CSV file, we had the project management office identify the servers that needed to move in a given timeframe. And the idea was that we would use some technology we had, in this case, a stretched vSAN, and we would automate moving the servers from our side over to the other side. And when they got to the other side, we would test them for connectivity and smoke test them. And if they passed, we would update the database, and we would also let the application team know that their application had moved.

Midway through, as you might imagine, we switched leads, again, to stretch and grow the organization.

In terms of results, we moved over 19 racks of gear without any downtime whatsoever. We dropped a ping as the servers were moved from one side to another, over 650 servers. We ended up with a happy CIO, but more importantly, this work served as foundation for our 2017 efforts that we would do.

The next idea, and we've seen this already in the first two presentations from CSG and from KeyBank, we built full-stack infrastructure teams. In our case, we called them pods instead of squads. And we were influenced by the work that Alice Goldfuss presented at Velocity.

In her talk, she said any engineering team has new work, maintenance work, and break-fix work. And she called those rock stars, builders, and janitors. And the idea was that you'd rotate engineers between the different types of work.

Instead of doing that, we built full-stack teams of four or five engineers, each team having a little bit of the capability of the team: a little bit of orchestration, a little bit of server admin, storage, database, et cetera. And we thought that each of these teams should be able to handle small projects, maintenance projects, and what has turned out to be more importantly, they could also take a rotation with service desk for break-fix.

And the reason why that's important is because it catapulted the entire team into a beginner mindset. And because they were in the beginner mindset, they were starting to have empathy, not only for each other, but for the other teams that are in the orbit of these changes.

And we're now in our fifth quarterly cycle of changing the teams, continuing to vary the conditions to increase and encourage growth within the teams.

At this point, one other thing I'll tell you about the service desk team: at the end of their week, they do a weekly scheduled retrospective. What did you learn, and what do we need to get better at?

And for years, we'd never had a level two organization. We had a service desk, and we had an engineering team. And because we're refactoring work, because we're working differently, because we've broken the monogamous relationships between engineers and areas of interest, we're able to build a tier two organization. Matter of fact, we have one of these cross-functional engineering pods on loan to an agile product team. They're building the next-generation open data platform.

We threw a birthday party for our chatbot.

I'm not sure how the bot got named Ryu, of Street Fighter fame, but it's with us, and we've embraced it.

Why in the world would we want to throw a birthday party for our ChatOps practice? Well, because it reinforces the spirit of teamwork. It reinforces the camaraderie within the team. I also invited our boss and asked him to speak extemporaneously about what this journey has meant to him.

One of my favorite stories from this time was the engineer on the left. He was in the middle of a storage array migration. We also had a team builder, and he didn't want to miss the team builder. So as any good engineer would do, he built the tooling and telemetry to be able to manage this on his phone remotely.

In our next example, because we've built an inviting platform, our SAP Basis team, they were able to build their own chat script to monitor the health of their SAP environment.

And then one other thing I'll tell you: since we're a Microsoft shop, JavaScript and CoffeeScript don't come naturally to us. And so if we really wanted to lower the barriers to learning and participation, we needed to do something about that.

So one of the engineers wrote a wrapper, a PowerShell wrapper, that would take PowerShell, and anytime we checked it in, it would build a CoffeeScript along with it. So as we're running through the CI/CD processes of our chat environment, imagine you could do a "Hello, world" of your chat script on day one, actually in hour one. And we stole that idea from Etsy to really get people involved from the beginning.

So successful was this that we eventually wrote a wrapper for Python to encourage more network engineering participation.

And at this point, we have well over 400 scripts in our version control. And every day, it's about 200 times that these commands are being used. And although we have a platform engineering experiment team of 20 folks, we're touching 120 folks worldwide with these new capabilities, these new self-service capabilities, in which we're really demonstrating automation first.

And it's funny. Some days I'll come in and there's two or three new chat scripts that have just popped up overnight because you have engineers that are empowered, and they're solving problems in real time with their business partners.

At this point, our ChatOps practice really spans the vast majority of our practices, whether it's SAP or Commvault or Teradata. The things that you would never expect that you'd have automation for, we've been able to build in our chat system.

And it also reinforces tier two capabilities. So you could actually have two engineers pairing, even across different locations, and working through these examples.

Increasingly, we're using pipelines to manage activities. We like pipelines because it gives us the repeatability and dependability that we're looking for. And as mentioned in the KeyBank example, if you don't do things very often, that is a great thing that you should look to automate.

One of the killer things that we've done here is we're using pipelines to do our SOX 404 controls. So we have all these controls. They're in our version control system. And because they're in our version control system, I don't have to hunt down the one engineer that knows how to run the control.

And because they're in the version control system, we could actually run them in a pipeline. Not only can we run them in a pipeline, we could automate them to run every day. So think about instead of getting good once or twice a year, you're actually running them on a more or less continuous basis, and you're really minimizing the audit or control drift that you might have otherwise.

We're using pipelines also for our day-one onboarding, DNS changes, a whole lot of other places.

Periodically, you're going to get gifts that pop into your lap. In this case, we had a penetration test. It was a few months ago, and we figured it out relatively quickly. And when I briefed the management team, I said, "I intend to continue to let the penetration test go as long as it needs to go. We've already paid for it. Why not get the most out of this opportunity? Why not get the most out of seeing your team react in real time to an issue?"

And then finally, our DevOps practices of the 20 folks in this experiment has been so attractive that we're starting to work on our open data platform with these new practices. So again, this idea of bringing DevOps to the work rather than having a dedicated DevOps program.

So it's really been a great marriage between the enterprise architecture team and the platforms engineering team. We are heavily leveraging Microsoft Visual Studio Team Services in order to deliver our changes to the environments. And because we're leapfrogging IaaS altogether and we're moving to PaaS for our effort, we have the same expectations for managing those PaaS artifacts just as you would IaaS.

So whether we're talking about Service Bus, Event Hub, Data Factory Pipelines, it's the same thing, and we really want to use the power and repeatability of our pipelines to do that.

If you're interested in learning more about the open data platform and how we're onboarding new hires into that ecosystem, we have a demo tomorrow at 2:50, and we'd love to have you attend.

So we've learned quite a bit over the last couple of years, and we've made our fair share of mistakes. Some of my favorite examples of the transformation with the team is moving from a quick-first mentality to using pipelines, automation first. Moving from maybe confident and complacent to becoming more comfortable with being uncomfortable and more skilled at change. Moving from over-reliance on vendors to really owning what we operate and using loosely coupled technologies, including a fair bit of open source.

And maybe my favorite is moving from postmortems only when someone makes you do a postmortem, to we're going to run weekly scheduled retrospectives. Because there's always something to be learned, not just on the outages, but near misses have just as many opportunities to learn as anything else. In fact, I would say near misses probably have a better thing that you need to learn, because hero culture is generally not sustainable, and you probably got away with one.

So fundamentally, we've moved from a paint-by-numbers organization to an organization increasingly using public cloud, APIs, and an open data platform.

Now, it hasn't been easy. If the engineers were here on the stage, they would say, "Yeah, a lot of times I felt like I was running on empty and I wanted to tap the brakes."

And so we decided to engage with that. And so to steal the words of Courtney Kissler, if we were to honor and extract that reality, we asked them a few questions. How would you describe the journey that we've been on? What advice would you give to your 2015 self? And then give us the war stories. What hasn't been good, and what's been good?

It's been a steep learning curve. Oftentimes in the Microsoft universe, it's harder to automate than it needs to be. It's getting better over the last few years, but still that residue and that scar tissue remains.

Constant changes, all the experiments, never feeling comfortable in our skin, and still having to support all the maintenance and the break-fix and projects and drive-bys and all the other stuff, and doing this new thing on the side.

Be prepared to disappoint someone. It could be your teammate. It could be your spouse. It could be your business partner. It could be your manager. It could even be yourself. You're already working 100%, and adding these new activities is going to feel like you're adding 30% more.

But find a way to help your business partner to be able to help themselves. Find a way so that you can spend more cycles working on the true elements of infrastructure.

And if you have any doubts about where this industry is heading, just go to Microsoft Ignite conference and count how many sessions are in the IT Pro track.

In moving from a paint-by-numbers organization to this new world order, I need to find a way to work differently with folks. I need to find a new vocabulary and empathy because there was really nothing to disagree about in the past, and everything in the future, it's all disagreement. All these different technologies that are oftentimes changing very rapidly, finding a way to work through those issues.

At first, I felt like an imposter. I didn't feel like I could do the work. But gradually, I gained confidence and felt like I could contribute.

Don't be overwhelmed by DevOps nirvana. Each small change is like a contribution to a savings deposit that will compound over time.

So incidentally, just 29 dominoes would theoretically be required to knock over the Empire State Building. There's a white paper there.

Lean into the opportunity. This is a once or a twice in a career kind of opportunity to work differently, find a way to work on the new ideas.

And then in terms of our road ahead, at this point in most presentations, you'll see this nicely paved road going off into the horizon. That's probably not us. Remember, we have started with an experiment over the last couple of years, largely bootstrapped, and we're going to have some bumps in the road, probably some unmarked obstacles, no mile markers, and we may not even know what's coming around the corner, and that's okay.

What I do expect to have happen is we're going to take a lot of the practices that we've started in the first couple years and now start to work with business partners. So we have squads, or in our case, pods, and I do expect that we should be able to embed some of those into those different application teams for a period of time and help them to help themselves.

And this is a riff on the Target dojo idea. So rather than have a dedicated learning facility, we'll bring the learning facility to the people.

So there's a few things that we don't know how to do yet, and we'd be more than happy to talk to you offline if you have the secret sauce. But a lot of it is really working through our organization model. We largely still have the organization model that we had two years ago, and we've started to work differently. And so really kind of figuring out how to proceed forward.

And as we move into open data platform with a lot of API management, really bringing some of that services management idea into our organization.

So thank you to this community. Thank you to all of you that shared your stories that really inspired our own. And thank you for letting me tell the story of the engineers up there.

You have everything you need to start your journey already.

And we'd love to stay in touch, so let us know how we can help you.