Log in to watch

Log in or create a free account to watch this video.

Log in
London 2018
Share
Download slides

When the Business Partners with Tech and They Do a Dojo

To address the challenges in machine learning model delivery, Capital One's Credit Card Technology group formed a cross-disciplinary team with associates from Data Science, Data Engineering, DevOps, and Product Management. The teams began by organizing into a DevOps Dojo.

Chapters

Full transcript

The complete talk, organized by section.

Aimee Bechtle

Hi. Thank you.

John Schmidt

Thank you.

Aimee Bechtle

Hi. Good morning. Thank you for joining us. It's an honor to be here. I'm Aimee Bechtle, and I started leading DevOps transformations in 2013.

John Schmidt

And I'm John Schmidt. I am not an expert in DevOps, so thank you for letting me crash your party.

Aimee Bechtle

A few years ago, Capital One embarked on a technology transformation. We started to adopt Agile, DevOps, and the cloud. We are embracing open source, microservices architecture, RESTful APIs, and security is always a top priority.

I joined Capital One, the credit card technology organization, in December of 2015 to help them adopt and mature their DevOps practices and solutions.

John Schmidt

And I joined Capital One four years ago in product management, and really excited about investing in software to change the world and make our customers more successful.

Aimee Bechtle

This is a relative, notional depiction of John and I in the organization. And as you can see, there are several layers and complexity, and I'm amazed that I found John. And when we met and found each other, we formed a partnership to help his team start their DevOps transformation.

So in early 2017, my vice president sponsored the initiation of a DevOps acceleration service, where we would provide DevOps engineers to the application teams to help them adopt continuous integration, continuous delivery, continuous operations in the cloud.

So I'm leading this new service, and I have been asked and charged to help drive a transformation. And I ask myself every day, as a change agent, "How can I support a DevOps transformation in an organization of this size and complexity?"

What I'm hoping today is that when we share our story about our partnership, you will take away some of those tidbits and nuggets that helped us drive the transformation for John's teams.

So I'm leading this acceleration service. Our motto is "no fear releases." I have a new team, a new service, and I had no customers yet, and I had to go out and start talking to my tech colleagues to get some business.

I was surprised when I started talking to them that I was hearing, "We don't have time right now. You're going to have to talk to our product owner. That's up to our product owner." And I kept repeatedly hearing I needed to talk to the product owner.

I was taken aback, but I guess I understand now that we're all in on DevOps. They're sharing a backlog, the DevOps engineers and the software engineers, and the product owner is prioritizing the backlog. And they're having to weigh the priority of the functional requirements versus the non-functional.

I had this new insight, but I also had an insight that I had a really narrow focus, and this is a little bit of foreshadowing for what John's going to talk about later, in that I wasn't looking at the people outside of my line of sight. And I had to go do something that was not instinctive to me and something I was not very good at, and that's go talk to the product owners and talk to the business.

I shared my insight about the product owners with my director at the time, who then arranged for me to brief the product managers forum that John was leading and is still leading to this day.

I prepared this great deck filled with facts from the 2016 State of DevOps Report, going 22 times faster, just espousing the benefits of DevOps and the service and how my team was going to help bring humane IT to these software engineering teams.

I was surprised after briefing these product managers that I received silence and glazed-over looks. And after getting to know John and hearing from him, these are some of the things that they were thinking, very similar to a lot of my tech colleagues.

John Schmidt

Mm-hmm.

Aimee Bechtle

There was one person, though, who was listening and smiling and speaking up and asking questions, and that was John.

John Schmidt

Yeah. And I was immediately impressed with Aimee and knew what she was talking about. And I'd been struggling to get better speed on my team.

We had a very exciting set of value hypotheses and to invest in capabilities that were going to allow us to do a lot more interesting things for our customers, but we just weren't going fast enough.

And this whole DevOps thing was a bit foreign to me, but it sounded very interesting. And I immediately realized it was going to take a lot more than a couple non-functional stories squeezed into the backlog to get to the speed that we needed.

Aimee Bechtle

So John and I began talking and getting to know each other. I'm disappointed to hear that I didn't have any magic words that made him go, "Aha! I want to do DevOps." And I'm sorry to disappoint you, our presentation is not going to give you a script or prescription for what those magic words are.

But I did learn that John was facing some challenges that had not yet been solved for.

John Schmidt

Yeah. And the team that was working on this particular platform was well established. They were able to deliver, extremely skilled. It was a very advanced engineering investment for us.

But it still took three weeks, at best, to get a line of code from dev into production.

Aimee Bechtle

At this point, I want to commend John. I think it's really hard to recognize and then admit that you have anti-patterns, and that it's even harder to do something about it. And that's what he did.

So my presentation wasn't what did it. What I learned is that John had visited the Target Dojo. Target is a retail chain giant in the United States who stood up a large facility and ran multiple teams through to help with their cultural transformation to a tech company. And he had visited that, and he had learned about the benefits of that model and the benefits of the Dojo.

I had also been practicing a modified form of Dojo when I was in Shared Solutions Engineering in Card Tech, and I also was aware of those benefits. We agreed that that was a model that we wanted to try or experiment with for his teams and for my acceleration service moving forward.

I could probably spend an hour on this slide. I'm that passionate about this subject. And I am not a learning expert by any means, the way that John's not a DevOps expert. But it was fascinating to see that this type of model and what it can do for a team and the whole-team learning model.

The Dojo, after I've run several of them now, I believe is the fastest and most effective way to adapt a team to an engineering culture and build accelerated expertise in DevOps in the cloud.

I mentioned that our motto for the acceleration service was "no fear releases." Now our motto is "no fear change."

After John and I agreed, I started meeting with his product leadership team. He formed a product leadership team comprised of the product owner, the scrum master, and the tech lead. They sat together every day next to each other.

We just got dev and ops sitting together. My mind is blown that now we have tech and the business sitting together.

And I spent about three weeks planning with them. We planned with two teams, two web APIs, nine to 10 engineers, which is about the max that you want to go. This team had worked together for quite a few months. They were co-located. They had some, but limited, DevOps skills.

And we scoped out a six-week challenge to help upskill them to adopt CI/CD pipelines for two different APIs that supported a federated code contribution model, where they would have multiple teams contributing code that had to be integrated into their application and executing the same test code in both pipelines.

I am a planner. I've never done this model before. I was trying everything to get this lock-tight plan in place, and my anxiety went up. And John saw that. This is normal. It's understandable. And instead of quelling it, we embrace it.

And John looked at me and he said...

John Schmidt

It's going to be messy.

Aimee Bechtle

And I just heard Chris Hill talk about psychological safety. I knew when he said that, I now know what that means. He and we were creating a psychologically safe environment in which this team could transform.

This is our journey map. We spent about nine weeks together, the first three preparing. I would love to go into every detail, but we don't have time today, about what each week entailed.

This is an emotional journey. I want to highlight the two dips because these are very important points that are probably going to be experienced in every transformation. And they're points where you can either stop and quit or decide to move forward. And they're very important points in which, as leadership, you need to decide what you want to do and if you're going to push forward, that you're critical in that pivot.

This particular team, as we mentioned, had been working together for quite a while, and their tech lead knew them very well. In that first week after we kicked off on week zero, the anxiety was high. They were uncomfortable. They felt exposed. They were being asked to learn skills that they had never done before. A Chef developer is learning Java. And they were asking, "Why are we doing this? It's not broken."

It was working. And their tech lead just looked at them and said, "You know what? It's a leap of faith." That's what this team needed to hear in order to get the pivot.

And we got the pivot. Then they went over the hump, and they said, "All right, we get it now. We're good. I want to get back to code and business as usual." And we said, "No, you're going to meet the challenge." And they did.

By the end of the six weeks, they had reduced what was about three weeks of testing to three hours. And that's afterwards, I interviewed John for a blog post, and he had these wise words.

John Schmidt

Yeah. I don't think there's any other way we could have gotten there. And the team is truly set up for long-term success.

Even now, just over a year after we completed the Dojo, that acceleration has only continued. And I don't think we could have got there. And it wasn't just the skills and getting the pipeline running. It was also a change in their skill set, mindset, and culture.

Aimee Bechtle

Yep, it was a behavior change.

We moved on. I went on to do a couple more Dojos. I had some great new influencing skills with the product owner and the tech lead. And John moved on to a machine learning and data science space within our organization and started to observe some of the same challenges.

John Schmidt

Yeah. And the slight difference here, I'm just going to say slight difference, was that we had some fantastically brilliant people, data scientists with some really strong technical skills, but fairly limited engineering experience and almost no DevOps experience. And so we were facing a pretty steep hill.

And so I thought, "You know what? I'm going to go see what Aimee's doing."

Aimee Bechtle

And we said, "Let's do another Dojo." And so John called me, and starting last November, I met with another new product leadership team comprised of the scrum master, the tech lead, and the product owner, and we scoped out a new challenge.

This was a sharp contrast to the first team that we had worked with. It was still two teams, and there was a data modeling team and a platform engineering team for the data models. Still about nine to 10 engineers. They were brand new. They had never worked together before. They were distributed across multiple locations. They had no DevOps skills. And they had a pretty steep challenge with trying to adopt the CI/CD pipeline for a couple different machine learning models into a container orchestration in the cloud.

And because of the steep hill, we scoped out a 10-week challenge. We co-located into an off-site facility, and we started. Oh, well, we got a... We knew again. My anxiety went up. And my team did not know machine learning either. We had never done DevOps for machine learning or data science, and these guys had never done DevOps. So it was scary.

We were afraid, to be honest, and our anxiety went up. And once again, John said...

John Schmidt

It's going to be messy.

Aimee Bechtle

So here's a 10-week journey map. And we hit the same first dip, what I call the fear dip.

Do you think if I had said to a bunch of data scientists, "It's a leap of faith," how do you think that would have gone over?

So we had to look at who this team was, and we called in the big guns. We called in an Agile coach. We ran an anonymous survey, and we did what was called a mind mining, where we wanted to understand what they were thinking and what they were feeling, and we would address them.

And so we had a half day of talking about what the results were of that survey. And we started implementing team-building exercises each week. And we established norms and values with this. And we created a psychologically safe environment.

So we got through the pivot, and then they doubled down, and they started flying with learning DevOps. And then we hit the holidays. So we had our second, what I call the mojo dip.

But I think it's important to highlight that what worked for the first team to pivot wouldn't have worked for the second team. And that as a leader, and we want to influence these teams, we have to understand how that team thinks, what their values and beliefs are, and adapt our approach to support them, not the other way around. They're not going to listen.

So as you can see, there's a similar pattern with two dips, and what I call the Dojo mojo dips. Usually the first dip is a "can we do this?" dip, and the second one is, "when can we just get back to code and business as usual?" But that's not what we want. So we commit to at least six weeks when we try to do this.

And you might ask, why six weeks? I don't know how many of you have tried to adopt a new habit and break an old one or establish a new routine. It takes at least six weeks for that to become intrinsic.

So there's a lot of studies on this. Instead of reading all of those studies, what I highly recommend is there's a great YouTube video called "Backwards Bicycle." And if you haven't seen that, it will reiterate this point.

But even after the six weeks are over, I don't think that's enough. In order to sustain this change, it will take repeated practice and continued commitment from both the business and tech.

Today, those teams that had their DevOps journey jump-started by the Dojo and by John are experiencing continued success. And that Dojo model has proved it can produce faster, lasting results even when teams have different skills, technologies, and needs.

John, do you think that you would have gotten as far as you did today had you not done the Dojo?

John Schmidt

No. There's really no way, and I can't really think of how else we could have gotten there. And in many ways, it exceeded my expectations. I didn't tell the team at the time. I told them afterwards.

But in fact, things went so well that I can't even share with all of you a lot of details about what we were able to accomplish.

But a few things we wanted to summarize are: the team was able to create several reusable building blocks that will fundamentally change our ability to innovate. And by innovate, I mean test, develop new types of features, get them out into production into the world, and get feedback from reality if these things are really resonating.

But the hypotheses that we're working against are, what can we do to give our customers a lot more control over their money and set them up for success? And that's something that has been a game changer for us.

The second thing is, and there's probably a DevOps term for this, but there's something from a product perspective I love to call the oops tax, where sometimes you might be able to get everyone together, get some temporary speed, get something in market, but then inevitably, technical debt, other problems creep up, and you start crashing. I call that the oops tax.

And really, the foundational stability, safety, and resiliency of the two platforms has really facilitated ongoing acceleration. So we've had a very smooth acceleration curve.

And the speed is real. I mean, the ability to go from weeks, or in the case of these machine learning models, a very uncertain multi-month journey, to being able to get new code out there in hours has been fantastic.

Aimee Bechtle

I realize now that John is more the exception than the rule. I like to say that I got lucky. John says I created my own luck. It doesn't matter.

What we're hoping is to now shift gears and provide you some insight into how product managers or the business thinks, so that you can take that back to your organizations and hopefully get an engagement like we got.

John Schmidt

Mm-hmm. Yeah. So I'm going to switch gears and use a lot of metaphors. And I'll apologize in advance. I grew up in Canada, so some of these metaphors might not be as accessible to all of you, but I'm going to try to stitch these together. And I'm going to try to, if I can, make a connection between bears, picnic baskets, an angry bag of money, and rowing on a river.

So I want to start off with a quick review of what I consider to be somewhat of an ancient secret, in that if you want to motivate an audience or you want to persuade someone, you need to really understand what's going on in their motivational framework.

And generally speaking, my perspective is most people want to be a happy camper. But in order to be a happy camper, they want to avoid bears and they want to find picnic baskets.

Now, again, I grew up in Canada. Bear safety is an important life skill to learn at a young age. And the picnic baskets, you can imagine whatever you want in the picnic basket. It could be intrinsic, extrinsic motivators. I personally would love to find a nice bottle of single malt scotch and maybe an internet connection in my picnic basket. But it can be whatever you want.

But those are the two major things that are going to motivate your audience, and things you need to tune into if you want to influence them.

And I want to double-click on this point briefly, because it turns out the way our brains are wired, bears are much more persuasive than picnic baskets. And the thing is, this sounds obvious, but it's actually something that I forget myself in many cases. I get in front of someone, and I'm really excited to tell them all the benefits of this great idea I have or the team has and all the cool things it's going to do for us. But I realize only later that I completely neglected to address the bear in the room.

And because bears are so much more persuasive than picnic baskets, it almost doesn't matter how many awesome picnic baskets you put in front of the person you're trying to convince. The bear will outweigh them all.

So, as a product manager, one of the things I can share is I don't necessarily know what all of your business and product management partners might be worried about, but I can definitely share with you a little bit about my bear.

And for that, I want to introduce the angry bag of money.

That's my bear, is an angry bag of money. And I'm just going to tell you a little bit more about the angry bag of money.

So to me, this is what it feels like when I've succeeded in getting investment support. So we have a good set of value hypotheses. The senior executives that are investing in us feel great. But then I feel like we're not making progress fast enough. Maybe our options are too constrained. Maybe we have really painful bottlenecks that are causing an increasingly painful Game of Thrones-like trade-off battles, and/or there's that background fear that competitors and disruptors are going to get there better or faster than us.

And so it feels like the money's done its job. It's there. It's sitting on the table, and it has this look, and it's staring at me and saying, "Hey, I showed up. I'm here to support you. Why can't you go faster?"

So in order to help all of us get to know the angry bag of money a bit better, I'd like to get your assistance in participating in a brief game.

So the way this game works is Aimee is going to pose a question. She's going to ask all of you if the angry bag of money cares about something. And then we're going to want you to respond. You can use volume, say, shout out yes or no, to see if the angry bag of money cares about the thing that Aimee asks about. And I'll be here to kind of provide clues and tips if you get stuck.

Aimee Bechtle

Does the angry bag of money care if your code is in a Docker container?

Audience

No.

John Schmidt

That's right.

Aimee Bechtle

Does the angry bag of money care if you use Jenkins or CircleCI?

Audience

No.

Aimee Bechtle

Ansible or Chef?

Audience

No.

Aimee Bechtle

Azure or AWS?

Audience

No.

Aimee Bechtle

Does the angry bag of money care if on Monday morning, the release is out there and it knows that it's working, but everybody spent all weekend trying to get it out and had several rollbacks?

Audience

Yeah. No.

John Schmidt

Yeah, they're really good at this game.

Aimee Bechtle

I am.

John Schmidt

All right.

Aimee Bechtle

So what does the angry bag of money care about?

John Schmidt

Yeah. Well, does the angry bag of money care about getting value to the business and to the market faster?

Aimee Bechtle

Absolutely.

John Schmidt

Does it care about fast feedback from that customer and the ability to revert to a previous known good state faster?

And does the angry bag of money care whether the people got to go to their personal events or soccer games or parties over the weekend?

Audience

Yes.

John Schmidt

This angry bag of money does.

Aimee Bechtle

Yeah. Capital One, anyways.

John Schmidt

Yeah.

Aimee Bechtle

Capital One does.

John Schmidt

Yeah.

So a lot of the benefits of DevOps often get framed or explained in terms of speed. So I wanted to... And I realize it's probably risky to attempt a physics joke in a room where there's probably a lot of people with a strong engineering background.

But I kind of think of this as a law of physics, is that speed minus safety equals crash. I think it's called the law of trying not to crash. Actually not a real law of physics, but I think the notion of speed, it sounds cool, it sounds fun, but I think speed that isn't paired with an appropriate amount of safety inevitably leads to a crash.

And so I think one of the things that Aimee said to me early on, which really resonated, was the concept of no fear releases, because I thought that was key. Having that element of safety is fundamental to being able to go fast at a sustainable pace.

So gone through a few metaphors, and now I want to kick it all the way up to an allegory and walk you through a very brief story where we have a hero who is rowing on a river.

This hero's great, great rower, knows what he's doing, strong, making a really good pace. And it's actually a race. He's racing against other rowers. He's pulling ahead. He's getting up. He's doing some freestyling. He's actually looking ahead to see all the exciting stuff that's in front of him. He knows he's going to win this race. He's so much faster than the other rowers.

The difficulty is, even though he's really good at this, there's a problem. And there's a problem that's outside of his relatively narrow perspective.

And that problem is that the downstream speed of the current is faster than he's rowing. It's faster than the speed of all the rowers. And actually, not only that, but at the end of that river, there's a very precipitous waterfall. And all of these rowers are going to go over that waterfall at the pace they're going. And they're just too focused on this race that they're in.

And I think the fundamental point I want to make here about speed is that if the rate at which you're shipping and learning is slower than the rate at which the world is changing, inevitably you're all going to go over the waterfall.

And in the short run, you might feel like you're doing great. Maybe you're shipping just as fast or a little bit faster than the other teams in your enterprise, or maybe you're doing a little bit better, a little bit faster than your competitors.

But eventually, someone is going to either change the game and completely outperform all of you and wave to you as you fall over the waterfall, or maybe a disruptor will come in and say, "Hey, maybe I'm not going to row. Maybe I'm going to use an engine." And then your whole industry goes over the waterfall.

And I think this happens all the time. And so for me, when I think about the necessity of speed in a competitive environment, because as a product manager, I want to invest in software to create or sustain a competitive advantage for my business.

If I take that broad perspective, I don't want to be in an industry that goes over the waterfall. And I know that if I can't ship code, I can't learn fast. And if I can't learn faster than the rate at which the world is changing, I'm going to go over the waterfall.

Aimee Bechtle

I began this presentation and I asked the question: how, as I, as a change agent, can affect change in an organization of this size and complexity?

I know we didn't give you magic words or a script to take back and talk to your product owners. I believe, hopefully, that we gave you an approach and a methodology. And if you were to take John's words of wisdom and the approach that we took with the Dojo, it's actually 10 steps. It's on the next slide.

The nine things that we would recommend, and if you adopt how we did it, that could hopefully get you the results that we got.

Step 10 is the most important of all, and that is building a relationship between business and tech.

I've been on an extraordinary journey. It's been probably some of the highlight of my career, has been working with these really challenging teams and implementing the CI/CD and DevOps solutions for them, and learning a lot about how teams think and what's going to work to help influence them to change.

So the irony is that I was hired to help drive change and change how people work, and they ended up being the ones that changed me. And I just want to thank you for that and for all that we have done.

John Schmidt

Thank you.

Aimee Bechtle

We are asked to end with what we need help with, and I'm hoping that we get to talk to many of you today at either a Q&A or at lunch.

We want to hear from you how you have supported culture change inside your organization. Not just technology change, but how you influence the people.

Specifically, what did you do to influence your business to invest in DevOps? And then finally, how did you talk to the engineers and influence them to want to change their skill sets and work differently?

Thank you very much for your time today, and we look forward to meeting you.

John Schmidt

Yeah, thanks.