DevOps Transformation at Scale
The transformation story from waterfall and team silos to DevOps in Starbucks Technology. How to initially begin and scale across your organization.We will share our strategy for transformation which was created and is being executed by the team members themselves, not through the direction of a consultant or shared service coach.
Suzanne brings the perspective of a manager with an evolving role and long term employee.
Sarah is a member of the working team and has been with the company for one year.
Chapters
Full transcript
The complete talk, organized by section.
Suzanne Nielsen
We are here to share our transformation story. Gene mentioned that there would be experience reports. We're here to share what we have been doing. We definitely consider ourselves early in our journey. So if you are new to starting, we are here to show you what's possible. And if you are a ways in your journey, we're here to learn from you.
This is the Starbucks mission, and as Courtney mentioned, I am very passionate about Lean. And when I was thinking about joining Starbucks, the mission really spoke to me. It captures WIP. It talks about one person, one cup, one neighborhood at a time, and it really leads with empathy. I think you've heard that through many messages that have been here today. So our journey, all of those things are focused on empathy and meet people where they are with respect.
So most of you have probably heard about Starbucks by now, I imagine, but you might not have known how large Starbucks actually is. So we have over 25,000 stores worldwide across 75 countries. Up to 90 million occasions per week, people come in our stores. And we have over 330,000 partners, or employees. We call our employees partners.
And so most of you, if you've been in IT very long, have probably worked on POS systems. We actually work on point-of-sale, POS, systems. And so that's all of the hardware and the software in the stores that people come up and order coffee and make their transactions. So the devices, the scanners, the swipes, and everything like that.
So it's about our journey, and the point of sale team at Starbucks is almost up to 100 people. So it's a pretty large group of people to really maintain, operate, and enhance the system. And our story is more about the team that we've been working with, the sub-team of that over the last few years.
Okay. Here's our little bio. But as Courtney mentioned, this is my second DevOps transformation. The first one, I was part of the mobile team at Nordstrom, and we know that for enterprise, we're not the unicorns here. But that team that I was part of was kind of considered the unicorn of the enterprise. And I wanted the challenge of what does it take to turn a monolithic system, somebody using Oracle, all of those things. So I thought, why not go to Starbucks and start working on that here from the perspective of strategy?
Sarah Shewell
And I'm a long-term Starbucks partner. I've been at Starbucks, it will be 20 years in February. So that's a long time. You just saw the slide before this, how big Starbucks was. When I started 20 years ago, we had, I think, about 15,000 employees and 1,500 stores. So that's a lot of growth in the 20 years that I've been there. And I'm a native Seattleite. I think Sarah also is. Two of the very rare that are native Seattleites.
Okay. So through our talk, you will hear us mention many things from these books. This is kind of our quick must-read list. We are very passionate about Lean, some of the classic DevOps books, The Phoenix Project, The DevOps Handbook, and then also design thinking. We draw inspiration from all of these things. If you'd like to talk in depth about that after, I will geek out about that over a beer with you for a long time. But you will hear us mention these things throughout.
And so one of the things that I think every organization struggles with is: how do you start this DevOps transformation? What is it? And I think one of the things that we started with was converting from Waterfall to Agile. So I think there's probably still people, it sounds like from the sessions, that are still working in Waterfall. So if you're looking at how to start, that's a great first step. It would really be hard, I think, to go from Waterfall right through DevOps without kind of transitioning to Agile. So that's kind of where we started.
And we're going to be talking about how do you persist. I really loved Scott's slide yesterday that showed a dirt road. It's not a paved road. It's a bumpy, rut-filled, tough road to kind of keep persisting. So we'll talk about some of the challenges and bumps we've hit along the way and what we do to keep moving.
We're also going to be talking about starting small, and we've been hearing the same theme of kind of the domino and the tide coming in. You have to really celebrate those wins along the way, and then you can see your progress. So we'll be sharing some of those.
Suzanne Nielsen
Yep. And this is actually a sand road. You get a chance to look at some of my vacation pictures along with our talk.
Sarah Shewell
Okay, here is a little snapshot of kind of the timeline of what we'll be covering today. Suzanne started with a Scrum transition before I came to Starbucks, so she'll be sharing about that and we'll talk through the timeline.
Suzanne Nielsen
All right. So I already mentioned a little bit that transitioning from Waterfall to Scrum to Agile is a big change for people that have been working in Waterfall for 20 years, for example. So it's important as a leader of your team and your organization to make sure that people on your team know that this is a difficult transition. They're going to be learning, they're going to be trying things, they're going to be failing, and that's okay. So it's really important as a leader to support them through that.
Another vacation picture.
One of the mistakes I made early on when we were going to start doing Scrum is I said, "We're going to try this. We're going to try Scrum." And so after a few months of trying Scrum, people are like, "We don't really like this. This is kind of hard. It's challenging. So we want to go back to the old way of doing things."
So that would be one thing I would just say to all of you, if you're leading teams, your language and the way that you lead your team is important. They're going to listen to you. So make sure you commit fully and stay the course. Don't say, "We're going to try this."
And then a few of the things that really helped us transition from Waterfall to Scrum and continue to help us on our DevOps journey is getting the team co-located. So before we had analysts in one spot, developers in one spot, testers in one spot. And when we co-located and organized more by product and brought the teams together, kind of the way you've been hearing, full stack, that really made a big difference because they could stand up, right, and their cohorts are all around them, and they can collaborate all day long.
The other two things are around leadership. So it's important that you have, you've been hearing this all along, again, transformational leadership. Having leadership all the way up at the top that supports you and understands what that is, and also the other managers on your team. So it's a learning journey, not just for your team, but leadership around you at all levels as well.
And then the other thing Courtney mentioned a little bit, my aha moment: it was shortly after she started, and I came into her office and I was like, "Half of my job is gone. What do I do?" So that was a little bit of a risky thing for me to do, I felt like.
If you're leading a team and you switch to Agile and Scrum, you get a product manager, you have your team that's self-organizing. So before, if you were the one that were working with the business to understand priorities, do your intake, and then to delegate out to your team and kind of manage and keep status tracking and things like that, that changes with Agile. The product manager is really doing that intake, and the team is self-organized, and they work on their work.
So as a leader and a manager, again, it's really important to know that that's okay. Your job is going to change. And what I did is, like Courtney mentioned, is like, what do I do next? How can I learn the next things on our transformation to help support my team and help set them up for success going forward?
Sarah Shewell
And I actually made us use this photo because I love it. It's in a value stream session, and the people sitting there are writing down the tasks that they do as part of their job, and several of those tasks that they're writing are things that used to be responsibilities of Suzanne. And I said, "Look, your hand's not even on a Sharpie for this." They're writing those tasks. They own it, which I love.
And then really it was like, again, our first step was Agile, and then next was DevOps. And how do we teach the team? How do we become coaches and remove roadblocks for them?
All right. So when I arrived, they'd been doing Scrum for over a year, and we practiced retrospectives. We were talking about what happened in the sprint, and there were a couple themes that occurred over and over. I helped the team see there's a different way to do it.
But one of those problems was that production and business priority continually interrupted our sprint. And these things were not just because priority is changing, all of that. They were truly things that we needed to do to bring the most value to the customer. But the team was continually feeling like they failed because they weren't making that commitment. So we had these bookends on a chunk of work and an artificial timeline that really wasn't helping anybody.
So these emergencies would come in. We said, how could we plan differently for that?
And the second one here, mostly when you see containers, you get excited and cheer. This is to reference the piles of work that we had injected into our system. So the definition of done for the Scrum team, we had downstream testing teams. They would hurry to deliver this work to the testing team who was still testing something else that they had delivered a couple weeks early. So we had relearning that was continually happening. We were having teams push work to other teams where they were not ready for it.
So we really wanted to reduce that aging work, just the cognitive overload of how many things you need to be thinking about at one time. That was another reason for breaking that paradigm of the sprints.
Which leads us to the next phase. So we decided to keep the majority of the ceremonies that happened with Scrum, which was a good baseline. So we still knew we wanted to break down our work in some of the same ways. We wanted to reflect and pivot and change. But we were no longer going to make a commitment over a period of time.
So we said we will take in one story, we'll take in our bugs and reserve capacity to handle production at any given time, and we're also going to work on technical debt. We'll keep that limit very tight, and we will work in that way, and it takes us as long as it takes us, and we want to be consistent in that delivery time.
Suzanne Nielsen
And this is also about the time that I came to the DevOps Summit last year. So I had learned a little bit about DevOps. I came to the conference, I was like, wow, yeah, we have to do this. So it really energized me to help, again, lead the team and remove the bottlenecks and the pain that the team had been feeling.
Sarah Shewell
So right about this time, we knew one of the things that we needed to do was share with the team, here's the start of our journey. We needed to do a value stream. And I've been part of many value streams before, and I think it's very important to go into that with the mindset that this is not a one-day workshop where I get lunch provided for me and then I go back to my regular job.
So I had one morning when I couldn't get Confluence working, and I was meeting with Courtney, and so I grabbed a Sharpie and drew out this roadmap really quickly. So that starts with completing our value stream, and then we wanted to show that we'll be refining that. We will be moving into documenting what that is. Line balancing to understand where a bottleneck is. We'll be using data from the value stream to get that. And then our continual cycle of PDCA, A3 problem-solving, and all of the just do it. I love the JFDI that we heard this morning. That was amazing.
So that's going to be on there now. But this was to show the team that this is not a one-day event. This is a multi-year journey, and we're sticking with this. And I think the most important thing at the bottom is we are never done. We're continually improving. This is a road that we are getting on, and we will not be coming off anytime soon.
Suzanne Nielsen
And this drawing immediately became famous in our work area. It's posted all over the place, and just, again, it kind of puts: you have some context of where are we in the journey, and we're going to be here for a while, and it is a long transformation journey.
Sarah Shewell
Okay, so this is a snapshot of that actual value stream. So we had some finger-pointing happening between teams. We had those handoffs that were distinct, and people not understanding what activities were happening. You have your macro-level value stream, which contains your process-level steps. So I do intake, I develop the thing, I package the thing, I release the thing.
This is at a micro level, so it takes all of the steps that are accounted for in every one of those processes to understand what's happening, who's doing it, how much time does it take, how much lead time do you have, what's your percentage of accuracy on that? Those are all things that were considered as we were putting this together. And as you can see, this is about half of it as I was laying it out. It's pretty massive.
Suzanne Nielsen
And it was the team. We did a whole-day session, and we had all the different groups that worked in our value stream. So our product owner, analyst, technical product managers, developers, testers, deployment team. So everyone came together. We had this all-day session, laid out the value stream, and it may be kind of hard to read in the green, but we had 187 steps in our value stream from where we took the request in from the customer to deliver it out to our stores. So that was a lot.
And we had, Sarah mentioned the distinct handoffs. We had eight handoffs from different groups to get it out the door. And our cycle time, it took 84 days to deliver to the store. So it was pretty interesting, and the team saw this, and one of the things I would say to watch out for is when you spend time on this and the team sees this, they're like, "Wow." They see so many things that they can improve right away. It's really important to step back and do what Sarah just showed on her diagram and do some scientific approach to problem solving and look at where is your bottleneck, that long pole, to help you focus in on what you want to change first, because there's a lot of things that you could change.
Sarah Shewell
I think one of the other things that's important to note is because teams were working in sprints, they had the impression that they were delivering customer value every three weeks. They weren't actually delivering something to the customer with that frequency. So to get that number of, you work on that for three weeks, and then you pass it to these successive teams, it's 84 days before we get a single code change to our customers, was shocking to the team, and they immediately wanted to improve.
So you go to this workshop, you see these numbers, and you just really want to get in there and get to work on this. You have a lot of momentum leaving the workshop, and then just like we had with sprints, production happens. All the things that happen with normal life start happening again. So this is where it's so key to go into that session with a plan of how you will sustain it. How are you going to keep giving that message to your team over and over and show that you're improving?
And this is one of my favorite quotes from my peer, Liz, and it says, "All improvement requires change, but all change is not improvement." This is why we measure.
And this is with our PDCA board. So every change that we put into the system, we don't want complete chaos. We said you can try any change you want, within reason, as long as you have ownership of that, it can be done hopefully within two weeks. But you need to say what you think is going to happen, make sure you come back, and act or adjust accordingly. So we do track all the changes in our system so people have awareness.
Suzanne Nielsen
And then also the important thing is, after your team goes through that all-day session and they get all jazzed about working on improving the system, it's important as a leader to make space for your team. So I had set up two hours every Wednesday afternoon for all the different teams to come together that were in our value stream to look at, where is our bottleneck? What are some experiments that we can do to make progress on that? So again, setting that space up so they know this is... And I didn't end it. It's like from now forever, the meeting series goes. So it's a learning journey. We're going to continue to learn and improve.
Sarah Shewell
Yes, and just bringing back the books that you might have gotten last night or in some of the sessions and say, "Hey, go read this," that might not be very successful. So make sure you make space and make it part of their regular work.
Suzanne Nielsen
Yep.
Sarah Shewell
Okay, and so with our new rule of WIP, I just want to clearly call out having a WIP limit is painful. It exposes those problems. It's really easy to keep yourself busy all the time and move to another task. But if you don't hold tight to that WIP limit, you're not showing where you have gaps in knowledge. You're not creating T-shaped people, and you are allowing everybody to do their own thing.
I was asked probably daily for three, four months after we transitioned to this, "Can't we add another story in? Can we add another story in?" No, no, no. We're going to stay the course. We're committing to this. The team all now has seen the results and the numbers, and they fully realize that now that we have slowed down, we've actually sped up in our delivery.
Suzanne Nielsen
And really, the key thing is that everyone now swarms on issues. So instead of running into a problem, today I go work on something else. Now the team swarms and gets us through that problem. So we're actually moving through stories more quickly because everyone's swarming and getting it resolved. So it's great. But it definitely was a few months transition to do that.
Sarah Shewell
And here's that journey map again. So we sped through those first four steps in about five months. Now is the tough part, when we talked about the ruts and the bumpy part of the road. We will be in experimentation phase for the next several years. We know where our bottleneck is. We are continually working through that. So it's really important to, again, hold that time sacred and keep people motivated and celebrate those wins along the way.
So the next piece that we did that I highly recommend is the DORA report, which is the DevOps... oh, shoot.
Suzanne Nielsen
Yes. Research and assessment, I think.
Sarah Shewell
And so this piece, you have your entire organization or team take a survey on how you're doing, and you get a comprehensive report back. We were really excited about this to understand how do we measure up internally and team against team, how do we measure up industry and then against top performers in the industry.
The thing that I would tell you is that you can read The DevOps Handbook, and you have some really good examples. As I mentioned, we're early in our journey, and the gap between what you read in the handbook and what our reality is, is very wide. So it's nice to get some very specific feedback of here's what you can be doing.
DORA will tell you where to start focusing, but not the how. And I think you need to go into it knowing, okay, we're going to get all this information, which will greatly help us, but we have to be ready to jump in and create that plan of how we're going to act on it immediately.
Suzanne Nielsen
Yep. And I would just echo. When I read The DevOps Handbook, I was like, this is great, but the gap is large. We are here, and we want to be here. So the DORA assessment really helps you focus on, oh, these are the four or five big wins that we can focus on and make progress, because there's a lot.
Sarah Shewell
And so one of the things that we would mention as kind of a failure bow on this is we had the results, and we didn't have that plan of how we were going to move into this right away. And that slowed down a little bit of our momentum. So people were working on their experiments, and I think this is part of the pain of when you're doing very small pieces of things. You could be making all kinds of changes to your system, but if you don't have those aligned to a higher goal, you don't know that you're actually making traction in the way that you want to be making traction.
So we decided we're going to just fit all of the work that we're doing every day into the same priority of our DORA results. So we had five things that we need to work on. Top one was automation. That was not a surprise. Working on modernizing and decoupling our architecture, improving our release management, and modernizing that. So all of these things were things we knew that we needed to work on, but it gave us a very clear priority of what to tackle first.
Suzanne Nielsen
I think one of the things that I did too with my team is we looked at this, and we said, again, making the space also make it part of people's goals. So I had things in my goals that were around the transformation of what we wanted to do around cycle time and improvements, and people on my team had that as well. So it just helped get more accountability and buy-in. It's like we're on this journey together. We have goals around it, and let's go forth.
Sarah Shewell
And another thing to plan for is opportunity fatigue. So we came out of our value stream session. We have all kinds of opportunity things that we can work on. We have DORA. We know we're not measuring up in some areas, even to industry standard. We have a pretty far gap in some of those things. It can be a little defeating and frustrating.
And so we didn't see quite the action we expected to see. One of the books Suzanne and I read at that time, based on Courtney's recommendation, was A Sense of Urgency, which really helped guide us and talk about how do you make that emotional connection, because it can just be overwhelming to know the amount of gap that you need to cover.
This brings us into single-piece flow, and that is where we currently are today. So we expected through some of the experimentation, I thought the team would say, "Let's remove some of the handoffs." The handoffs were all distinct teams. So making a recommendation as an IC that says, "Let's go ahead and disband this team or kind of work in a different way," was a little bit too far of a leap.
And there was a topic at table topics at Lean Coffee yesterday about how do you decide when you just make a rule or how do you let the team kind of get there. This is one of the things that we forced a little bit, following Model Line in Toyota. So we said we're going to take one of the teams, and we will have no handoffs that occur on that team.
And what they're going to be doing is following the same value stream that we have. So they're going to validate every one of those 187 steps. We're going to go through that, and if they don't need to do it, that's great, but they're going to test it. They're going to go through every one of those. And we knew going into this there would be a big learning curve. So we said, "You're going to fail at times. We're going to make a safe space around that."
And this is the team.
Suzanne Nielsen
Yep. And so we took a part of my team and spun it off to be our experimental Model Line team. So we had, I think Marie's sitting here in the front, three people on the team that we were going to experiment with. And then we had other related teams that were helping. So a product manager and agile lead, Sarah. Integration testing lead, Pat, he's here too. And a release manager.
So we tried to make the Model Line team small, but knowing that they were going to have to learn, since they're not doing handoffs anymore. They're not handing it over to QA. They're not handing it over to deployment team to deploy. They're going to learn how to do that themselves. So we had a lot of support from other folks on the team. It was painful, a lot of learning, things that they had never done before. But it was great to have the support, and the team came together and did that. That was their goal, the no handoffs. So it's pretty exciting.
Sarah Shewell
There we go. And here's some of the learnings we've had so far. So we started this in June, so it's only been a few months that we've been working. And I'd say it's a real roller coaster. I know both of them, Marie and Pat, would be happy to talk to you about that.
But they have increased their speed greatly, which we'll share those numbers in just a moment. But getting to that point has been very painful, about learning all these roles and constantly learning every step that you're doing. Okay, I don't know how to do this, now I've got to pull somebody in who this has been their job for a long time and have them teach me.
So I think if you do something like this, make sure that you have awareness and you don't just celebrate all the wins that they've had, because there's been some little tension between other teams, but show and recognize all of the pain that they have to go through to get those wins, so that everyone doesn't go into it thinking, "This is just going to be magical when I move to this model."
Suzanne Nielsen
And so here are some of the results. So we started with 187 steps in our value stream. With our larger team, we've been able to reduce that by 40%, down to 111, and with our Model Line team, down to about 50?
Sarah Shewell
Yeah, we're definitely under 50.
Suzanne Nielsen
So that's a huge reduction in our value stream steps. And then we reduced our handoffs from eight down to four with the larger team. But again, the Model Line team took it down to zero, so no handoffs from Model Line team. And that's resulted in our cycle time going from 84 days, reducing 74% down to 22 days. So that's pretty incredible.
Sarah Shewell
Yes, and this is where you really need to celebrate those small wins all along the way. As you're in that moment, it can feel like not a lot, and honestly, till I put the slide together for this, I had one of those moments where, oh my gosh, this is huge. We've reduced this a huge amount. And as you can see, we'd love to get to daily. We're doing this multiple times a day, but 74% reduction in a year is pretty great.
And our next phase. So this is what we need help with. This is our next phase. We know that we have a lot of opportunity. Now that we have removed some of the bottleneck around our testing, we have a bottleneck in our deployment. We have got to get to CI/CD and open that piece up.
So we are really focusing on those DORA goals. They are present. We talk about them every week. How are we measuring up on our experiments towards that? Continuous learning is super important, too. So as we're approaching this, a lot of the people on the team have not done this.
We have Jen Henry and Caroline that are in our group, and they are putting together an enterprise-wide DevOps learning journey, which is an intensive three-month program where we have hands-on teaching. You can learn Kubernetes in that. You have a cohort that helps you from a broad group across the building. So many of our team members are starting that, and we're starting to spread the knowledge and the DevOps community.
Suzanne Nielsen
Yep, and also along with that, there's a leadership track as well.
Sarah Shewell
Yes.
Suzanne Nielsen
So it's the teams and the leadership on the learning journey.
Sarah Shewell
So what's next for you? One of the things that we've done since, I think, last year, and the work that we've done this year is find a community on Twitter, have some DevOps book clubs, attend meetups in your area, find a buddy here. Just connect with different people in the community. The DevOps community is an awesome group of folks, and everyone's willing to help, and everyone is learning. As you've been hearing, there's a lot of common themes.
Suzanne Nielsen
Speak at a conference, like this.
Sarah Shewell
Speak at a conference, yes. Come, and it's a good learning opportunity. We have just a few seconds left, but I'd reiterate from Scott's talk yesterday, and hopefully what you've learned from ours, you have everything you need to start this journey right now. You're sitting next to somebody right now that can help you on this journey.
We would love to connect with you. So we will continue sharing ways that we persist as there's challenges, and remember, in the moment, celebrate every one of those small wins.
Thank you.
Suzanne Nielsen
Thank you.