Zack Ayers and Joshua Cohen

Hey, everyone. Welcome to Andon Cords. If you're here to learn about Andons, you're in the right place. Thanks for sticking around to the very end, I think the last breakout session of the day.

What we like to do to start our stand-ups at Excella for some of our teams is give each other a fist bump and a smile. So we're going to do that right now. We're going to start this off with high energy, so look around you. Give somebody a fist bump and a smile.

Okay. We're from DC, so I'm excited that the Nats are playing in Game 7 tonight. So, there we go. I got a Nats fan back there. Cool.

My name is Zack Ayers. I'm a Scrum Master at Excella. We're an IT consulting firm based out of the DC area. We're in Arlington.

I'm Josh Cohen. I'm a senior developer at Excella, and Zack's going to go over our agenda for today.

So we're going to go over a couple different things. We're going to kick off talking about our culture of experimentation and how we constantly look for ways to be a learning organization, a learning project. We're going to talk about Andons, what they are, how we used it on our team, and we're going to go into a culture of psychological safety and why that was important to us.

There we go. Cool. So it's important that we give you guys some context of our project. Everything that we're talking about is in the context of us working within the federal sector. Our project was with USCIS, and we were a Scrum team of about six to eight folks back when the project started in March 2016. We had a product owner, Scrum Master, and we were having a lot of success, so they staffed us up a little bit more. We doubled in size, and it was at this time that we split into two different Scrum teams. We were working within a single product backlog. If you're familiar with LeSS, Large Scale Scrum, that was the scaling model that we were working in. After about a year and a half from when we started the project, we grew even more. So we're at three teams now, three different Scrum Masters, still working within the same product backlog.

We're not going to go too much into the structure of our team, but it's good for you guys to know this as we talk about one of the experiments that we ran at the product level.

So that was a little bit about our history, but we also wanted to go over some of our accomplishments during that time. The first one is our ability to deploy frequently. We deployed over 10 times a day, and this allowed us to incrementally release features, and that way we could get feedback from our product owner or our stakeholders as soon as the feature was done. We didn't have to wait two weeks, a month, or sometimes even longer for a release to happen.

Next is our ability to get code into production quickly. So if there was a bug that we needed to put in a fix for, we could take a code commit, have it go through a full suite of tests, and have it live in production in 20 to 30 minutes.

And then the last thing I want to highlight is our ability to acknowledge incidents within one minute. By being able to acknowledge incidents in such a short time span, we could come to a resolution faster.

So why am I mentioning these things? The reason for that is because we think the reason we were able to accomplish a lot of those golden metrics is because of our culture of experimentation.

Here you have a celebration grid that I'm going to run you through a little bit. On the right, you have practices. These are things that your team has decided you should be doing and generally have successful outcomes. What's an example of that? This isn't development-related, but say you have some savings and you decide to invest that. Generally considered a good practice and hopefully will have successful outcomes. But with any investment, there's risk. Maybe it might end up being a failure because the market crashes or something.

Then on the flip side of the grid, we have mistakes, things that we've decided are things we shouldn't be doing and tend to have negative outcomes. So instead of investing that money, you go downstairs to the casino, and you decide to bet all of those savings away. I'm not going to go into the odds of the different games, but generally, we consider that something you'd want to avoid. But there is a chance you could be very, very successful.

The point of this is, for practices and mistakes, these are things that we've already decided we should and should not be doing, so there isn't a lot of learning that happens there. Where we try to focus our experiments is in the middle, and that's where we have just as much of an opportunity to succeed as we do fail. That's where we also maximize learning, obviously.

During one of our team retrospectives, we noticed that our cycle times were starting to climb, and we had a case of what we called the "almost dones." Tell me if you've ever heard this before, but during stand-up, our developers would give an update on the feature they were working the previous day, and they said, "Hey, I made a lot of progress. I'm almost done." And the next morning they say, "Hey, I ran into some issues, but I worked through them. I just have a few more tests to write. I'm almost done." This would happen way too frequently, and we decided this is an area we wanted to improve on because our teammates were only bringing these issues up at certain times, like stand-up, for example, and we wanted our team to collaborate when these issues were realized, not at ceremonies.

This team formed in September of 2017, and it was at that time, one of the first retros that this team ran, where they said, "Hey, we want to be a highly collaborative team. We want to have great communication." And they started working on different experiments that they could do to foster a good culture of collaboration and communication.

As Josh said, we started noticing our team had the case of the almost dones, and none of the experiments that they were working on really seemed to land and resonate with them. So as their observer and as their Scrum Master, I had heard about this Andon cord, and I felt it was important to share the knowledge that I had with the team. What I'm going to take you through is an abridged version of a presentation that I gave this Scrum team on Andon cords. I really want to highlight that I didn't tell them what to do. All I did was present the information, the knowledge that I had, and at the end of my talk to them just said, "Hey, what do you guys want to do next?"

So what we did was we started with, what is an Andon? Before we talk about what an Andon cord is, it's important that we understand the Andon. What an Andon is, is it goes back to a Japanese paper lantern. It literally means a light.

The Andon cord was made famous in car manufacturing, specifically at Toyota. At Toyota, they have a string along their assembly line, and when a frontline employee noticed a defect in the process, the Andon cord would be pulled, and the Andon, the light, there would be a light that would light up on a light board. What that does is, it signals a manager to come check things out. A group of individuals will swarm together, and that conversation will lead to a path forward where they work together. What made this culture unique at Toyota, whereas at some places when there's a defect noticed, you might have to write a long report about a path to resolution, at Toyota, there was a sense of gratitude where the manager, one of the first things they would say is, "Hey, thanks for pulling the Andon cord. What can I do to help you?" And so they would work together to identify a path forward.

The team thought this was really cool, wanted to try this out, and what did we do? We decided we're going to have our own Andon cord. We weren't going to have a string along our desk to pull, but we were going to create some kind of metaphorical Andon cord for our team.

We wanted to make sure that when the cord was pulled, everyone on the team would stop what they were doing and then work together to identify a path forward. Why would we pull the Andon cord? Yes, if we noticed any defects, but also we wanted to make sure that we were improving our collaboration, our communication channels. If somebody felt stuck or needed the help of the team, they would pull the Andon cord then.

With our experiments, we want to make sure that we're measuring success. Things we were looking for were an improvement in cycle time. We wanted to make sure that we were improving our collaboration. Also, as Josh referenced, we didn't want to have the case of the almost dones. Instead of planning our collaboration where we would wait for meetings or stand-ups or retros, we wanted to make sure that we were talking about things as they came up.

Here's what we did. I don't know if the video's going to work or not. If not, I can talk through it. You can give us a thumbs up or a thumbs down if it's going to work. Down? Okay. All right. So I'll talk through this.

This is our Slack channel. What would happen here is somebody would type "Andon." When "Andon" is typed, you would get an @ here in the channel. You can see the @ here: "Andon cord pulled." People would get notified. That was how we notified folks in Slack. But we didn't just end there.

This is a picture of our room. I'm bummed that the video's not working, because we have a dancing tube man at the end of this. But we had an If This Then That integration with Slack. What would happen is when Andon is typed into Slack, it would trigger If This Then That to some Amazon smart plugs that would turn on this rotating red light. String Christmas lights would be flashing on and off, and then we had one of those dancing car salesman tube men just going crazy in the office. It was fun. When somebody pulled the Andon cord, it was a fun experience, especially early on. There was a lot of joy that came out of that.

I talked about how we wanted to make sure that we were measuring our success and visualizing our data. This is when we began the experiment on May 25th of 2018. This is what our cycle time looked like for the three weeks after we began our experiment: hovering right around three days in progress.

This right here visualizes our Andon cord pulls per day. After about the second week we were running this experiment, we were averaging about 0.4 Andon cord pulls per day. If you want to take a picture of this, I promise I got a cooler graph to show you.

This is the same data, just we're about to look at it at scale. We were pulling the Andon cord a lot early on, but what happened? In July of that summer, about two months later, we stopped pulling the Andon cord. But we noticed that because we looked at our cycle time. Our cycle time had hit an all-time high. We were looking at 11 days in progress.

Scrum's an empirical process. We inspect, we adapt. So we asked ourselves what was going wrong. Why weren't we pulling the Andon cord? We realized we like pulling the Andon cord. It was fun. We had a tube man go nuts in our office. That made us laugh, and it was great, but we weren't pulling it that often. It was because, one, I'm going to pick on Josh because he's here. If Josh was working on a story that he felt should be simple and everyone else thought it might be simple, but Josh felt stuck, didn't know the answer, he didn't want to necessarily reveal that he didn't know what the solution to the problem was. Another instance would be Josh is working really hard and sees that everyone else on the team is heads down. They're either pair programming, mobbing, and doesn't want to interrupt and say, "Hey, what I got going on is more important than anything you guys got going on, so I'm not going to interrupt what your workflow is."

So we talked about that and decided we were going to change when we would pull the Andon cord to be whenever we need the opinion of the team. One of the things I love about this group is that we really enjoyed spending time together. We would go get lunch, we would go to impromptu happy hours. Every week, we would talk about what our high for the week was, what our low for the week was. And so we started pulling the Andon cord for non-development-related things in addition to what we had identified early on.

Here's what happened. Again, before you see the next part of the story, remember, we began to pull the Andon cord for non-development-related things. So Andon cord pulls go up, and there's correlation there. Here's the rest of the story.

You can see how the Andon cord pulls and the cycle time are a leading indicator for each other. You guys can take a picture of that if you want to. It's a little bit of a heartbeat. We didn't get it perfect all the time. We had to continue to remind ourselves the importance of pulling the Andon cord and the importance of collaboration.

You can see in late December, early January this past year, we hit zero again. We moved offices, it was around the holidays, and so we said, "Hey, let's incentivize ourselves for pulling the Andon cord." Joe, one of the devs on our team, we decided we were going to pick on him a little bit and create a team currency. It looked like a Monopoly dollar. When somebody pulled the Andon cord, they would get a Joe Buck. The person with the most Joe Bucks at the end of the month got a traveling trophy. We did that, and you can see we started pulling the Andon cord a little bit more again. It was fun.

Same data, just shows it in a table form, how we started out with three days in progress with our cycle time and went to about 11 days when we stopped pulling the Andon cord, and then the happy ending when we started pulling the Andon cord more often and cycle time went to two days.

One of the big things for me as Scrum Master is looking at morale for the team. Before we started the experiment with the Andon cord, our happiness at a team level started to decrease a little bit. I was hoping that by doing the Andon cord, our happiness would increase. You can see when we kicked off the experiment, it was a fun project for the team, and they seemed just to be a happier group of individuals. Not only did it improve our work productivity, but it also improved the team's happiness at work.

Josh talked about the celebration grid. We decided as a team this experiment was actually a good practice. It was a good thing for us to do. So we moved it from the experiments column over to the practices column. It was a good practice. Why did we do that? It improved our cycle time. We saw greater psychological safety with our collaboration, and we continued to inspect and adapt our process. I talked about how we didn't get it perfect right off the bat. We had the Joe Bucks that we were doing, and we had a couple other experiments we did on top of the Andon cord just to continue reminding ourselves of its importance.

Joshua Cohen

Zack just mentioned psychological safety. If you went to Dustin and Nicole's talk on Monday from Google, they talked about how psychological safety was the number one element for high-performing teams. We feel like the Andon experiment helped bring psychological safety to our Scrum team. There was no longer a feeling that Zack mentioned earlier of, if we had new team members, they weren't worried about bothering the rest of the dev team, or if we had some teammates who liked working more independently, this helped encourage them to collaborate more.

I think what's so great about psychological safety, though, in addition to this, is not only does it help improve our development work, but I think it applies to everything we do. We found that teammates started coming up with creative solutions or improvements to our Scrum ceremonies. I think because we had psychological safety and they knew that this was a safe space to present their ideas to the team, they were starting to think about those things in the back of their head. Whereas opposed to the opposite, if this didn't exist, they wouldn't even try to think of solutions.

Up until now, we've told you the background of what an Andon cord is and how we used it at the team level. But there's other ways you could adopt it on your teams. We want to present how we used it at the product-wide level. When we used it at the product-wide level, that was reserved for what we called code red incidents. That was when production had an outage or something that was critical that was similar to that.

When the Andon cord was pulled at the product-wide level, everyone stopped what they were doing, and they tried to figure out how they could help. This was really beneficial for us because this gave us the key personnel we needed to resolve issues. If somebody was familiar with an area of the code base that was causing problems or had certain subject matter expertise that we needed, they were then available for these critical issues. Once we assessed what the problem was and started working towards a resolution, if you weren't necessarily needed, you could go back to what you were doing.

Zack Ayers

We're here at Gene's conference, and so we're going to talk a little bit about the Three Ways of DevOps and how we were able to tie that back to our team. As we were discussing what we learned from our experiment, we talked about the Three Ways of DevOps. For us, we looked at how we were able to improve our flow, specifically the cycle time, but more importantly, how you're able to visualize that with your teams and with your organizations is what made it really powerful for our team.

We have a physical task board where you see stickies going across the board each day. We visualize our cycle time in our shared office space. We also visualize our Andon cord pulls per day. At every single retrospective, I would have a printout for the team of that graph that I showed you guys earlier of what was our correlation between Andon cord pulls versus cycle time on a sprint level, on a monthly level, so on and so forth.

The Andon experiment helped shorten and amplify our feedback loops. Instead of waiting for daily stand-up to present your issue, you were reaching out to the team and getting help once those issues were realized.

What's most important to us is the culture that we have of continual experimentation and learning. Like I said, when this team started in September of 2017, we started trying to run experiments to address collaboration, communication, and when nothing seemed to land, we didn't give up and wave the white flag. We continued trying to run different experiments and chip away, and finally land on something that was impactful and meaningful to our team.

Let's give you guys some advice on bringing this back to your teams and your organizations. Five pointers that we have. The first is coach. Use the celebration grid. You can find it at management30.com. That link is also in our slides if you check those out. That celebration grid is from Jurgen Appelo, and you can check that out. Coach your teams on the importance of the celebration grid and experimenting, and that equal chance of success and failure and the power behind that.

Then teach your team. Educate them on what an Andon cord is, and what are the different ways that they could potentially use them on their Scrum teams.

Just as importantly, ask them if this is something they're interested in. We thought this was a really cool idea, so we decided to implement it, but if it was forced upon us, it wouldn't have been as successful.

You saw after two months of us running this experiment that it crashed to the ground. We weren't pulling the Andon cord at all. Our cycle time was at 11 days, and by all standards, it felt like a failure. But our leadership at our product level and at Excella has given our team the freedom to feel empowered to fail and fail often. Having that freedom allowed us to inspect and adapt and iterate on our experiment to create something that was more powerful and more meaningful for this team that had an amazing impact.

Finally, we had a lot of fun with this. We start every single day with fist bumps and a smile. We have a dancing tube man in our office with a police light and Christmas lights that go on and off. This is, yes, meant to call the attention that somebody needs help or somebody just wants to talk about something, but it's also supposed to be fun. For me, my vision of the Andon cord was to have something that was as obnoxious as possible. I would love to have an Amazon Alexa that starts saying, "Andon pulled." Something ridiculous, and it's supposed to be silly. It's supposed to be fun. Just trying to create joyful moments in your office place.

We haven't figured this out perfectly. We still have some problems. Josh and I are on a new project now, but some of the issues that we were facing at the end of this project were, one, pulling the Andon cord on a teammate. We felt like that was the next tier of psychological safety that we were striving for. So if Josh hadn't pulled the Andon cord, and I get the sense that he's stuck on a story for a couple of days, as his teammate, I should be empowered to pull the cord and say, "Hey, I think Josh is stuck. Let's talk, and let's make sure Josh gets the help." What that speaks to is a level of team-driven accountability instead of individual or top-down accountability.

We would onboard teammates. You can see that we continue to scale, and so we were constantly bringing on new teammates to our team. Introducing them to the unique culture of the Andon cord and just our culture of experimentation, getting that buy-in, and getting them to accept it as something that was just as important to them as it is to the greater team, that was a challenge for us at times. But ultimately, it worked out.

Finally, Josh and I, we're on new teams now, and so we're going through the grind of that five-step process that we laid out of coaching, teaching, asking and not telling, providing freedom, and having fun. We're trying to sell that to a new group. Different personalities, different individuals, different backgrounds. Getting them to see the importance of experimentation and maybe even the Andon cord is something that we're struggling with right now.

We're really proud of this experiment and want to share some team insights with you. These are actual members of our team. Jen is actually one of the leaders on our project. This is a Slack that she sent us before we came here.

She spoke to the happiness metric that I referenced earlier, where she saw the morale boost from the team working together. She said that it offered an outlet for all types. The inspiration of the Andon cord was just what the team needed to become more reinvested in the project overall. The code red Andon that Josh spoke about would not have happened if it wasn't for the leaders that emerged from this team. Not only was there the code red at the product level, but the other two teams within the product adopted their own type of Andon cord. That happened because the same members of this team, whose happiness was declining when we began this experiment, had an opportunity to show some leadership at the project level.

Now, let you guys hear from the members of our team. Andrew talked about how he was surprised that the team embraced the idea of reacting positively to all the Andon cord pulls and not being bothered. I'm sure it's a question that you all are thinking right now: if we introduce the Andon cord, are people going to get ticked off that they're being interrupted? But you'll see in some of this feedback that the team actually wasn't bothered by it, and wanted to help.

Brian talked about how using the Andon cord allowed the team to identify and resolve critical issues and blockers much faster. Drew spoke about how the lights were just distracting enough to get his attention.

We're excited to hear from Matt that the team members, again, didn't mind being interrupted in order to help a fellow teammate. Nick saw an increase in collaboration, and Andy agreed. He saw a higher level of engagement from communication. There was more pairing and swarming that happened as we pulled the Andon cord, and we were responding to incidents much more quickly.

We reached out to everyone through Slack, hoping to get some feedback about how they felt about the Andon experiment. We love Joe, but Joe's not very good at checking Slack. So we actually pulled the Andon cord, and Joe noticed all the lights going off. He asked, "Hey." He stopped what he was doing, and then asked how he could help. So we told him, "Joe, you're not checking Slack, but we're actually looking for feedback on the Andon experiment."

He told us that he knew this would never work, but then he admitted that he was wrong. It did. We had data to back it up. He found that the team was more engaged, problems were addressed sooner, and productivity was higher.

At the end of the day, this is just our experience with a cool experiment. There's probably a million different ways of doing this. We've used Slack as our communication when it's not face-to-face. This was something that worked for us. We would love to hear other stories of folks adopting this experiment within their own teams and continue the conversation with you all about issues with collaboration and communication you guys have within your teams and organizations. If this is something that you guys want to try out, we would love to hear your stories.

I think we have a couple minutes left if anyone has questions.

Q&A

Audience: You saw through your experiment that cycle time was reduced. Did you also find an increase in quality? I saw your cycle time reduced, but did quality increase as well, post-production?

Zack Ayers: Yeah. I would say the quality didn't decrease. It wasn't like because we were working through things more quickly, we saw more bugs coming through our system. The nature of the work evolved as the project went along, and so for us that would've been a hard metric to look at because sometimes we weren't always deploying or we weren't always working on something that was going into production and being interacted with users.

Audience: Your code red Andon, how big was the product team?

Zack Ayers: The product team? I would say at any given point in time, we could have 25 to 30 people at the product level. What would happen when there was a code red Andon pulled is everyone would read the Slack and stop what they were doing, but the code red conversation would offer context in Slack as to what the incident was. As Josh talked about, the Andon helped us deploy the right personnel to figure out the issue. It just really helped you kickstart those conversations.

Audience: I noticed in your data set there were a lot of ups and downs in the pulling of the Andon cord, and you said it might've had something to do with fear sort of reestablishing itself. Did it ever actually stabilize and just become a regular behavior? And if so, if not, why not?

Zack Ayers: What you're seeing is from the beginning of the experiment in May of 2018. This team was no more in May of this past year. We scaled down the project and are now onto different projects. So that's just the end of the-

Joshua Cohen: Yeah. And so it was up and down the entire time. That's kind of like that next level of psychological safety that we were talking about, that we felt that if we were able to start pulling the Andon cord on each other, that maybe it was going to stabilize, but we never got to figure that out.

Zack Ayers: Even if we stop pulling it, we knew it was a good thing, and we decided it was a good practice to have, so we try to find ways to bring it back.

Awesome. Well, thanks so much. I think we have a break after this, so yeah. Thanks.