Jeremy Castle

I'm Jeremy Castle. I'm a director at State Farm, and I have Kevin O'Dell with me, who's also a director. Today we're going to talk to you about State Farm's DevOps journey. I'm going to talk about all the great stuff we've been doing to enable DevOps, and then Kevin is going to come in and talk about the reality of what it took to actually implement it at State Farm.

To give you context on the size of State Farm, we're number 36 on the Fortune 100. We have about 19,000 agents and 58,000 employees, and we're based in four locations. Our headquarters is in Bloomington, Illinois, and we have hub locations in Atlanta, Dallas, and Phoenix. We're a large company spread across the U.S. in multiple locations.

To give you context on the size of our Enterprise Technology department, we have approximately 6,000 employees, maybe closer to 7,000. About 2,700 of those would classify as software developers or infrastructure analysts. We have approximately 2,000 web apps, and we still have a ton of COBOL and PL/I. Any type of technology, we probably use it at State Farm. We have about 1,200 products and about 15 different business areas in Enterprise Technology. It is a large department with a lot of challenges, and DevOps is an interesting journey for us.

A little bit about me: I have responsibility over a horizontal area at State Farm. I enable a lot of our developer experience: Jenkins, CI/CD tools, GitLab, and best practices for the software development life cycle. Kevin works in a business area.

Kevin O'Dell

I'm in the auto and fire, or auto and homeowners, area. We build the products that we give to our customers using a lot of the stuff that Jeremy and his team enable for us.

Jeremy Castle

Here's where I'm going to start: with a story. Back in 2015, I was a developer. I worked on a small, persistent team, and all of us had at least 10 to 15 years of experience. We were senior developers. We had just finished rewriting our auto quote and purchase application and were getting ready to put it on State Farm's internal cloud.

We started talking to each other: "When's the last time you went out to production?" One coworker said maybe four or five years ago. Someone else said, "I've been here 10 years. I don't think I've ever put anything out in production." For me it had maybe been three times in my career. At State Farm, DevOps for us meant very long releases. You were on projects for potentially years that would never get implemented. We did a lot of great stuff and had a lot of CI/CD practices, but it was really hard for us to release.

A couple months after that, I was promoted to IT architect and ascended into my ivory tower with whiteboards and Expos. The first thing my boss asked me was: "I want you to figure out how to get code from your workstation out to production in less than 24 hours." After I picked myself off the floor, I thought, "I don't even know how to tackle this problem." From a developer standpoint, I understood how to get code from my workstation to a test environment. After that, it was throw it over a wall to a test organization, then to operations, and then to prod. I had literally no sense of what it took to go out to production.

Where did we start? I needed to understand how we were delivering stuff from workstation to production. That's where I started with a value stream map. The picture on the slide is about 20 feet long, and it was on Lisa's office at the time. It was interesting because what we found was approximately 1,500 hours to go from workstation to production, about 150 steps, about 45 different handoffs, and about 15 or 16 different unique roles just to get something to production.

When we looked at it, we started seeing bottlenecks in the value stream. From a security standpoint, I used to have to talk to an analyst. They would run a security scan, send it to an InfoSec analyst, who would analyze it and send it back to the person who ran the scan, who would send it back to me. That was a 72-hour SLA just to get one security scan. We used the value stream map in meetings to say, "You may think this process is great, but when I put it all together, it won't work. We don't have 72 hours to wait for one security scan."

I applied the numbers from that auto release, and we came out to a little over 9,000 hours just to get it to production. That was with 130 test deployments and several deployments to pre-prod environments. We would walk into meetings and say, "We're spending 9,000 hours just to get something out to production," and people would say that was not possible. But when you looked at it, that was what it took. It was not all manual work; some was waiting around, but it was still a bottleneck.

As we applied that, our delivery time was two to three weeks. We were able to get it down to two to three hours using that value stream map to drive automation and make investments in more modern cloud-native platforms like Cloud Foundry and public cloud.

The way we tackled the value stream was that the first half was getting stuff out to test environments. We spent time automating pipelines, breaking through process, and finding unnecessary process. The second half had to do with our change process, which was a beast. Prior to this, we had a manual change process. There were 13 different processes in HP Service Manager. Sometimes there were over 100 manual fields to enter. Six roles were involved in the change process and three approvals. That took time, energy, and handoffs.

We focused on automating the change process, and we ended up burning the whole thing down. State Farm is heavily regulated. We have a lot of auditors, and we have to prove we have controls. We built APIs around some of the things humans were doing previously, extracted a lot of the data that went into change records, and pulled that from developer tools to pre-populate change records. We got it down to one approval and four manual fields. The change process went from weeks down to hours.

The cool thing is that we picked one platform at State Farm, then genericized the change process through automation. As we build other platforms like public cloud or Kubernetes environments, we've been able to bolt on this change process and get a big lift. Sometimes people can now go to production in 15 minutes, where that was a week-long thing before.

Then we reached the point where we had done this cool stuff, but we had 7,000 or 8,000 people to get on board. I stumbled across videos from this conference three or four years ago where Target talked about their dojo. Target was very welcoming. We asked to visit their dojo, flew up with our team, and they shared a ton of information with us. We thought it would work well for State Farm. This was in 2016.

We learned a ton about the dojo from Target and decided to apply it at State Farm. It was interesting because we had no support; it was grassroots. My team asked how we were going to do it because we had no room and no space. I knew a room in a building that no one really knew about, told the team to clear their schedule, and said we had a team lined up next week. We were going to figure out the playbook as far as we could and start rolling with it. It was exciting. How often do you get to do a startup inside a company?

Our first team went through, and we thought it would work well because you could see the confidence in the team to apply agile and continuous delivery principles to their real work. Over 2017, we built it out in all four locations. We had capacity for about 16 or 17 teams at once. We've sent over 170 teams through, tracked metrics through surveys, and seen a big jump in agile adoption and continuous delivery.

The next thing we found at State Farm was that we had CI/CD pipelines, did amazing stuff with the change process, and had dojos in place, but we were still working in projects, and that was hard. State Farm started going through a digital transformation, and last year we shifted from projects to products. Kevin can talk about the challenges they faced switching from projects to products and how he adopted some of our stuff.

Kevin O'Dell

Thank you, Jeremy. All good stuff that Jeremy and his team did. It was awesome watching them from a distance blaze the trail, learn from the industry, and supply that stuff for us. But let's talk reality. I'm in an area where we have to build product and get it out to customers. We have timelines, modernization efforts, and dates. On top of that, we have a whole bunch of people who are really experienced waterfallers. At State Farm, people have been there for 10, 15, 20, 25 years, and that is normal.

It was about having a different way of working and looking at things, and Jeremy's stuff enabled that for us. Before we made the journey from project to product, once we got a whiff of the DevOps mindset and different ways of working, and heard what Jeremy's teams were doing, we started challenging the ways we were working at State Farm. I have a lot of learnings and documentation, but I want to walk through five things that were difference makers in how we worked or what we did to get where we needed to be and set us up for moving from project to product.

First: if everybody owns something, nobody really owns it. When you start scaling agile, you see your organization put layers on top of layers that match the waterfall layers. The Plinko chip goes down and finally hits a dev team at the end when everything is specced out: just go build this. When we tried to adopt Jeremy's CI/CD and pipelines, we looked at a big solution and saw 40 or 50 red pipelines. We asked developers what was up with the red pipelines. They said they did not really own them, because the next feature might put them in a different code base and the next team would pretty up the pipeline.

We got the developers together and asked how to get green pipelines and do what Jeremy was teaching us. They said ownership. They wanted to own a product from initiation all the way to production and sunset. We sliced up our big solution, and developers said they had started building pieces and wanted to own them. I got to change the effort and say we would have product ownership and component ownership.

Then I went to our product owners, thinking they would like owning a product. It was not that great. Product owners said, "Now we actually have to know the product my team's working on." I sat back and thought: is that the behavior we want? Do we want product owners not knowing what their teams are building? No. That proved where we needed to go. A couple of weeks later, everything was green. People were refactoring and updating tests. It felt good, and Jeremy was going to be happy.

We saw teams move out of a mercenary mindset, where they are just told what to build, toward a missionary mindset. Marty Cagan talks about this in Inspired: you want teams to be missionaries. Give them a product to own and drive. We saw teams mature. They wanted to be closer to the business and understand how the business used the piece of the system they owned.

Second: it might be embarrassing. Marty Cagan has a statement like, if you are not embarrassed by your first release, you released too late. We needed to scale down business functionality and get something out to the customer. That is hard when you want to put in CI/CD because it takes upfront time to build automation and pipelines, while the business is used to big chunks of functionality. We had to talk to business folks about the benefits. When new functionality was light and automation was huge, business was not happy. In one meeting, a business partner said, "Oh my God, this is embarrassing. We are giving them nothing." They were afraid to go to agents or other users and say that this was all they would get right now.

We took it to our executive staff and explained that this was a different way of working. It would benefit us in the future by letting us make smaller incremental changes as we learned from customers. The executives supported us. The next year, after that small release, we went from about 12 releases a year to over 250 releases, and almost half used Jeremy's automated software change. A developer checked something in, got one approval, and it was in production within hours instead of being put into a test environment forever. We had a very happy business partner. The business got to make many changes. They could say, "Seven months ago we wanted this functionality, but now that we have seen users use the system, we want something else." That was why we were doing smaller changes. The business started seeing that it benefited them and was not just a shiny systems object.

Third: you get DevOps, and you get DevOps. I wish I could say there is a box under every chair with a key to DevOps in it, but I cannot. We have seen two patterns at State Farm. Some people say, "We need a DevOps team" to do automation, deployments, or packaging. Others say product teams should be doing DevOps. We have seen benefit in DevOps enablement teams, adoption teams, or adaptation teams: people who are leading edge on new technologies, test automation, and getting things into production. But they are not doing the work. They go to product teams and enable them to do the work. They may build technology for product teams to use, but the product teams become the DevOps teams. They put stuff into production.

A separate DevOps team creates a silo. We have seen areas take that route, and everyone points fingers at the DevOps team when production deployment does not work. That team does not know your application. Why are they putting it into production? Give it to the product team and developers who built it. They know how it should run in production. When they have to do those tasks, they take a different mindset. They build in better logging and better automation because they know they are on the hook for the 2 a.m. call if something goes wrong.

Fourth: embrace the dotted line. At this conference, you hear about flattening layers and moving to a flatter organization. When you move from project to product, you put more accountability and responsibility on product teams. The problem is that people are used to horizontal teams, especially in waterfall ways of working. They default to saying they need a team looking across all product teams to make sure they are working. But once you have a horizontal team across multiple product teams, product teams stop talking to each other. They talk to the horizontal team, and the horizontal team talks to the other product team, and it goes back and forth. That could be test, architecture, or business requirements.

It is important to get that knowledge onto the product team, but then draw dotted lines across product teams. Tell them to look left and right and work with sibling product teams because they are in the end-to-end solution together. Even if I only own one product, maybe I am in a dotted line called architecture, and I need to know the architecture around my system and work with those people as I did before. We are trying to get autonomy down to product teams and let them make decisions, but ensure they make decisions while looking around and understanding who they impact.

One key thing is that you have to learn to unlearn. People have been doing things for a long time and doing them well. State Farm is a leader in auto and home insurance. Waterfall actually worked pretty well for us. But the realities of the industry mean we have to be nimble and able to change in the future. We have to build products the right way and work differently. You have to invest in learning and work with folks so they are educated and understand a new way to work.

The conversations at the conference are great, but folks back at the office may not get this education or have these conversations. You come back and say you heard something at another company; do they believe you? You have to be intentional about educating people and making sure they hear it from the industry. Everything we are doing is from the industry. It is not made up, though I have heard that DevOps was just something State Farm made up. Use Google; you will learn a lot. But you need empathy for those folks and need to enable this for them.

All Day DevOps is next week. State Farm is sponsoring it. It is an all-day online DevOps conference talking about much of the same stuff we are talking about today. It is a free opportunity for people to learn what we are learning. Last year, many people in my Dallas office participated. This year, our whole Enterprise Technology organization has said: this is your day. Take it and learn. Go to one session, or go to seven. This is our new operating model and the new way we need to work.

Derek Weeks and Mark Miller created All Day DevOps. We have worked closely with them. They sent a personal message last year that we showed to our folks, and people felt like they were part of the community. After that, we had multiple DevOps-focused events: hack days and an IT symposium, a two-and-a-half-day internal conference, though that covers everything. We decided we needed focused new-way-of-working, DevOps events about changing how we work and behave. Disrupt is something we started at State Farm. We brought in speakers like J. Paul Reed, Ken Mugrage, and John Willis to give people a different perspective. I can say "You need to do CI" and explain what CI is, but if Ken Mugrage comes in and says the same thing, people listen because he comes from the industry and has credibility I do not have.

Another thing we did was start a DevOps cohort for leadership. We saw a gap in leadership, the people who need to enable this. I put everything I had in my collection out in Git and said, "Go through this material, and let's get a group together." We have four cohorts running now, two in Dallas and two in Bloomington. We are starting small and adjusting as we go. Everything in there is from the industry: training, videos, videos from this conference, and articles. It is material our people need to read and watch to get excited and learn more. We have seen success. More leaders and people are coming to ask when the next cohort kicks off.

The DevOps community is great. I am amazed at the responses I get when I email someone I met at a conference, and the help we get. People want to help. If you reach out to us, we will talk to you. Reach out to your friends; there are always people willing to answer questions. The community has been awesome. Also, a shout-out to our doodler: Lisa Frye works with us, and this is a hidden talent of hers. This presentation was ugly when we first created it. She made it a lot better.

The challenges ahead for us: autonomy versus alignment is a big one. We have empowered product teams that need autonomy, but we struggle with where the line is. Where is the alignment? What decisions are we comfortable with product teams making? Where are the curbs where we should say, "stay between here and here"? We do not want too much rigor, but we also do not want things to fly off the handlebars. We have had to pick our battles. With automated software change, we funnel everyone into that, and that is good because it gives a consistent experience. We have seen good results. The adverse change rate for manual is about 2% or 3%; people using automated are about 0.3%.

Helping teams get better is really about metrics. There are many metrics, and we do not want to tell teams they need to do all of them. We are focusing on the four DevOps metrics and asking whether those are right or what else there is. It comes back to enabling teams to get better. We have things going on in a couple of areas and are listening at the conference for what others are doing.

We have struggled with metrics because State Farm traditionally likes numbers like defect rate and hours until a project ends. People want to take velocity points, convert them to hours, and project out two years of work. That does not really work.

The last challenge is modernization. You hear that you should not necessarily have modernization efforts and should do it small. That is great, but hard in practice, especially with people used to big efforts and waterfall rollouts. We are trying to enable the mindset of smaller incremental deployments, especially when modernizing large policy admin systems, which are huge and intermingled today. We need a roadmap-ish view that gives people an idea of when functionality will arrive without strangling teams by forcing them to say exactly when everything will be delivered.

Jeremy Castle

From a DevOps enablement standpoint, where my space is, we have made investments in Cloud Foundry and public cloud. Just last month, we had over 1,000 production deployments to our PCF environment, and everyone is using automated software change to go out there. We have teams that sometimes go to production multiple times a day, which was unheard of for me back in 2015. Seeing where we were and where we are today has been night and day and extremely exciting.

Closing

Kevin O'Dell: Pretty awesome.

Jeremy Castle: That's all we had for you today. We have 20, 19, 18 seconds left, so we hit our time. If you have questions, send us emails or whatever.