Host Intro (Scott Prugh)

Hey folks, I'm really glad and feel so privileged to be here today. I really want to thank Gene and the community for their support over the years. Today I feel really privileged to introduce our next set of speakers from CSG, Erica Morrison and Steve Barr.

If you look back at the history of the DevOps Enterprise Summit, you'll see that CSG was fortunate enough to tell their story over the last nine years 17 different times. The CSG story is an amazing one that took an organization from 18-month releases and siloed organizations to nightly deploys and cross-functional build-run teams. This was done on traditional technologies like mainframes and data centers, proving that DevOps transcends specific technologies. It's more about the capabilities that you can use to unlock high performance.

Now for our fantastic speakers. Steve and Erica were instrumental in both the transformation at CSG, but also giving back to this community. Their learning, their expertise, and their leadership, the work they have done, has had substantial impact on moving us all forward.

Steve Barr started as a teacher before moving into IT. His first work was on the development side, building our enterprise desktop application. When I met Steve, he had moved to operations to help fix things there. Steve and I were struggling to unwind the problems that were resulting in frequent releases and massive handoffs between dev and ops. He was selfless in his work to bring us together and partner with engineering to find ways to cross-skill our teams. It was Steve who came up with the idea to stop having practice teams and game teams for ops, and create one ops team.

Erica has been one of my key partners in helping lead and implement the types of changes that we needed at CSG. She is so insightful in her ability to understand problems but also lead teams towards solutions. She has unbridled courage, brilliance, and work effort that helps her and her teams reach the right answers. Erica helped lead CSG through the worst outage we ever had. And at the advice of the late Dr. Cook, she then had the courage to present this outage to the entire community in 2020. I'm so proud to introduce my colleagues and great friends, Steve Barr and Erica Morrison.

Erica Morrison

I want to send a big thank you to Scott. Scott, you've done a ton for CSG. You've been a big part of our transformation story, so I'm very excited to be back here today getting to continue to tell that story. We're going to change things up a little bit today. I'm going to talk about transformational leadership, and Steve is going to share his perspective on our story: where it started and how it's going.

If you're not familiar with CSG, we empower companies around the world to build a better future, making it easier for people and businesses to connect to, pay for, and use the services that they value most. We provide services in revenue management, digital monetization, customer communication management, and payment gateway services. You can see some examples of our customers here.

Over the years we've done a lot of transformations and modernizations. Steve's going to talk a little bit more about those in a bit here, but we've learned a lot about transformational leadership along the way. We've fine-tuned our approach. We've come up with a set of best practices and lessons learned. We don't intend to go through a bunch of textbook definitions, but rather a set of practical advice from our experience.

I want to start with a very simple statement: leadership looks a lot like loving people. If you remember nothing else from today, remember that. Taking care of your people, treating them well, these are the most foundational of all leadership principles, whether you're leading a large transformation, making minor changes, or simply looking to sustain an organization.

Step one of any transformation: seek to understand before you seek to change. Steve first gave me this advice a number of years ago, and I can see why. I've come into some of these transformations excited. I want to move fast. I want to make a difference. But you've got to make sure you've got the context. You've got to slow yourself down. Remember, sometimes there are legitimate reasons things are the way they are. Sometimes problems are more complex than they appear, and sometimes you're just plain wrong.

In order to get that context, you've got to educate yourself: team meetings, one-on-ones, skip levels, go through data and metrics. Listen to people when they're talking. They deserve to be heard and understood. There's something unique about every transformation. You need to make sure you understand what it is so you can account for it in your approach.

Speaking of things to account for in your approach, make sure that you've got empathy for the mindset of your people. You've got to remember that humans are naturally wired to resist change. The graph you see here on the left, probably one that you've seen before, shows how people accept change. Key takeaway is that folks are going to take a little bit to get on board, and some are going to take a very long time, and will probably actively fight you in the process.

Many view change as a threat. I promised not a bunch of textbook definitions, but I do have one here that I really like. This is David Rock's SCARF model. It says if you change any one of these five things — status, certainty, autonomy, relatedness, or fairness — it can be viewed as a threat. The human brain is constantly evaluating everything as a threat or a reward. This is really helpful when you're coming up with your approach. I've also gone to this when something didn't go as expected, to figure out what went wrong.

Because people view change as a threat, you've also got to give them the "what's in it for me." What resonates with you may not resonate with them. They may poke holes in your reasoning. Make it personal to them.

I want to talk about some tips and tricks on several different phases of the change management lifecycle. If you look at books or articles on the topic, you'll see some version of this cycle here.

Let's start with communicate. You've got to share the why with folks. They need to understand why you're doing something in order to get on board. Regularly communicate via multiple formats. I have put together what I felt was the perfect messaging, and I got it out there in an all-hands thinking I was done, and I mailed it, only to find out that some people never heard it at all and for some it just didn't land with them. You've got to say it multiple times. Some people like formal presentations, some like emails, so make sure that you're going through all different formats, including informal discussion.

Communicate bidirectionally. Don't just talk to people; make sure that you're allowing them to provide feedback as well. And communicate at multiple levels. I have made the mistake of going primarily through my directs for messaging, expecting that they'll carry it forward. But it just takes one link in the chain to fail: somebody who isn't bought in or doesn't understand the vision, and it's not going to go as you'd expect. Make sure you're personally carrying that message forward.

Engage your people. Make sure that you include everyone. Even if it's as simple as giving them status updates and letting them give feedback on how things are going, it's important that you include them so they don't feel passed by and that you're just making the change without them.

Educate. This can look like formal training. We have had great luck loaning out architects as mentors and role models. We've also brought in coaches. Speaking of coaches, you should expect to spend more of your time coaching and supporting than you normally do.

Finally, find your change agents. You cannot do this all by yourself. Make sure you are identifying those early adopters and innovators. Invest in them. Help them carry your message forward.

Implement the change. You want to change how people do their work. We have done this instead of focusing just on the culture change and then the culture change will come, and this is based on the research of John Shook at Toyota.

Build on the positive. I have tended to fixate on the negative when we're in some of these transformations — really, what's the stuff we're going to fix? Make sure that you also find the positives and build on top of that.

Don't get stuck on prioritization. If prioritization is clear, by all means follow it. But if it's not clear, I have been in some of these transformations where I'm looking around and I'm thinking, oh my gosh, where do I get started? Scott shared the advice with me: in a room full of trash, just start picking up trash.

Roll up your sleeves, get in the weeds. You should expect to spend more time working alongside your teams and coming up behind them to confirm changes are working in a transformation.

Finally, iterate. The first two items here have to do with open-mindedness, which is a very important characteristic of a transformational leader. Remember that the goal is to get all the good ideas on the table, not for your idea to win. I first came across it worded this way in the book Crucial Conversations, and really like this.

Also know your absolutes and where you can compromise. You should expect to compromise, so plan for that. Demonstrate psychological safety. Your people are going to mess up. Make it safe for them to fail, to learn, and to grow.

Demonstrate vulnerability. I spent the first part of my career purposely doing the exact opposite of this, always trying to put my best foot forward. What I found as my organization grew and people didn't know me as well is that can come off as cold and unrelatable. I had to change how I work. It's really important in something like a transformation that you are vulnerable, even if it's uncomfortable. Do things like admit when you're wrong. Finally, celebrate success.

We've been talking about taking care of your teams and how you work with your teams. I want to spend just a minute talking about taking care of yourself. Remember that change is hard for everyone. That's going to include you. Be cognizant of burnout. I have run myself straight into a wall multiple times. Look for those warning signs, set some boundaries for yourself. Don't just tell yourself that you're going to be okay.

Find a sounding board. This can be lonely at times, leading a transformation. Find someone that you can vent to, that you can go to for advice. You need that support system. And finally, don't give up. There are going to be days where it's going to be very frustrating. You're going to wonder if you can do it. Courage is a very important part of being a transformational leader, and simply some days, just keep going.

With that, I'm going to turn it over to Steve, and he's going to share his perspective on our story.

Steve Barr

Thank you, Erica.

Our transformation story begins with a modernization project. How do you modernize a mission-critical middleware application that was developed in the Windows environment, runs in production on the Solaris environment? Well, of course you port it to AIX. And you do so during a massive data center move from Denver to Omaha.

In 2010, during the initial phases of this big-bang endeavor, I moved from the development organization under the CTO to the operations organization under the CIO. Of course, I went there with the goal of fixing all the problems they had. I was going to bring in things like Kanban boards, stand-ups, and stuff. I thought my job would be done in about six weeks. But what I witnessed was that tickets were worked in one system, changes were worked in another, and new requests were worked in yet another system.

There was very little prioritization. In fact, everything was the number one priority. There was complex manual configuration that existed at both the platform and application levels. There was a lack of shared and standard telemetry, although I would argue everyone used packet captures and Wireshark as their go-to. And then there was a handful of Brents that made it very difficult to get things done.

What I began to learn was that the work in operations wasn't understood by the development organization or anyone else. Their work was taken for granted, which led them to being unappreciated. Most of us in the development organization thought the really hard work was getting the software done. Once the software was done, getting it to production, deploying it, and supporting it was like autopilot. Why are these people so busy? This was going to be more than just about boards and stand-ups. We had to make their problems our problems.

While our first attempt was an abysmal failure at migrating to AIX, with multiple customer-impacting incidents, we took what we had learned and applied it to the second attempt. This time we needed to protect the business and create a safe environment for our teams. This time, instead of a port, we decided to rewrite it using the Microsoft stack.

The things we learned were that we needed to have automated parallel testing, which meant the output from the new transactions needed to match the output from the old transactions verbatim. We incrementally rolled every transaction out one at a time, which decreased the blast zone. We would run the old system in parallel with the new system so that if we got into trouble, we could roll back. And we would deprecate the old functionality using the Strangler pattern.

The good news: it was some success. Both the development and operations teams were on that first deployment call. It was truly a DevOps moment. The bad news was it was the first time the operations team had really seen the new application. We operations people had been configuring the new system, which required us to write Lucent Stinger macros in vi, and everyone was surprised that we didn't know anything about Process Explorer, WinSCP, PowerShell. There were a lot of high fives when that initial transaction went to production, but we had a lot to improve.

What we realized is that we had three different teams operating environments in very different ways. Most ops teams on the non-production environments were doing deployments and system changes multiple times a day, whereas the production ops teams were doing it at best weekly. Feedback from operations to development was sparse, oftentimes requiring the production operations team to learn the same lessons that had been learned in the non-production environments, but do it on game day. The practice team was not the game time team.

What we created was a shared operations team, and the shared operations team would be married to our product development teams. With that, environment management became homogenized. Operational artifacts were shared and under version control. Not all environments had the same capacity, but they all had the same components.

Things were better. We would often brag to ourselves, hey, we're taking the system view. But the reality was I was taking my system's view, and they were taking theirs. We still had large batch releases four times a year. There was little or non-contextual change control, where people were put in a position to approve or disapprove a change and had no idea what the change was about. Change was limited to predefined early morning windows. If you were unlucky enough to not get your change in the current window, you could wait up to a month before you would get your change in.

There was still mismatch in prioritization between dev and ops. In operations, we had to support things like security patches, network changes, and our customers. Those priorities didn't line up with the development organization. There was an incredible lack of feedback between operations and development, in that 92% of all the production tickets were closed by operations with no feedback to development.

Finally in 2016, we reorganized and we went full-on DevOps. We organized the product teams. There were many bumps in the road. I remember Erica saying to me in the hall one time, "I don't want to change anything. Change always breaks stuff. I just want to sleep." To which I said, "Ah, finally she's an operator."

With help from the product management team, operational work became visible and our priorities began to align. Operational work, stability, and incident resolution were scoped and engineered into new feature development. Because we were one team, practices began to be shared. For example, our test team was using SpecFlow and Gherkin to test our application, and our operations teams created a synthetic transaction portal using those tools that would help us validate change and identify issues as they arose in production.

Modernization began to spread. On the mainframe, we migrated our VSAM and QSAM files to DB2, and our assembler and COBOL programs to Java. In Output Solutions, we rewrote our document archival solution in .NET and plumbed it to Amazon S3. We rewrote our document composition software using cloud-native technologies. We did all these things using the same countermeasures and safety that we'd established while migrating our middleware application.

This bled into our processes. Incident management was changed in that we normalized incidents into impact minutes and created goals to drive those down. We created a post-incident review process that was focused on what happened and how to prevent it. Outage management was changed in that we were all trained in the National Incident Management System, or NIMS, and we began running our outages with things like incident commanders and CAN reports. Change management was changed in that we went from those big large CABs to local CABs, which brought the context necessary to approve changes and provided collision control.

You can see the results speak for themselves. Our incidents per month are decreasing. Our impact minutes, or impact score, is decreasing. Our release impacts, because they've gotten smaller, have gone down. More importantly, our employee Net Promoter Score continues to go up. We've improved our time to market from 12 months to four months or less, and we've moved away from those big batch releases: 70% of our features are released on demand. We did all this while growing our business from 50 million subs to 80 million subs. This stuff really works.

What started with a modernization project became a way of life at CSG. Erica and I are lucky. We get to talk about the amazing success we've had at CSG, but we are just two of hundreds of people that made all this happen, and we are truly grateful to them and just proud to be a part of this.

We are looking for help in that we would like you to help us by telling us how you are expanding your recruiting and hiring pipelines to hire and attract a more diverse workforce. Thank you very much.