Host Intro (Gene Kim)

Thanks so much, Maya and Ross. If you're like me, you were blown away by Doug Parker, CEO of American Airlines, talking about how transformation is everyone's responsibility and certainly not just IT's. If you are interested in learning more about the American Airlines journey and the talent they're looking for, please join the session they're hosting during networking time. We'll post the exact details in Slack right now. Okay, one of the next speakers is the only person who has spoken every year here at DevOps Enterprise Summit, all the way back to 2014.

That person is Scott Prugh, Chief Architect and SVP Software Engineering at CSG, a leading provider of customer engagement solutions, and they are a publicly listed company. Over the years, he has spoken about one of the most amazing transformations I've ever seen. The focus of his stories have primarily been around the systems that generate the majority of revenue for the company, which includes changing how the teams work, modernizing mainframe code that goes back over 40 years, migrating away from hostile vendors whose business models are in conflict with their customers, and much, much more.

I'm so excited that this year he is speaking with his boss, Ken Kennedy, Executive Vice President and President of Technology and Product. Ken will be speaking about how he, as a profit and loss owner, values the work that Scott and the technology teams have done and why it's so important that Scott continues to build more leaders within the technology organization, and his advice to this community on how to convince executive management on the importance of DevOps. Also, Scott is on the programming committee for this conference, and he also talks about the leadership lessons learned, which helped enable all the amazing outcomes he's talked about in previous years.

Here is Ken and Scott.

Ken Kennedy

Gene, thank you for that introduction. Hello, everyone. My name is Ken Kennedy, and as Gene mentioned, I'm the Executive Vice President at CSG and the President of Technology and Product. And in that capacity, I'm responsible for our product management organization, our engineering organization, our public and private cloud operations, and our corporate IT functions. And my goals center on revenue growth, profitability, and customer satisfaction. And where I spend the majority of my time is looking to optimize the flow of work so we can deliver the greatest value to our customers in the shortest sustainable lead time.

And I'm joined today by my colleague, Scott Prue.

Scott Prugh

Hi, folks. My name is Scott Prugh. I'm Chief Architect and SVP of Engineering at CSG, and I've spent the last couple of years really focusing on high-performance engineering organizations that both build and operate their software.

Ken Kennedy

And today, Scott and I would like to share our story. We will share with you some of our experiences over the past ten years, the importance of strong leadership, and how to convince executives like me that DevOps matters when it comes to corporate performance. Before we do that, I'd like to briefly share some information on CSG. CSG is a provider of innovative customer engagement solutions that help businesses acquire, monetize, engage, and retain their customers. We've been in business for about 35 years, and we have 500 clients spread out over 100 different countries.

Our annual revenue is just shy of a billion dollars. We have 4,000 employees around the globe, and we spend about $140 million each year in R&D. The solutions we provide fall into three main categories. The first is around digital monetization, or revenue management is a more common term, and that is basically enabling service providers to be able to monetize the services that they deliver to end customers. The second is around customer engagement, which is really driving the best possible customer experience by delivering the right message to the right individual at the right time through the right channel with the right context.

And the third is our digital payment solutions, which enable the transfer of funds, whether through electronic funds transfer, credit card payments, debit card payments, and we have a payment gateway solution that enables our customers to get compensated for the services they deliver. But one of the things I'm most proud about is CSG's repeated recognition by Gartner as a leader in their Magic Quadrant that comes out every year. And in being recognized as a leader, CSG is highlighted for not only their completeness of vision, but also their ability to execute.

Our journey over the last ten years has been a very interesting one, and we look very different from when we did when we entered the last decade. Over the past ten years, we've created accountable, cross-functional, high-performing teams that have allowed us to be more efficient. We've introduced a lean portfolio management that enables our product management to focus on those features that are going to drive the greatest value and manage our work in process. We've also invested heavily, to the tune of tens of millions of dollars, in automation and technology modernization.

And we've achieved faster throughput and significantly improved quality by some of the changes we've made embracing DevOps principles and optimizing flow. And finally, we've fostered a culture of continuous improvement and learning by making this a safe environment where we learn from our mistakes rather than punishing people for mistakes. But to get into a little more detail, I'd like to hand that off to Scott Prue.

Scott Prugh

Talk a little bit now about really our journey and what we've discussed over the last few years at DevOps Enterprise. We started in 2014, really talking about our agile transformation and then really carried that the next couple of years through our DevOps transformation. And that focus was really around the dramatic people and process changes we made to reach higher performance. Last year at DOES 2019, Gene reached out to me and he asked for me to talk about our modernization journey and really how we went about implementing CI, automated testing, and even large-scale code conversion and porting across a lot of our infrastructure.

And that even included modernizing our mainframe and basically starting to convert a lot of the code off and actually converting from traditional VSAM systems to modern relational databases. And this year, both Ken and I will be talking to you about executive leadership and how DevOps matters to corporate performance. So a little look back at some of the metrics we've covered the last couple years. Through our agile transformation, kind of early DevOps, we did grow our subscriber base quite a bit. And then also at the same time, transactions in our system grew dramatically.

And as you can see, really kind of up to date, we've grown from 50 million subscribers to date to about 67 million. And over that same period of time, we've grown from 750 transactions on our platform to over 8,000. That's almost a 10x increase over that period of time. And we have done all that with keeping our costs constant. Additionally, when we look at incidents or impact on the system, we've made dramatic improvements there. We've gone from close to 1,600 incidents per month and dropped now to under 300, to 286. That's an 80% improvement.

Additionally, we measure something called release impact, which is when we put a release into production, what impact or incidents come out of that release. We've dropped that dramatically, over 97%, from 507 to 13, through the improvements that we've made. And the final thing, and the most compelling, is really to look at two things here. One is release on demand, which basically is the deployment frequency that we put code into production. Basically, we've decoupled many of our feature deployments from the release, so that we don't release in a big batch.

We release when they're ready, and we call that release on demand, and we've been measuring that percentage since 2018. Additionally, we measure something called impact minutes, which is basically the recovery time multiplied by a blast area radius. And as you can see, we've reduced that from 22,000 to 4,300 over this period of time. So release on demand has increased dramatically. The deploy frequency has gone up, and the impact on the system has basically gone down. So now kind of a quick summary of all these metrics. And really interesting to us is how they tie to accelerate in the high-performance metrics documented there by Nicole Forsgren.

Our change failure rate, the change failure percentage ties to what we were tracking, the incidents per month and release impact and also part of the impact minutes. Mean time to recovery is directly tied to impact minutes. Then feature cycle time and release on demand really affect that lead time. And then finally, deploy frequency is basically what we were driving with release on demand, which is deploy faster when the code is done. And really, all of this basically enabled the outcomes below. The ability for us to grow subscribers, grow transactions on the platform without increasing costs.

And what I'm really most proud about is basically our employees are five times happier having done all this. CSG delivers our software as a service. Some of our products include revenue management or billing for services, field service management, which is managing technicians and trucks as they do installations and service calls, and output solutions, which includes the composition and delivery of billing statements. We expose these capabilities through a common service layer called SLBOS, and the service layer supports 30 of the largest cable and satellite providers in North America.

It has 180,000 call center representatives. It serves 67 million households across North America and over 1,000 unique application integrations developed by our customers. And our first story takes place in 2010. Back in 2010, our software releases were major events. Because we are a SaaS company, when we push out a new release, 100% of our customers get the new functionality simultaneously. At the time, we delivered big batches of functionality simultaneously, so there was often fallout. And it was not uncommon for us to deliver a release and spend the next two weeks triaging and correcting issues.

And I'd like to share with you a story of one fateful release in 2010 that I will certainly never forget. The release went in very early on a Sunday morning, as it usually does, and on Sunday, volumes were light, and we had a usual number of tickets to work through. Then on Monday morning, as volumes picked up considerably, we realized that we had a major problem on our hands. The response times for our SLBOS service layer were unacceptably slow, and the situation was going from bad to worse. Every single application was impacted.

Calls to our service desk were at an all-time high, and my mobile phone was blowing up with calls from CIOs at Comcast, Dish, Time Warner Cable, and Charter Communications, just to name a few. We pulled together the development team responsible for SLBOS and reviewed the changes that went into the release. To our surprise, there was no smoking gun that could explain a broad performance degradation. We engaged a separate operations team for assistance, and then we received the news. Operations had decided to migrate from Solaris M4000s to IBM AIX servers to save money on hardware.

The hardware change went in as part of the release, and the old hardware had been powered down. And we knew we had a big problem, but we didn't know we had a massive problem. It took us 48 hours working around the clock to stabilize the situation. We had to power back on the old Solaris servers. The proprietary boxes were underpowered to support the pent-up demand. Our deployment processes were entirely manual. Configuration changes to proprietary middleware required us to cycle the server, and the cycle time was 45 minutes long.

Compounding the issues, we rushed to restore service. We made manual mistakes, which would cause us to entirely restart the process. When the dust settled 48 hours, I knew our service layer represented a massive single point of failure that we had to address. And at this point, I turned to the development lead I trusted most, and I said, "Scott, I need you to fix this." Well, when Ken approached me with this challenge, I was pretty apprehensive. I had heard about the problems with SL Boss. The massive outages were legendary.

The daily struggles and conflicts between the development and operation teams were often discussed. So I turned to one of my good friends at the time, and mentors, Mauricio Zamora, for some advice. We called him Mo. Mo's advice with a grin is, "How could you make it worse?" He then elaborated that leaders look for opportunities to lead and solve problems for their org, and this is an opportunity to do that. He also comforted me and said, "Hey, you have the skills to make this better." So I took his advice. Sadly, Mo died a year after this conversation, and he would never see the fantastic transformation that occurred with SLBOS and at CSG.

I know he would be proud of what we have accomplished. So a little bit more about what we did here. We basically went and took our proprietary infrastructure and put automated builds and automated testing around that to start as a foundation so we would understand how it worked and so we had faster feedback to build the system. We then set off to port some over 300 transactions to commodity infrastructure. And as you can see, kind of the results here, they were pretty unbelievable. We lowered the cost some 300x. We were over $7,000 per TPS, and now we're around $20.

Our feature development time increased dramatically, the amount of time we could spend on feature development from 15% to 55%. Build and deploys went from almost half a day to five minutes. Server recycles, kind of as Ken mentioned, we can now recycle servers in under two minutes, where before it took 45. Then finally, we ended up with automated tests. The other great thing that we actually did here is we started to work differently. I got introduced during this project to Steve Barr, who's actually been featured in a lot of our other stories, who is a great transformative leader who started to think differently.

He's the one who proposed that we create a shared operations team. Basically, that shared operations team would run non-production and production the same way. They would do deploys every single day and get practice, and they would run their production environment with the same tools and techniques. That piece was the start of really our understanding of how powerful DevOps could be and how that could drive a lot of our transformation to provide higher performance. Now I'll turn it back to Ken.

Ken Kennedy

For our next story, I'm going to fast-forward to 2016. At this point, I was now responsible for product management and development. Operations, however, reported to another leader within the organization. Now, one thing that has been constant was our customer demands. They wanted us to deliver new features quickly and with high quality. However, we were struggling mightily internally within our four walls. On one side of the organization, we had my development organization. That development organization was focused on delivering new features with speed.

If quality suffered, that was a secondary concern. Then they complained of the need for better tooling and demanded that operations deliver environments more quickly. And they hated our change management process that required an act of Congress to push through even a minor low-cost change. On the other side of the organization, we had an operations team that was focused on quality. After all, they were the ones who would have to get out of bed at 2:00 a.m. when something went wrong. I had one senior leader in the operations organization tell me that it was his job to protect customers from my organization's bad code.

One thing they had in common with development was the disdain for our change management process, but it was plainly evident. When you have two organizations that have different objectives, you're not going to be successful. When you have two organizations that plan work separately, you're not going to be successful. And when you have two organizations that lack empathy for the other organization, you're not going to be successful. And I knew something had to be done. I knew these two teams had to come together as unified cross-functional teams with shared goals, reporting to one leader.

We needed leaders skilled in both development and operations practices with the technical aptitude to get into the details to streamline our service delivery. And I felt so strongly, I went to my boss, our CEO, and I told him we had to change. He needed one leader sitting over development and operations. And I felt so strongly, I told him that if he had to make the decision and that didn't involve me, I would go look for another job. To his credit, our CEO made a very difficult decision because he knew what had to be done, and fortunately for me, he chose me to lead the combined organization, or I wouldn't be here telling this story.

However, our work was just beginning. This was not a popular decision among the teams. We were asking people to change the way they'd worked, and some of those people had worked the same way for decades. And we defined new expectations. Chief among them, we told developers that they would be on call. If there was a problem with their software, they were going to be the ones to fix it. We also told operations people that they had to learn how to write code to automate a large number of manual processes. Once again, I turn to Scott, as he was now burdened with making these combined teams successful.

Scott Prugh

So one of the things we kind of really thought about here was really changing the way that we were structured and going from a resource-efficient organization that ran a lot of projects to teams that were organized for flow and knowledge efficiency. We brought the operations folks together on the same teams that were engineering that software. That was difficult, and it took a while for us to get adjusted. But our leadership team was fully in support of this and fully supported our people in this transition. We worked hard to make sure that people got the training that they needed and that they were ready to actually go through and create these cross-functional teams.

Even though we worked really hard to make sure that everyone succeeded, there were some folks that didn't make it, and we had to make some tough choices to basically transition those folks out of the organization with this change. Although it took a little while to get started, we immediately started to see some great results, and we started to understand how the structure of our organization influenced both the behavior of our teams and the quality of our software. When we started to dig in with these cross-functional teams, we found that some 98% of those 1,600 incidents were being handled by those level two ops teams, those teams that previously were kind of on their own to deal with these operational issues.

When we dug in a little bit further, the team started researching. They found some 92% of those incidents were quick fixes. These were fixes that basically were corrected once but didn't get at the root cause of the problem, so they would occur again the next month and come back. Once we got these cross-functional development operations teams to look at this, they quickly started to burn down and resolve these issues. And this was one of the key drivers to that dramatic drop in incidents that you saw in the beginning and really kind of returning a lot of capacity to these teams because they were no longer firefighting, fixing these incidents all the time.

The other thing that we realized is that the problem management processes and this multi-tier support architecture that we had in the organization wasn't helping us. It wasn't going to save us from this problem because the handoffs basically prevented us from getting to the root causes of the issue.

Ken Kennedy

Over the last several decades, Scott and I have been through several battles together. CSG is the third company we've worked at together. From our consulting days, to jointly founding a startup, to working at CSG, Scott and I have shared a strong working relationship built on mutual respect and trust. And during that time, I've seen Scott grow. In the early years, Scott was an amazing programmer. The term 10X developer applied to Scott in the early part of his career. He was 10 times as productive as his peers, and I could give him the most complex coding assignment, and he'd regularly deliver.

The problem with the 10X developer is that he or she is limited by the number of hours in the day, and even if someone is the best programmer in the world, there are limits to the impact he or she can make. Many years ago, I sat down with Scott and I said, "You're reaching the limits of what you can personally do." I told him I needed him to evolve from a 10X developer to a 10X leader. I need him to have a broader impact. I need him to share his knowledge beyond his immediate team. I need him to cultivate and grow and learn new agile teams.

Today, I couldn't be more proud of Scott's leadership. Not only has he been the catalyst for transformational change, but he's also built an amazing team of leaders, some of whom have spoken at prior Does summits, and all of whom were pivotal leaders in the transformational journey. But rather than have me tell you about the importance of leadership, let me pass it off to Scott.

Scott Prugh

So I want to talk now about what we learned from growing leaders over the years. My hope is to provide a framework for folks going through a transformation and a leadership journey that they can use as a guide. The first thing I'll say is decide to lead the vision. This is really about building self-awareness. You need to ask yourself the question, do you want to lead, and are you up for the challenge? Then figure out what your vision needs to be to help your org reach its goals. And provide clarity to your people about what that vision is going to be and where it's going to take you.

The next, I would say, is about technical to strategic excellence. You need to figure out how you go from being a great technical leader to a strategic leader. In this case, you need to move from being great at execution or great at the domain that you were in and instead look at how you provide strategic value and leadership for your org and your company. The next is org needs and your leadership needs. You need to determine what your org needs are and those gaps, and then you need to adjust your leadership needs to fill those gaps.

This requires adjusting your leadership over time to help fill the org gaps that you may have. For me, this meant adjusting my leadership across many areas, setting vision on our technology roadmap, leading organizational design, doing cross-org alignment, building people and teams, as well as the international talent development. The next is build people and leaders. Basically, you need to make a strong investment in not just building your people, but also your next generation of leaders. You cannot execute a large transformation and a journey alone, and your people and leaders are vital to scaling your leadership.

Great leaders have great people, and they also have great leadership teams. Next is transparency, empowerment, and safety. You need to provide transparency for your people, as well as empowering them to make the decisions aligned to the vision of where you want to go. This also includes creating an environment of psychological safety where people are free to experiment and fail without fear of punishment. Be inclusive and bring different views to the table in your decision-making process. The next is build influence networks.

You're not going to be able to convince all at once to follow your vision. This is especially true for peers and parallel organizations. This is where influence networks are key. You need to find like-minded individuals across your org that can help influence the change. You need to build purposeful connections and identify opposing views. You need to understand the concerns and work to influence them through their directs and other peers. This is also a great opportunity to build connections across groups to improve the working relationships and productivity, because you will likely need to work with these folks later to deliver success for the company.

For example, at CSG, we built a community of learners and leaders through our DevOps culture series. This brought together parties across CSG who are passionate about working differently and creating a high-performance team. Also, extend your influence network outside your company and find practitioners in the industry who can help validate and enhance your thinking, but also influence your organization to get to where you need to go. These industry practitioners also can help form a support network to help you weather the changes as a leader you will need to work through.

For me, personally, the leaders and friends in the DevOps community have been a critical part of both my influence network and support network. Develop a learning culture. High performance organizations view learning as a vital investment and also a competitive advantage. Note that your organization will shadow your values as a leader, and as such, you need to demonstrate that learning is vital and part of what is done every day. Read and research voraciously. Build a deep and profound understanding about how complex systems and organizations work, and how this applies to your organization.

Share your learnings with your team and create space for this learning to occur. For example, as part of our transformation at CSG, we've pulled expertise from theory of constraints, Lean, Toyota, Agile, DevOps, and even the military via works like Team of Teams and One Mission. Also, this community of DevOps leaders have been vital to teach us so much about the best ways to think about transforming into a high-performance organization. Finally, give back to the community. Share your expertise through events like this in writing.

Teaching what you know is a fantastic way to give back, but it also refines your understanding and furthers your learning. The final is have courage. Leadership is very hard and success is not guaranteed. You'll need to have courage to try out many different options, fail at many of those, and try again. You will need the persistence to build a coalition to weather the transformation. The journey is very long, and it only looks simple and neat looking back. Be assured it's very hard, and without courage and a support network, it's nearly impossible to do.

The final thing I want to talk about here is how we found that really some of these capabilities in this framework tie back to accelerate and transformational leadership. And the capabilities of transformational leadership are vision, intellectual stimulation, inspirational communication, supportive leadership, and personal recognition. We have found that many of the growing leadership capabilities on our roadmap tie back to these high performance capabilities in both Nicole Forsgren's book, "Accelerate," and the State of DevOps report in 2017.

We highly recommend both those works to understand more about this. Now I'll turn it back to Ken.

Ken Kennedy

Our final topic today is how to work with executive leadership. When Scott and I met with Gene two months ago, he described a challenge many of you face, the challenge of convincing executive leadership to invest in DevOps. Well, for you to convince the executive leadership to invest heavily in DevOps, you must do two things. First, you have to make the correlation between DevOps and organizational performance for them. And second, you have to build credibility and trust. Let's face it, executive leadership is focused on organizational performance.

Executive leadership talks about revenue, profitability, market share, and customer satisfaction. Those are the things that are discussed in the boardroom, and those are the things that drive compensation. Right or wrong, many executives do not think in terms of systems thinking, flow metrics, feedback loops, experimentation, and learning. My advice to you is to share the State of DevOps report with your executive team. The report produced by Nicole Forsgren, Jez Humble, and Gene Kim did a wonderful job of studying the effect of specific business practices and the correlation with improved organizational performance.

It statistically proves that the correlation between software development performance and organizational performance. It statistically proves how improving throughput and stability will positively affect productivity, profitability, market share, and customer satisfaction. Even more specifically, it shows how lead time, deployment frequency, change failure rate, and mean time to resolution separate top performing companies from average performing companies. Once they have this agreement, you can define your objectives in terms of throughput and stability, rather than having to go through mental gymnastics of showing how automated deployments may improve the market share.

The next step is to build credibility, and this is only achieved with a proven track record of success. To build credibility, you must do three things. First, operate with an economic mindset. Business cases matter. Wherever possible, articulate the hard cost savings or the increases in revenue. These are the easiest investments for an executive like me to approve. The second thing is prioritize investments. I have no doubt that all of you have a long backlog of investments that you'd like to make. You must acknowledge that the budget is a finite resource, and no business has the luxury of eliminating 100% of the technical debt while putting all other initiatives on hold.

Prioritize your investments and focus on those items that deliver the greatest value in the shortest time. And then finally, deliver on the commitments. Nothing builds credibility like delivering on commitments. Show the benefits of your efforts. Don't be reluctant to broadly communicate your success and recognize the people who drove that success. So Gene always asks, what are the things that we're looking for? And while we're proud of our accomplishments at CSG, we know there's more work to be done, and we strive for continuous improvement.

We're looking for help in three areas, and if you have thoughts or suggestions, please don't hesitate to reach out to Scott or me. The first is around extending DevOps to client projects. While we've fully embraced DevOps within our four walls, we often struggle to convince clients to embrace DevOps principles when we're doing client-specific work for them. The second is around addressing architecture and skill bottlenecks. I'm sure there are several in the audience that have faced this challenge. We'd love to hear from some of you in terms of how you've successfully addressed bottlenecks.

And then the third is around building more fungible teams. We strive to prioritize features that deliver the greatest value, but often we find ourselves in developing lower priority features because the resources we need with the domain expertise or the technical skills are over-committed for other features. So if you have input on that, again, please reach out to Scott or I. And then finally, we'd be remiss if I didn't take a moment to thank our amazing teams. I get a lot of credit for the incredible work done by our fantastic leaders and their amazing teams.

They work tirelessly to maximize customer delight, and they've been relentless in their commitment to continuous improvement. To all the 4,000 employees around the world, Scott and I say thank you. Thank you, folks.