Host Intro (Gene Kim)

Yesterday, we heard Erica Morrison's amazing story about how they dealt with a P0/P4 outage. Up next is someone else from CSG, Scott Prugh, who is their chief architect and SVP of software engineering. I so much admire the work that Scott and team have done, which is why Scott Prugh is the only person who has spoken at all seven years of the DevOps Enterprise Summit in the US. In previous years, he's spoken about his journey transforming software delivery and operations at CSG. This year, I asked him to talk about the amazing engineering journey and architectural transformation, which is one of the most breathtaking I've ever seen, involving mainframe code over 40 years old and migrating off of hostile vendors whose business models might be in conflict with those of their customers.

You're going to love this presentation. Please welcome Scott Prugh.

Scott Prugh

Gene, thanks for that great introduction. I'm glad to be here. Very quickly, just a little bit about CSG. CSG is the largest provider of customer care, billing, order management, and digital monetization software in the world. In the US, we have some 65 million subscribers on our platform, and we support about 70% of the US cable market.

We have customers like Comcast, Charter, Dish Network on this platform, and we support getting bills out to many of the subscribers in the United States. The story today really talks about the transformation of some 100-plus global DevOps teams that support our applications across 50 apps and 20-plus tech stacks. So I want to say I'm really glad to be here, and I want to thank a whole bunch of people. So Gene, really thank you for this opportunity. And I just want to be clear that I get to be the storyteller in this, and that really the thanks goes to all the incredible engineering teams, the platform teams, the product management teams at CSG who have given us the opportunity to continue to improve and evolve our platform.

I also want to say that it's been a very difficult time, and I want to recognize all the people at CSG, people in the IT industry, and people in the other industry who have shown incredible resilience during this time. And finally, I want to say that at CSG, that two of our core values are being authentic and being a good person. This means that we treat all people, regardless of race, sex, color, or origin fairly and equally. So be a good person, shape a better world, use your platform for good. The final two things is we have a lot of people virtually here from CSG.

Reach out to them on Slack, congratulate them for all the great work that they've done, and also engage them. Ask them questions, what it's like to work in a transformative DevOps organization that continues to improve the way that they work. So I'll give a little history of where we were. In DOSE 14 through 18, we talked a lot about the people, process, and cultural transformations. Now, as Gene mentioned, today we're going to talk about our technology transformation.

That transformation was really always underneath this, but we just never went into details before of what we did and exactly how we did it. version 2.0 tag for new discoveries and actually new metrics and statistics that we found along our journey. So back to DOSE 14, I presented about our agile transformation in 2012, where we basically removed silos, collapsed several development silos into cross-functional teams that both basically design, build, and test their software. This resulted in a reasonable improvement of 83% to our releases. Then in 2016, we went through a DevOps transformation where we brought together development operations onto cross-functional teams that both build and run, and this resulted in a 74% reduction in incidents per month.

And then what also we were able to do during that time is grow subscribers from 48 million to 62 million, and also TPS on the platform grew from 750 to 4,000, or 433%. Then we spent the next few years actually spreading DevOps across the organization. So you'll note in 2018, I presented with Brian Clark, our head of product management, about lean portfolio leadership and PM meets DevOps. With that transformation, we were able to improve impact minutes some 58%, increase something called release on demand, which you'll see more about later, 460%, and our employee NPS over 400%. So looking back in 2010, we couldn't tell the future, but there are two things we knew we needed to do.

We needed to grow, we needed to lower costs, and we needed to go faster and be stable. And these were things that both our executives and our customers were asking us to do. So if we fast-forward, you'll see that that is actually what happened. We've actually been able to grow our subscriber base today to 65 million. That's a 33% increase.

And TPS on the platform grew some 800% over that time period. But with that growth, we could not increase our costs some 800%. So really, this story kind of takes you through the things that we did to actually maintain that and actually reduce costs over that time. So the problem with legacy systems is how do you maintain, patch, and secure them? How do you combat external threats and market forces?

Forces like regulations like GDPR or market events like COVID. How do you increase stability and safety? How do you go faster and deliver features, and how do you support growth without a massive increase in cost? And finally, how do you minimize exposure from hostile vendors? So my definition of hostile vendors are vendors that you spend more time in audit, compliance, and legal than you do in engineering.

It's vendors who surprise you on contract renewals with 500% increases, and it's vendors who create Byzantine compliance and audit scenarios where it makes you impossible to leverage basic core infrastructure and run them effectively. So one of the sayings we have at CSG is that we honor the past, but we also inspire the future. And we decided a while ago that we weren't going to be and act like a legacy company. We were going to be heritage, and we're going to modernize. And great engineering companies allocate time for modernization.

It's something that they do. Modernization, it fuels DevOps. It creates safety in environments. It reduces technical debt and improves things like productivity, quality, lead time, recovery. It also reduces risk.

Reduces risk of the legacy technology, hostile vendors, and also workforce risk that can continue to maintain the applications for many years. The job of modernization is never done. Once you finish one modernization, you'll find you need to do another. So it's something you have to constantly integrate into your practices and continue to drive that modernization path. So here are the realities of modernization, and I tell this story really through four sub-stories.

So, the first story is about our API platform, the second and third about our mainframe, and the fourth about our composition platform. These applications are foundational. They support a key piece of CSG's business. Now, if you were going to do this to your house, you'd move out. We don't have that luxury.

We have some sixty-five million subscribers, a billion transactions a day, and we process eighty-seven billion a year in customer revenue, and we produce over seventy-five million bills per month. So we basically had to go through this transformation supporting all of this without impact on the platform. So how do you approach a massive application modernization? The answer is very carefully with great perseverance and engineering excellence. But there are key capabilities that you can master to actually get you there.

So the first set of four key capabilities are very familiar for folks familiar with DevOps. So things like automated testing, CI, telemetry, and infrastructure. But where I'll spend most of my time today is on these capabilities above in gray. These capabilities of feature switching, code porting, incremental rollout, and strangulation. They are key to modernizing applications.

I also want to go into a little bit on kind of what I call these foundational pitfalls, and there's really three. The first is the bimodal trap. The bimodal trap is really kind of thinking in terms of old applications and new applications, and then doing things like these foundational key capabilities and modernization only on the new platforms. That really kind of gets you into a trap where your old platforms can no longer be evolved. You have these emergent threats, either regulations, security, or even hostile vendors that will catch up with you in this scenario.

The next is the rewrite trap. It's the trap that says, "You know what? That old application, it's on crusty technology. I'll just rewrite it on this really new, cool technology." That road is really long, and you also will have a very extremely hard time if you don't have testing that really exposes the behavior that you're actually trying to get to.

Doesn't mean you shouldn't at some point rewrite that, but you really need these foundational things underneath to mitigate the risk. The final thing is tech debt and giving away your pivot. Tech debt slows you down. The road is going to be very long on these modernizations. By having these key capabilities in place, you can then pivot.

You can pivot away from vendors. You can pivot away from infrastructure. You may change your approach, but if you can rebuild tests, continually integrate, and introspect the system, you basically can now kind of avoid this pitfall of really kind of giving away your ability to pivot quickly. All right. So I'm going to tell the first story, which is our ESB API, which is about a Unix-heavy enterprise service bus.

And I really start with this. It's what I call golf course software. My definition of golf course software is heavy software that gets selected higher up in the organization and then forced onto development and operations teams to implement. And so look at this sales executive and CIO. They're really happy to be doing this deal, and they're saying things like, "Low code.

I don't need developers, just map your data. It's really easy to operate, and it's also integrated with everything in your enterprise. It's really great." Please don't do this to your people. Let them pick their own tools.

So, a little bit about the problem we had with this. This heavy platform had poor developer aesthetics. It had high build and test efforts. It had poor operational aesthetics, low TPS density. It was unsustainable cost to support the business growth.

So our approach was move it to commodity, port some 300 transactions leveraged by 1,200 integrations, strangle the old platform off by using feature flags and canary, and apply these foundational modernization capabilities, things like testing, CI, telemetry, and infrastructure. So a little bit more about how we did this. So we had a software load balancer, and we actually had that from previous implementations, a route table that we were able to leverage. And in that route table, we added a flag, and it was called the route flag, where we could actually direct traffic to old and new applications. So what we did as we went through the port, we would take a smaller client, lower risk transaction, and we would flip that route flag and send it to the new services written in that new technology.

And we were then able to go through a client and transaction at a time and effectively test and make sure that we could actually roll this out. Contrast this to large batch go dark modernizations, this is a much safer way to go about it, because you will find unknowns along the way with different usage patterns that you need to be able to adjust for. And then finally you get through, and you roll through all route flags, and then everything is pointed at the new technology. And then finally you strangle that old technology off and take it out. Some other things about the foundational modernization that was really important here was automated testing.

So before we had some heavy proprietary test tools that were only used by testers, creating silos. It was a high cost and increasing. It turns out that this was a hostile vendor who was continuing to increase our cost. And there was just a high manual test effort still, because these were really test case-driven tools and not automated tools. So we moved all this to basically Gherkin, and then we put the tests in version control with code, and then the testers and developers could collaborate on those test suites.

And with this, we were able to really build up a full functional test suite for the application. As part of this, we really grew these automated tests in 2012 from almost zero to close to 14,000 tests today. And we really documented that journey in this previous DevOps Enterprise Forum paper that you can leverage. So, a little bit more on how we did the testing here, and I want to go through, is that we also wanted to basically confirm that the old platform and new platform worked the same. So we did something by injecting a route flag into the transactions, running the test suite through that application, and then actually recording the old results pointed at the old application.

Then we actually flip the route flag to be new, record the new results, and then we can compare those. And we would run those every night through a series of tests. And basically then we would know that the new infrastructure produced the same results as the old infrastructure, and that mitigated a ton of risk for us by applying automated testing to this modernization that way. So, a little bit of statistics. One of the things that we really struggled with the old platform is the TPS density was really low, so we increased that from around 200 to over 1,300.

The dollar for TPS was just dramatic. Basically two orders of magnitude times two cost reduction in TPS. Able to direct a lot more effort from the team to feature development. Build and deploy dropped from hours to minutes. Server recycles from tens of minutes to a few minutes, and really drive the automated testing was another key result of this.

So I have to recognize and thank a bunch of people. Picture on the left is the SLBOS team, really, who took us on this five-plus year journey. Really have to thank and recognize them. And because of the modernization that was done here, we're able to actually extend and grow that team to our Bangalore office. Now with commodity code, it's a lot easier to train and upskill people to support this.

The other people I want to thank, individual Mike Battalucco, he was the director and manager of that team, who just showed incredible resilience of continuing to help manage this effort. And then my colleague, Jeremy Van Haren, who was a great visionary, really helped lead the teams and really get us through this. So finally, I want to zoom in on a plaque that you'll see everyone holding there, and I've also got on my desk. It says, "The best time to plant a tree was 20 years ago. The second-best time is today."

So modernization is something that you really want to start. Even if you haven't today, it's not too late. The second thing here is one of my favorite quotes from one of my favorite movies, from Edna Mode: "Luck favors the prepared." The types of things that we've been able to do here by being prepared by doing this migration, we can spin up new environments, we can segment off customers basically on their own instances. These are types of things that we could never do with the old infrastructure because of cost and the inability to manage it effectively.

Second piece of this picture is this, is the celebration. And I'll tell you that Wes did say that in "The Unicorn Project," but I did not say that actual quote. What I said actually could not be repeated or printed. And additionally, that is not an 8U server. That's a 2U server.

There were 8U servers in this environment, but they were actually too heavy for us to get them into the parking lot, so we just used the 2U servers. So the next story is about our mainframe and DB2. The problem there that we had is just this lack of commodity data access. We could only get to it by a CICS. Maintainability was at risk, and these unsustainable cost increases continued jeopardizing us.

So we approached it by converting VSAM master files into DB2 tables, creating an incremental rollout feature-switching approach, strangling off the VSAM data store, and offloading these read transactions from CICS to direct DB2 queries. So, I've updated this because I did miss one thing in previous, but what we had here is a feature-switching mechanism I call the data store edition. Porting data store is harder than code, and the pattern goes like this. You make the old data store be the primary. That's the first step, which is kind of where you are.

Then the second step is the old data store is the primary, but the new data store becomes the replica, and you do need to backfill that. So you need to backfill that old data store. You can do that in batch, or you can do that online. There's several different techniques there to do that. And then you compare them.

You compare them to catch any issues. Then the third step is making the new data store primary, make the old data store the replica, compare them. And then you finally make the data store the primary. You can use this pattern with any type of data store. You can use it with document databases, relational databases.

The industry has seen this pattern in many places, and we use this for our mainframe data migration. So here's a little bit more around how it works. So actually at the mainframe level, we had a bunch of switches. And we were able to use those switches and carry them through a set of sequences to switch the read and writes. So the first switch is switch one, which is VSAM only, where we just read and write to VSAM, and DB2 is inert.

The second switch is switch two, where we basically read and write from VSAM, and we write through to DB2. We also backfill that so that the data is in there for these transactions. The next, switch three, is basically we read and write from DB2, we write to VSAM, and at this point, we can roll back to either switch one or switch two. And this is key because when you find issues, and you will find issues because there's going to be data that's either different or corrupt that you need to clean up, you're able to actually roll back very quickly. And then use those nightly batch compares, which we have to actually find a lot of those issues.

The final switch, switch four, DB2 only, basically is we read and write right to DB2, and then VSAM becomes inert. So some other things that we actually did here and to facilitate through this API platform is build on top of our route flag that you saw previously. So we added a flag called the read flag, and in this, we actually can now direct reads to CICS, which is the old path, or we could direct the reads to DB2. And so we really went through that same set of steps to actually phase through the VSAM to DB2 conversion through the API platform. So some kind of metrics here.

Data access was VSAM via CICS before, and now we can go DB2 direct. Read transactions were 100% to CICS. We now go 62% of them direct to DB2. We have high data accessibility, and surprisingly, our average response time dropped. We did all this with near zero customer impact by being able to flip through these flags and roll back during the day.

So the next story is about our mainframe Java evolution. 3.7 million lines of high-level assembly. Maintainability was at risk. Really the productivity, the agility, the workforce sustainability, and we continued to get this unsustainable set of cost increases. So the approach here, we built some cross-compiler tooling with specialty companies that actually do that.

We wanted to target the more complex update logic. We wanted to take all this converted code and get it into these really foundational capabilities, things like CI and the test coverage. And then we wanted this code to run off board, and we needed to build an incremental rollout strategy here, again, with feature switches and do deploys during the day. So a bit more about the details of the steps here. It really kind of starts with that foundational test coverage.

We were able to leverage a lot of these automated tests that we previously built to cover the entire legacy code base with that testing. So we got the breadth and the depth of that testing first, and that really helps us understand actually how it works today. Then a code analysis. So we use code analysis really to understand the dependencies, and these are actual pictures of dependencies in the system. And this allowed us to really choose key modules that we would actually pull out because the dependencies are sometimes very deep.

So we could start with some modules that were easier on the edges. Then cross-compilation. So, we use leverage cross-compilation to basically do mass migrations, codify patterns, and then additionally refactor learnings. 3.7 million lines of code. So one path is you just start typing.

The problem with that is you'll really never finish, and your cycle time to make adjustments is really, really long. The other is to build a cross-compiler, and that's actually what we did. So we built a cross-compiler that takes the high-level assembly, runs through this compiler technology, and it produces Java on the other side. And just a quick on what this looks like. On the left, we've got high-level assembly.

On the right, we actually have the Java that's output. And for folks who read assembly and Java, you'll note that they're isomorphically equivalent. But this is what we were able to actually do, and now we got this new Java code that we can test on the other side. And so with that cross-compiling, basically take that and we get foundational version control, CI, and unit tests across that. Then we refactor it.

So with that, we can do things now which would be very hard to do by hand. We can actually feed patterns into the compiler. So this is a very simple one where we actually just have logical names that we want to assign to the symbolic names. We can recognize that domain-specific pattern, insert that into the compiler, and now we can increase the target code base maintainability on the other side. And the final thing is running that functional test coverage again on that target code so that we actually verify that we are getting congruent behavior across both the original code and the ported code.

The final thing is the feature switching, the telemetry, and auto rollback. We leverage the feature switching to roll out and roll back. We integrate that heavily into production telemetry, so when we detect issues, we can then trigger things like auto rollback upon failure. So a little bit what that looks like. We actually have more flags that we've added into these route tables.

And with those flags, we can direct the traffic up to CICS, or we can actually direct the traffic to a Linux Java cluster where the ported application code is running. And one of the things we did here is actually we wrote some functionality to auto-detect failures, and when we detect those failures, we can fall back immediately to CICS. So it gets our recovery time close to zero, so that if there is something wrong with the code or wrong with the new cluster, we still have the ability to fall back, and we do that very quickly with almost zero impact. We're able to actually do that during the day. So the final thing, some of the results is that, right now it's 100% high level assembly.

We're looking to get 85% of it to Java. This project is still in process. It's not complete. We want to get a lot of the code going direct to DB2. The maintainability becomes much more supportable, much higher productivity, because we can use a lot of the shared tools across groups.

And then, we can use open telemetry across this, which before we had a lot of isolated telemetry that wasn't shared across teams, and we really share those practices and tools, and again, with near zero customer impact. So, with this, there's just so many people to thank. I do want to thank the mainframe teams that were involved in this. It was just an incredible journey, both the DB2 and the Java. It's just amazing.

I also wanted to call out one individual in particular, Gary Gouger. Gary just hit 36 years at CSG, and that is pretty amazing. And one of the things about Gary that I'm so amazed with is his ability to just continue to learn and really continue to take the challenge to evolve this platform and modernize it. Thank you, Gary. All right, so the fourth story is about our composition platform.

So our composition platform really takes output from the billing system and turns that into electronic statements or even printed statements that go into the US mail system. So with this platform, we had about four million lines of proprietary COBOL that's been around for twenty-five plus years. There was no version control, no CI. There was proprietary Unix, proprietary and hostile vendors. We didn't have unit and functional tests.

There wasn't great telemetry. It didn't scale horizontally. There was just this unaffordable vertical scale. You had to have bigger and bigger irons to actually make it work. We approached it by adding foundational CI, COBOL unit testing, end-to-end functional testing, adding feature flags to support trunk-based development and incremental rollout, and converting the proprietary Unix and COBOL to Linux and GNU COBOL, and we moved difficult and hostile vendors.

So what I'm going to do is I'm going to tell this story, actually, Steve Barr is going to tell this story for us, by going through the celebration day when they finally actually completed it, and it's really great. For the folks who actually are familiar with transformational leadership, either from Stephen Mayner's work or from Nicole Forsgren's work in Accelerate, I want you to look for a couple things in this presentation. Really how Steve exhibits the behaviors of transformational leadership. Things like vision, things like recognition. These are things that kind of inspire people and it's a really great example of this in this video.

Steve Barr (recorded clip)

For the first time in 25 years, a big part of our application is under version control. That's a big deal. There's more. We also have continuous integration, and what that means is, is that as a developer makes a change to the software, their change is automatically compiled and automatically built as a part of the build process. So we know right away when something is working as soon as we check it in.

That's great. There's more. We also have COBOL unit testing. So COBOL is considered a legacy technology, but we've put some modern techniques in place to where when a developer tests or checks in the code, it's automatically tested. So there are unit tests now around the COBOL.

We've really built a learning culture. We learned that we have to change things. We have to improve things. We learned a ton through this effort, and that's what I'm probably the most proud of. There's a lot of things that I'm proud of, but we've begun to create what I would consider a safer environment.

All right, guys, I'll kick it off. You ready? Yes. All right. Hip hip.

Hooray! Hip hip. Hooray! Hip hip. Hooray!

Good job, guys.

Scott Prugh

All right. That was great. So for folks who enjoyed that, please get on Slack, give Steve a little hip hip hooray. Give a congrats to the team. Give them a hip hip hooray, because it was really a great and successful project.

We're 95% complete. We're really on the last line here, and I'll give a little commentary on that. We've converted four million lines of proprietary COBOL to GNU COBOL, and GNU COBOL is very interesting because it compiles to C, so you can actually integrate in all kinds of other third-party C libraries, almost your own packaging certificates. Really great telemetry around the system. We're on a commodity solution, Linux and COBOL.

And one of the reasons this has taken longer than we had hoped is because of hostile vendor leapfrogging. Some of these vendors do not allow you to basically kind of dual license across multiple platforms even though you're using the same capacity. So we actually have to remove those vendors, bring in either open source or friendlier vendors who will allow us the flexibility to actually migrate over time. So, just a couple things at the end here really to kind of give a process update, and some things that modernization does also allow you to do. So modernization improves lead time.

It improves things like deploy frequency, your change failure rate, and your meantime to recovery. So we have something at CSG that's called release on demand, abbreviated as ROD, and it means releasing value when it's ready and decoupling it from large releases. ROD ends up being a forcing function, which is a behavior-shaping constraint for high-performance behaviors and foundational modernization capabilities. So if you kind of look at our ROD progress, back in 2017, we're around 5%, and we kind of kicked off a program to say, "How can we release features faster?" And this was really our mechanism to do it.

This was our mechanism to move to a batch size of one. As of our last release, 20.1, we hit 69%. So what that means is 69% of all of the features were put into production before the release day, and we're getting really close to no longer having a release day and really just having continuous flow of features through the system. And you couldn't do this without practice changes, cultural changes, but also you can't do it without having great modernization in place. So a bit of refreshing the metrics, and this is what I presented at the last time, '18, and I've really got an update for you on what stuff looks like today.

So release impact continues to improve. We had one of our best releases, and a 97% improvement. And if you think about release on demand, the releases are getting smaller, so there's less impact. Our incidents per month have really held steady at that 80% improvement. Subscribers have grown.

Actually, even as a result of COVID, we actually saw that subscribers on cable systems have actually gone up, so we've grown to 65 million. TPS continues to grow, hit over 6,800. That's 812%. We closed out impact minutes last year at a 74% improvement. We just talked about release on demand, which is a 69% improvement.

We've got new eNPS scores from our employees. That has gone up again, and so we're at a total of 500% improvement of our employee net promoter score with really kind of all these changes. And a new metric here, feature cycle time. We measured our cycle time at the beginning of this journey, which is when we actually start development to actually when we install. It used to be close to 250 days.

We've actually dropped that to 56 days. That's a 77% improvement. And finally, I want to say that if you haven't read Nicole Forsgren's fantastic book, "Accelerate," I highly recommend it, but I tied the accelerate metrics, the change failure percentage, the meantime to recover, the lead time, and deploy frequency, basically to our metrics, and you'll see how the change failure percentage ties to release impact incidents per month and also impact minutes. Meantime to recover ties to impact minutes. Lead time, deploy frequency ties to release on demand, and lead time and deploy frequency also tie to feature cycle time.

So just fantastic correlations between what the research in "Accelerate" predicts and basically what we were able to show with this transformation along the way. So what did we learn? So you can modernize your legacy applications. Be heritage. Don't act like a legacy company.

Modernization is vital. It requires engineering excellence. Leverage CI, automated testing, telemetry, and infrastructure. Don't fall into those pitfalls, bimodal, the rewrite trap, the pivot giveaway. Really leverage those foundational capabilities.

Optimize for developer and operations aesthetics. Use feature switches, code porting, incremental rollout, and strangulation on your modernization journey. And finally, fuel DevOps. Create safety and reduce that technical debt. Improve things like productivity, quality, your lead time and recovery, and finally, reduce your risk.

Reduce the risk of legacy tech, the risk of hostile vendors, and your workforce risks. You can modernize and fuel DevOps with these techniques. So finally, help that I'm looking for. Capacity forecasting and estimation and wishful thinking continues to be the most difficult problem in computer science, I'm convinced. Trying to forecast your capacity, get good estimates.

Next, portfolio-level WIP constraints, really kind of stopping this technique of feature Tetris, of trying to squeeze in that one last feature that really continues to overload your teams. Improving intake lead time with traditional IT mindsets, having to have an SOW that's fully scoped for every piece of work, and then creating capacity for backlog swarming. How do you get to that capacity for those next set of lower impacting incidents and kind of swarm and resolve those? Thank you.