Jakes Olivier and Anthony Anter

Jakes Olivier: We are grateful to be able to share our experience with you: effectively, leaders of digital transformation from around the world. So I think give yourselves a nice round of applause.

I do not know. I am not really feeling that hand clap. Let us give ourselves a really good one.

My name is Jakes. This is my friend Anthony.

Anthony Anter: Hello.

Jakes Olivier: And we want to know: is DevOps for legacy systems, Anthony?

Anthony Anter: Yeah. Yes, DevOps is for legacy systems. Thank you. Any questions?

No, we are just kidding. We are going to go into a little bit more depth than that. We thank everybody for coming. Like Jakes said, we really appreciate it, and we want to talk about DevOps.

We specifically want to talk about DevOps on those legacy systems that are core to your companies or core to the companies that you work with. Typically, when you see these kinds of systems, these are the systems that generate money. These are the systems that authorize your credit cards. These are the systems that have all of your user data. Can you bring DevOps to those core platforms that are decades old?

Like we jokingly said: yes. But what we want to do is talk about the hows. We want to talk about some of the things you are going to run into as you bring that kind of technology onto these platforms.

Jakes, do you want to introduce yourself?

Jakes Olivier: I have been with Nedbank since 1997. Nedbank has been around for a little bit longer, so our legacy has got legacy. I started straight out of school in the print room, went into operations monitoring, monitoring mainframe batches and online as well as networks and ATMs. After that, I went and learned COBOL and became a mainframe developer. After that I went into test environment management. That exposed me to the distributed world. Then I went into software development lifecycle tooling after that, and that really put me in a front-row seat to witness the digital transformation and DevOps journey within Nedbank.

Anthony Anter: Thank you. My name is Anthony Anter. Before coming to work for BMC, I worked for a major American credit card company where I was a director of engineering for CI/CD. As such, we put a whole set of DevOps pipelines together that took source loads or changes from development all the way to production on the distributed side for about 90% of the company.

Eventually, they came to me and said, we have a real problem with two-speed IT. We have a real problem with the mainframes going at one speed and our distributed applications going another. What can we do about that? As such, I took the same methodologies, pipelines, everything that we had, and I pushed them onto the mainframe side.

So what Jakes and I want to do, since we both lived this journey, is talk to you about some of the things you are going to see, some of the things you are going to encounter, and some of the ways you can deal with these legacy platforms. What you are going to find out is these legacy platforms are not as legacy as you think. These legacy platforms are just another platform.

Anthony Anter

Anthony Anter: I want to start talking about DevOps and the sweet spot for DevOps. It is the cloud. DevOps was built for the cloud. That does not mean DevOps is only for the cloud, but the cloud was built for DevOps and DevOps was built for cloud, like lobster and butter, Joanie and Chachi. These are things that were meant to be together.

The cloud was designed and is the exact kind of platform that can maximize DevOps, can maximize the benefit you get out of DevOps. As I show here, it is your stairway into the cloud.

But with that you have this idea that you cannot use DevOps on older platforms, older platforms that do not conform to cloud capabilities, that are not in the cloud. There is the stigma that if I have a legacy platform, I cannot really automate that. I cannot do anything with a platform that is not in the cloud, that is not Azure, AWS, Google Cloud, or whatever. That is not true.

We are here to tell you that is not true. We have the scars on our back to prove to you that is not true. What we want to do is help you see that you can bring DevOps to your legacy platforms, and it does not have to be cloud capabilities.

Jakes Olivier

Jakes Olivier: Some of the obvious challenges you are going to face are pretty much similar to what we see on distributed as well. People: we have to get the mindset right. Culture is important. Shared responsibility: quality is everyone’s business. Processes: we had heavy processes, governed to death. We really had to make it simple for developers. Technology goes without saying. And then technical debt: there is a lot of application modernization that needs to happen. A lot of these applications were written many, many years ago.

Anthony Anter

Anthony Anter: Let us take a look at the first thing you are going to run into, or the first thing that we ran into: people. You have people on your mainframe, on your legacy platforms, who have been there for decades. They have been working for decades. They have been doing work the same way for decades.

How do you deal with that? How do you deal with people? You have a workforce that has paid their dues, that is getting ready, either in the process of retiring or looking at retiring in a couple of years. How do you deal with the perception that this is an older platform with older processes? How do you bring in that next generation? How do you get the kid out of college, who is going to engineering school, to want to be on this platform, to want to develop these applications that quite frankly are running your businesses, that quite frankly are the core of what you do?

Developer experience: you have to provide people with the right developer experience to make their day-to-day job a delight.

Jakes Olivier: Yeah. Can we expect to get excitement using old tools written years ago?

Anthony Anter: Let us be frank. You are getting someone and telling them: here is a tool that probably was written in the 90s. Go ahead and use it, versus what you learned in engineering school, which is VS Code, Eclipse, JetBrains, whatever your IDE of choice is, using Git, using modern practices, modern tools. You have to make that developer experience a delight. If you make the people excited about what they are doing, then what they are doing will be cool. It does not matter what platform it is. It does not matter what they are pushing.

Jakes Olivier and Anthony Anter

Jakes Olivier: We identified culture way up front as something important to address across the entire organization. So do not go chasing waterfalls.

Anthony Anter: I am not going to talk to you. Who knew he had a voice like that? I thought we were just here for a tech talk.

Jakes Olivier: We have to get the mindset right, and that really goes across the entire organization, all stakeholders, not just developers. It is really about instilling that continuous improvement, continuous learning culture. We have to plan for it. We have to give them the time to go and innovate and learn new things.

Anthony Anter: Do not be waterfall. Do not let your teams say: oh, we are agile. We have a sprint review. See, we have a scrum meeting. We have a Jira project. We are agile. You have to change the culture. You have to go all in. You have to get that mindset correct. Once you do, then again, this becomes just another team, just another technology.

Anthony Anter

Anthony Anter: Process. This is the way it always works. This is the way we have always done it. Oh, well, the business wants blah blah. Oh, you know, we are different. This is different. This is a different platform. That is not how the mainframe works. The standards say...

Where I came from, there was a process to see the process, to review the process, to change the process. You had to dig eight processes deep to get to the process people were talking about. This is daunting. It can be daunting. But again, we did not let it stop us. We did not let it daunt us. It is going to be peeling an onion, and I use this specific term of an onion versus a parfait, number one because it is a funny analogy and number two because there will be tears. You are going to have to peel those pieces off. You are going to have to peel it. But when you get to the core, you have got something great.

We are not advocating throwing out all your processes, throwing out all your standards, throwing out whatever. But review them. Look at them. See where they make sense. There are going to be processes where you say, you know what, that actually is valid. That is a valid point. I am in an auditable industry. I have to answer to certain people. Security, audit, compliance, SOX. Automate those.

You have to change the processes and make the processes work for you. Since I have been here, I have listened to talks about people that make audit their advantage, automating compliance, automating change. There is no reason all these things cannot be automated, and there is no reason that you cannot make them better. Do not let "well, that is always the way it has been" be the reason that you do not do that. You can make this your advantage.

Off-topic Q&A Interlude

Unknown speaker: I like getting my feature flag tooling in early, just like I like getting my testing tooling in early. The hardest test to write is the first test because you have to put all the tooling in place. So if you do that early on in the code base, it is already there, even if you only have just one test. Same thing with feature flags: put in the lines of code you need to initialize the feature flag tooling and then wrap something in it. That is it. Now you have it so that a month down the road, when somebody says, hey, we now need feature flags, great, just go and create that feature flag and put it in the code. I believe in doing it really, really early in the development process.

Unknown speaker: All right. Any other questions? No? Thank you. All right, well, thanks everybody. I hope you enjoyed our talk.

Jakes Olivier and Anthony Anter

Anthony Anter: Did you run into that at your place yet?

Jakes Olivier: At the end of the day, it is a shared platform, and we actually saw these guys were doing some good DevOps practices for years already, so it was not all bad. But we used value stream mapping as a very valuable tool to help us identify what is important to our stakeholders and development. What are the pain points? What is important? That helped us prioritize and make sure what we deliver to them is ultimately going to add value for them.

Anthony Anter: That is a great point too, because this is not just me or Jakes, or my team or Jakes’s team, sitting down and figuring out what is best. You have to bring people into this. With those value stream mapping sessions, you bring the stakeholders in and you work together to get this. I think I am smart. My mother says I am a bona fide genius, but that does not mean I am smart enough to figure out how the end-to-end of this should work. Involve your stakeholders. Involve them in these process discussions. Get security involved.

Security is just a guy like you. He has got a job. He has got deliverables. He has got things he has got to do. He is not this big scary monster that is there just to tell you no. If you involve them up front, you will find there are a lot of benefits to that and they will work with you.

Jakes Olivier: When you do something to someone, they kick against it. If you do something with someone, show them the art of the possible, show them the benefit, they buy in very quickly.

Anthony Anter: That is right.

Jakes Olivier

Jakes Olivier: Technology is obviously important. Digital transformation for us was critical. We could see fintechs coming into the market, disrupting our business. So for us to really stay relevant, continue competing, and actually reach that goal of being number one, we had to transform.

Modernizing your development environment, modernizing your applications, using modern tools really is important. Your architectural runway: we have to enable teams, so we have to put those things in place. We definitely wanted to leverage what we had already. We initially started with our distributed teams. Digital transformation was a priority, and that is where we started, but we quickly realized we cannot leave the mainframe behind. Any organization, any chain, is only as strong as its weakest link.

Then metrics and measurements. Initially it was very difficult for us to justify this change. It was difficult because we did not have proper metrics. We did not have proper baselines. We really had to estimate the gains that we were going to receive through this. But ultimately afterwards we now have a wealth of metrics and information that can quickly point us toward where we can do further improvement, where we may not be doing as well as we thought we would.

Anthony Anter

Anthony Anter: If you cannot measure it, it does not exist. That is something my boss told me years ago, and it has never not been true. I know that is a double negative, but that is as good as I get. If you cannot measure it, it does not exist. You have to measure at the beginning, measure during your process, and measure at the end, because your boss, your organization, is going to come to you and say: where is my return on investment?

You asked me to spend X million dollars. You asked me to spend all this time, hundreds of man-hours maybe, to implement this. Where is my return on investment? How have I paid this off?

This goes back to developer experience. Modernize your developer time. We save technology toward the end because people, process, and culture come first. Then you worry about technology. This is not a tooling discussion. This is a mindset discussion.

The last thing we want to talk about is technical debt. Let us be real. We are talking about applications that have been running in production since I graduated from high school, and you can see by this gray hair that was quite a long time ago. Decades. Decades of time.

How do you deal with that? How do you bring DevOps to applications that have been running unchanged 20 years? You have to break down those monoliths. It is not going to be easy. If it was easy, we would not make as much money as we all make in this room, and they would have anybody doing it. It is not going to be easy, but it is worth it.

You are on platforms that have to be highly available because they are the core, they are the essence of what a lot of your companies do. Those change windows shrink. You not only have monolithic code, but you have very volatile, highly slender change windows. So you have to make sure that your systems run well.

Legacy challenges: you have older code bases, older technologies, older architectures. There were no microservices, or at least no one writing anything back then was writing them in microservices. You have to break all these down. You have to take it and modernize this application. See, the platform is modern. I will argue with anybody in here: the z15 and the z16 are as modern as any cloud platform out there. They have all the capabilities that you need. What needs modernizing is application development and application architecture. That is what needs to be modernized.

You have to pay off this technical debt. You have to bring in concepts like automated testing. You have to bring these things into the platform, into your application development on the platform, to get that true benefit. None of this is going to be easy. These are all going to be challenges, but they are challenges that are worth it. They are challenges that will pay off. They are challenges that will actually help you get better. My examples are from a few years back, but Jakes has a more recent change he just went through at Nedbank, and he can tell you about the benefits that they saw.

Jakes Olivier and Anthony Anter

Jakes Olivier: Saving time and money, obviously. We increased the flow and got higher quality. That was expected. But just to put it into context, we measured a simple piece of work and looked at what it used to take a developer to get his changes through the lifecycle, doing the governance to get it into production. We are talking about a very small piece, not complex work. As the complexity increases, the gains just shoot up significantly. This is really a very modest estimate that I am going to give you: we reduced the time it took a developer from five hours to 18 minutes. It is like a 90% savings, and it took away a lot of those mundane tasks that are energy-draining, demotivating, like documenting change control to go through a heavy governance process. Today they do zero documentation.

Anthony Anter: Hang on. They do not do zero documentation. It has all been automated, right?

Jakes Olivier: Correct. Yeah. The documentation is still there.

Anthony Anter: I think I saw one of the compliance guys just pass out. It is still there, but it has been automated. It goes back to process: automate the process.

Jakes Olivier: We also expected quality to increase initially because of the test automation capabilities that we were delivering. But we actually saw that the adoption of automated testing was lagging. So the quality improvements we saw were simply by automating the deployment process, removing manual intervention and human error. Just that in itself already showed benefits.

We could obviously release software quicker because it takes developers a lot less time. It frees them up to focus on innovation, the things that really matter to them and add value to the organization. We could see faster recovery with fixing forward because it takes us less time, or with improved rollback capabilities. It is easy to fix forward or roll back.

Anthony Anter: Think about that from a two-speed IT point of view. If you are in a bank, if you are in an insurance company, if you are a utility, if you are a municipality, I guarantee you have two-speed IT even if you do not know it. Your web, your mobile, all of that is a facade. It looks like a western town in the front; you go behind and it is nothing but cardboard and two-by-fours. The backend is where your data, your systems, your money is stored, your systems of record.

How much value is it to bring that to the same level of change and sophistication that you have on your cloud? To be able, within a couple of sprints, to push a change through? How many people feel like sometimes their changes are handcuffed because I have to make this change, and it involves one of these backend systems, and I do not want to go to Don and ask him because he is really grumpy?

How much value is it to your company to change that mentality, to not be handcuffed? To take a piece of code, have your engineers work on it, check it in, and the system decides what the best platform for it is. Have your engineers do what they love, what they are designed to do, what makes them happy, and what brings you the most value. How much is that worth? Do not answer the question. I want everybody to ponder that on the flight home.

Jakes Olivier: Just to add on to that, now we have got mainframe teams involved in the organizational planning. We have got a proper framework that gives the organization that single heartbeat. We can plan together. We can manage our dependencies. We can deliver together.

Anthony Anter: That is right.

Jakes Olivier: DevOps really gave me a sense of purpose, not just professionally but personally as well. Initially, we approached DevOps as a role. We were a bunch of DevOps engineers creating pipelines. We realized we could not sustain that, so we changed our role to coaching and really became servant leaders for our application teams. Now we are all about empowering each and every agile team within the organization. We really want to not just empower the organization and the different teams, but really empower individuals as well.

Anthony Anter: I have talked to multiple companies, multiple people who have gone down this path, and their story is similar to our story. There is a blueprint. There is a plan. So do not think that DevOps is just for the cloud. Do not think that DevOps is just for your newer platforms, your newer teams, your newer developers working on the latest technology. DevOps is for anybody.

Twenty-five minutes on the dot. There we go. Ladies and gentlemen, did you see that? I want you to write that down because I have been in a lot of presentations, and they always have ten sentences left at the end. We hit it right on the mark, baby. Thank you.