Host Intro (Gene Kim)

I had the pleasure of meeting Ron van Kemenade, who was then CIO of ING Bank, when he spoke at the first DevOps Enterprise Summit in London in 2016.

I loved this presentation because, in so many ways, he modeled all the things that we want from a leader as we change our ways of working. I loved how he described how technologists had become order takers in the business, trapped inside of processes that required requirement documents, approvals to do anything at all, and a sourcing scheme that required seven calls to figure out what was happening for anything at all.

And then he described the journey to help unleash the full creative potential of thousands of developers across the ING enterprise. And what I loved about the talk so much is that mostly it was about helping people learn to co-create outcomes, not just with their fellow engineers, but with their colleagues within the business units.

He talked about their mobile application at the time, which he said was one of the worst apps in the App Store. Quote, "The only reason it wasn't worse was that because you couldn't rate an app less than one star." He talked about changing how development was done for their mobile application, which was a spark, and the focus was then how to create a fire.

My reaction after seeing him talk was how much better the world would be if every technology organization had the mindset of someone like Ron. So I was so delighted earlier this year when Ron reached out to me because he had just achieved something he was incredibly proud of and wanted to share it with this community. So here is an interview that I did with Ron a week ago.

Interview (Gene Kim and Ron van Kemenade)

Ron van Kemenade: Thanks, Gene, for the opportunity to reconnect.

Gene Kim: Sure.

Ron van Kemenade: And I'm proud to be back at the Enterprise DevOps with IT Revolution. So back then, you announced me on stage as the CIO of ING. Actually, last year, I joined the board of ING, globally, so I'm now the chief technology officer, which includes scale technology. Otherwise, I wouldn't be there. It does cover information security, data management, and actually payment and settlement services. So it's a bit of a broader domain, but I still cover my beloved technology domain.

Gene Kim: Awesome. And can you describe maybe how ING has changed or the environment that ING competes within has changed since 2016?

Ron van Kemenade: Yeah, we're still very much operating our wholesale bank in all the regions that we do, in the Americas, in Europe, in Asia, and Australia. We have slightly reduced our retail countries footprint, are focusing now very much on return on equity we make in every market we're in. So that meant we left some smaller countries. But we're still very much investing in our major markets, which includes, obviously, the Netherlands, Germany, Belgium, Spain, Australia, many more. And their technology is still very much supporting the growth of our business in country as well as cross countries with our bank technology platform.

Gene Kim: Wonderful. So when we talked a couple weeks ago, you had shared something that you've been extremely passionate about. In fact, you even hinted at it in your 2016 presentation, and you had described how you actually hit an amazing milestone recently. Can you tell us about that incredible achievement?

Ron van Kemenade: Yeah, I still remember I made a couple of promises in 2016. I remember, Gene, I talked about automating the evidence that we are in control over our risk and security, and we have very much done so ever since. And we could talk a full separate item on this one.

I did announce that I believed my engineers could build and could create value actually for ING by taking away all the heavy lifting from the teams that need to create value for our customers, so creating features for customers and employees. And all that heavy lifting has been consolidated into what we call a bank technology platform.

And I still remember, I almost announced we would sooner or later decommission our mainframe, and we actually started in the Netherlands with decommissioning that mainframe, which was the by far largest one supporting the wholesale bank, the bank in the Netherlands, some of our group applications.

And I even got a bit criticized by some people, experts in the mainframe domain on The Register, as you may remember.

Gene Kim: Oh, yeah.

Ron van Kemenade: But, yeah, that's really the thing I'm very proud of because we took it on and we actually made it happen, right? And we made a difference there because we, three weeks ago, completely decommissioned our mainframe and literally disconnected it from the network, powered it down.

By the way, it's quite a difficult machine to power down because it will typically ask you four times whether you are really sure, and then it will finally give up, and we did so.

Gene Kim: I kind of imagine, the scene in my head is these giant loud fans finally turning off. Was it as dramatic as that?

Ron van Kemenade: No, it is as dramatic as that. And even that, Gene, the machine, if you have fully shut it down, then the fans will continue because it actually doesn't trust you that you really meant to shut it down. So it keeps its right temperature for maybe like an hour or two after you have actually shut it down because it doesn't expect you to turn it on again.

Gene Kim: That's amazing. Those amazing engineers, always thinking about us. And by the way, just to help me confirm my understanding, this isn't like one application that was running on the mainframe. Can you just maybe describe the scope of what was running on that mainframe and what business processes it powered? Just to give us a sense of scale.

Ron van Kemenade: Yeah, so indeed, that's actually a very good question because it sounds so monolithic, and to a certain extent, the actual physical machine with all the operating system and all the utilities running on there, basically half of the bank was running. So our payment engines, many of our product engines, like mortgages, consumer loans, business loans, a lot of business processes related to those products. Our customer and agreement administration was running on it. Part of the financial accounting was running on it. So consider it literally the core of the bank back then in 2016. Like over 250 big applications. So not small APIs, which you could count as an application in itself, but really big product, customer, payment-related applications.

Gene Kim: And when we were talking, you had gone to extreme lengths to mention that you were not denigrating the mainframe technology itself. We both look in awe about those amazing amount of engineering to make sure that those things always stay on, can run those transactions. But it no longer was providing what ING needed. Can you talk about that, that it's not really about the mainframe?

Ron van Kemenade: No, it really isn't, right? And if anything, any engineer should only look in awe and admire the great engineering that a mainframe represents, right? It's one of the most long-living and fully supported and very much alive platforms in the world, still supporting a lot of enterprises, governments, harbors, what have you, railways, even armies.

But why then look at it and say, "We're going to take this thing out"? And it had nothing to do in itself with the mainframe. It had to do with all the applications running on it. Many cases, applications that dated back to the early 80s of last century, programmed in COBOL, some parts even in ALGOL or PL/I. You remember, Gene, Programming Language One, right?

And you don't get the best engineers anymore for those applications. So from a, how do I maintain a great engineering workforce, it wasn't good. Secondly, many of the people who knew how all these complex programs and routines actually worked had left the bank by now, right? People do retire, actually.

There is not a lot of very modern tooling around it to do fully automated continuous integration, continuous delivery, simply tooling that engineers don't endorse that much anymore. So the engineering experience was a good reason, the business continuity for the bank, maintaining the right knowledge, and last but not least, it had a cost benefit to the bank as well, like a couple of tens of millions of euros, which is really relevant, right? Because you'd rather invest that in modern and more value-adding applications and work than keeping it locked up in a platform which doesn't create that much value anymore.

Gene Kim: So our industry is strewn with stories of people who attempted to do what you did. Maybe not hundreds of applications, one application, and they spent tens of millions of dollars or even $100 million multiple times in their attempt to replace these applications that were written 40, 50-plus years ago. So what sort of maniac would embark on a path, trying to essentially replace hundreds of applications that remain mission-critical to daily operations? Can you talk about what sort of beliefs you must have had that enabled you to see this through that took over six years?

Ron van Kemenade: Yeah. So maniac is maybe a bit of a harsh word, but you need to be a believer, right? So you need to be a believer. You need to be convinced, first of all, about the ability of your engineers, because if you don't trust your people to embark on such an adventure, and if you don't believe that the organization actually has the stamina, the endurance to sit it out throughout the time, and your business might actually lose belief in this, is this really going to work?

So you need to trust a couple of things, right? But the main thing behind it is you need to have the strong conviction it will bring value to, in our case, our bank, let's say, but to any enterprise, any organization. And like I said, it does modernize your application landscape. And you need to refactor functionality into modern languages, Java or Scala or sometimes Python, whatever. You want to simplify your landscape and make it more robust, and make it more consolidated. So migrate all the products, the data that come with that, your customers, into your target state. That helps in maintaining a decent and simplified landscape. And in the process, you actually bring concrete benefits to your customers.

And if you have that belief, and you trust your engineers on this, and you get a bit of support from your executive board and your business colleagues, then you can do it. And anybody should actually do it, because nobody wants to sit on a pile of legacy that sooner or later people don't understand anymore. From that perspective, I believe any CIO should take on the adventure, and I can't promise it will be easy, but it's a pure necessity.

Gene Kim: And when we talked, you had mentioned, you used this word believer, so you were a believer, but I think one of the things common to these type of stories is that the mainframe software developers, they weren't believers. So maybe could you confirm what the people on the team tasked with replacing these applications, what did they think? To what extent did they believe it was possible?

Ron van Kemenade: So back in 2015, I was sitting down in a room, actually in Arnhem, which is famous from the movie, A Bridge Too Far. So Arnhem, Netherlands, where our primary mainframe people were located, and they literally said, "Ron, you're crazy. This can't be done." And it wasn't pure resistance that they didn't want to do this. They simply looked at the sheer amount of work and couldn't believe we would actually do this, right? Which, in the end, you can't argue, because if people have more an emotional view on this, you can't argue with emotions. You can only say, "Okay, let's try," right?

And gradually we'll gain confidence while we take out these applications one by one by one, and then people actually start, "Hey, this might actually work. If it works in mortgages, why don't we do this in business loans as well? And if it works for financial accounting, well, we might as well try some of the payment flows," right?

And so we did, and I can tell you, Gene, along the way, we encountered a lot of things we hadn't foreseen, complexities. We had overlooked people that were retiring and took some knowledge with them that we desperately needed. Funding that dropped away, other business initiatives that took priority for one or two years over this, so... But still, we soldiered on. And in the end, you can see the light literally at the end of the tunnel. Like, "Okay, it might still be difficult, the last mile, but we'll get through."

And people gradually gain more, and more, and more confidence, and then the last, literally weeks were still quite eventful, I would say. Because people say, "Yeah, we have done so many tests, but we have missed a couple of use cases, and are we really sure? And did we really test a year batch of the year closing?" And actually, we didn't. I said, "Yeah, if so, then we'll fix that in production," but we did test month and the quarterly batches, so...

And in the end, Gene, you take that final step. We'll migrate the last parts, and now everything is in production. We kept it running in shadow for even a couple of weeks, and then you finally say, "Now we're actually there," right? And the last moment, I can tell you, Gene, is the least eventful. Because then everything is gone, right? And it turns out that everything you did was more eventful than actually shutting it off.

Gene Kim: Yeah. And just to make sure I understand, so that uneventful part, you had run the last batch. You had done the last quarterly reporting and you had gained confidence that its job was done, and that you could finally turn it off. Am I understanding that correctly?

Ron van Kemenade: Yeah, exactly. Exactly. Yeah. In our case, the last one was a month batch, and that was validated, and it ran on two sides, so on the mainframe and on the applications migrated off the mainframe. You compare both databases. It turns out the same amount of money is on the left side as it is on the right side, because along the process, you can't create or lose money, right? That's kind of essential.

And then, indeed, people start literally... So that famous batch ran on a Monday, and then people say, "Yeah, be careful, Ron. We need to validate this, and that will take till Tuesday or maybe even Wednesday, and maybe we need to keep it running over the next weekend." So, people, they do care about our customers. They do care about everything they are responsible for, and giving them all the trust they deserve, encourage them a bit, and sometimes take tough decisions as in, "Hey, I truly understand your concerns, but I'll take responsibility and now we're going to go."

Yeah. It's only natural that people find it quite difficult to make that last step, right? Because as long as you don't make the last step, you can always turn back. And at some moment in time, you need to say, "Guys, whatever happens, if something breaks, we'll fix it on the other side," right? "But we need to leave this world behind."

Gene Kim: Yeah. How did that feel to run that last batch? How did it feel for you? How did it feel for the developers and operators to finally see that last batch complete?

Ron van Kemenade: Like I said, the moment itself was a bit uneventful from a technical perspective, but having these people together in one room, actually a bit symbolically shutting and going through the shutdown procedure. We had a little drink with that. And people were happy, and honestly extremely proud.

I was standing next to a guy that I had briefly met years ago, and he was telling me... The guy was 63, so in two years he would retire, and he literally told me, "Yeah, Ron, I was there when the first machine entered the building, like 40 years ago in '82 or '83." And then he was the one that actually stood next to me going through the shutdown procedure, entering his password. So he literally went full circle, and I can tell you, the guy was not sad. The guy was extremely proud that he had been the one who guided us. He took it personal. He guided us through this whole process, that he was the last man to shut it down. And he said, "Now I can actually retire." So he felt this was a personal thing to him. But not from a sad point of view, but really proud.

Gene Kim: That is wonderful. And so what other benefits did you have in mind when you committed to this path, and what other benefits have been realized?

Ron van Kemenade: I'll tell you one story about... So we had our international payments running on the mainframe, again, very old COBOL code with batches running overnight, and then ingest it into another database, and then another batch would run, again overnight, and then the transaction would be booked for the customer, so two days. And when they started, in this case, it was refactoring, so the same functionality programmed, in this case, in Java.

And then it actually turned out that there was so much obsolete code, and that whole throughput time of two days was actually not necessary, and we could offer our customers international payments that would literally be processed in 15 minutes. Which would be the fastest in all of Western Europe at that moment.

And then you see that when you talk to your business colleagues about why you want to take on this big monster of that 250 plus applications running on a mainframe, it's abstract to business people, right? But when you start creating these, I wouldn't even call them small proof points, they're quite big proof points, where you take out complexity, and again, complexity is something that resonates with nerds, but not per se with businesspeople. And then all of a sudden they get emails from business customers saying, "Wow, what happened? You processed my dollar payment in 15 minutes." Then it starts making them endorsing the whole thing as well.

You could say basically three types of benefits, Gene. The benefits to the customers, like I just gave you an example. And to be very honest, when we started with this, we did not per se have the customer in mind. Maybe indirectly, like with the second group of benefits, which is continuity to the bank, continuity to the organization, maintain a life cycle of your code and your technology. That would be the second big benefit. And continuity obviously is good for customers as well, but more indirectly.

And then the third category is, like we alluded to a bit earlier, the benefits to my engineering workforce. Because people deserve modern technology. Nobody wants to have PL/I on their CV, as an example. That was the third category. And then, of course, last but not least, which is beneficial, again, to the enterprise, were the cost benefits. Because in the end, maybe you don't have the best business case in the world, as in a return on investment, but you need to be accountable for how much you invest and how much you get out of it. So even the CFO needs to sign off.

So that's what we did, and we made our business case to the full degree, as in our annual recurring cost savings have fully materialized, which we set out to do only a bit later, because we initially thought we could do it in five years, and in the end, we did it in seven years. So at a moment in time, Gene, I had a bit of an impatient CFO at my side. "When am I getting the full benefits out of this?"

Gene Kim: Right. That's right. That's so good. So I know there are many technologists who look at this knowing that they face challenges that mirror the ones that you faced back in 2016, and maybe they've had similar aspirations, but maybe they haven't had courage enough to fully commit to this path like you did to the end. So what advice would you give to leaders who may be pondering now whether to do this or not?

Ron van Kemenade: I was thinking a bit, Gene, about your question because it would almost sound arrogant, because there is always a specific business context in which such a program would need to be started. And I believe it starts with the topic we discussed. You need to believe in this can be done. Because the first, I believe, mental impediment that many people will have is, like I told about these engineers back in 2015, and they literally said, "This can't be done. We'll never get there." So you need to have that belief and build it in your team, we can actually do this, because otherwise, don't even try.

The second thing is you need to be convinced about the benefits it will bring to your organization, to your people, the enterprise itself, your customers, et cetera. And I believe the most convincing part is you need to realize every day you maintain these applications running on the mainframe, every day you lose a bit of knowledge. The currency, the accuracy, you build up legacy knowledge. And that is something that every CIO should be worried about. And that's maybe the most convincing reason why you need to do this for the organization, and you need to have a bit of a strong back because it's heavy lifting, it's difficult work.

I won't make it any lighter than it is because it is literally difficult work, and you need some stamina for it and some stomach because it won't be easy. But I believe every CIO who takes responsibility for a large enterprise technology environment, think about your knowledge legacy, because sooner or later it will come... It sneaks up to you, and it will come back and haunt you, right?

Gene Kim: And you told me something that I thought was astounding. You said there was one particular problem of where to start, and your answer was startling. Could you share your answer when someone said, "Where do we start? It's just a plate of spaghetti."

Ron van Kemenade: Yeah. And the answer is start somewhere, literally. It doesn't matter where you actually start, but start somewhere. There is always a small end you can pull, one of the strings out of the plate of spaghetti, and I believe the parallel I made is, I believe officially the way to eat a plate of spaghetti is by twisting these strings around the spoon, right? And then you lift it, and there is this art of eating spaghetti, and it becomes extremely difficult. I never mastered it.

But there is a way of just using your knife and fork and cut all these long strings into smaller pieces, and then it becomes way easier to fully eat your plate of spaghetti because every time you take small pieces. And that's what I believe is a bit of a metaphorical way of looking at it. You need to cut it in small pieces, and if you do so, it doesn't really matter where you start. But don't try to do this in the most complex way.

Gene Kim: Yeah. By the way, the reason why I think that's so funny is that if you have just a homogeneous ball of spaghetti, I guess definitionally it actually doesn't matter where you start. The end outcome is the same no matter where you start.

Ron, I am so delighted that you could share this amazing story, and I love that, and I very much believe that you are so right. I genuinely believe that the DevOps Enterprise community are the people who will appreciate and respect this story the most. And I think it's just another example of how you seem to fully unleash the full creative potential of people in your organization. Is there any particular help that you're looking for these days? Something that you actually want people to reach out to you for?

Ron van Kemenade: I'm always open to challenges that people may have, people that are open to give their advice. I'm a big believer in sharing knowledge and open source software, so this is open source to the world, right? How we did it.

Gene Kim: Wonderful. Ron, thank you so much for the time, and I so much look forward-- I will find an excuse so that we will have our next interaction in far less than six years.

Ron van Kemenade: Oh, that's great. I'm looking forward to it. Thanks a lot.