Amine Boudali

My name is Amine Boudali. I'm the delivery lead for the platform stream on the core banking program, and I have with me...

Jose Quaresma

I'm Jose. I'm with Accenture, and I'm the DevOps lead in Denmark.

During this session, we will take questions at the end, please.

Amine Boudali

So in the coming time, we will be talking about our journey throughout the core banking, how we implemented the platform, and our challenges and how we tackled them. Also, how we eased the pain that we were facing.

So, a brief introduction to Nordea. Nordea is the largest financial services group in the Nordics. We have around 11 million customers, separated into 700 towards retail or private customers, and then 700 corporate.

And if we look at the history of Nordea, it was a collection of mergers and acquisitions, which actually led us to a relatively complex process landscape, but also application landscape. In some of these countries, you have around four, five, or even more core banking systems that are more or less doing the same thing.

Therefore, an initiative has been triggered, which is the core banking program in a larger initiative, which is simplification. Currently, this is the largest transformation program in banking in Europe at the moment.

The focus here is on delivering incremental and frequent business value, but also positioning Nordea to be the digital bank of the future. And how we do that: we have partnered with Temenos as a software provider and then Accenture as a system integrator.

Jose Quaresma

A little bit about where we started. Since the beginning of the project, that was almost two years ago, there was a big focus on these guiding principles that we're showing you here. And you've probably been hearing about quite a few of these throughout yesterday and today. But we wanted to tell you a little bit about them and what was the status in relation to these principles.

So automation was a very big goal for us. And this was both on the environment provisioning, on the test automation, and also on our CI/CD pipeline.

Then we also had this goal of having everything as code. And here, it's both for the environment configuration, but also, again, for the delivery pipeline. So we really want to be able to have everything in the repository, and so being able to always replicate things and have better control on the changes being made.

And finally here, we'll talk a little bit about the short feedback loops, where we really wanted to enable the developers and testers to get quick feedback on whatever they were doing and what they were testing. And this would be, in our view, really helpful to get them to learn fast, to fail fast, and really to help them moving forward with their development and with their testing.

So a little bit of our situation. BC, and here BC we're talking about before containers, not the other BC.

This work was mostly done in the first one and a half years of the project, and there was very good work done here already.

In our automation side, I wouldn't say we are in CI/CD, but we are good on our way there. So we do have continuous integration. We have automatic build and deployment to our development and test environment. And one of the things that we're focusing on now is really pushing that further up the environment so we get full advantage of the work we're doing there.

From everything as code perspective, from that principle, we do have configuration management in place. And here we are using a custom-made solution at Nordea, and it's not quite everything as code. But again, it's something that we're striving towards.

On this side, so for example, so that you get an idea, if we want to provision a normal core banking application server, we would have to first order the VMware servers from one of the teams, and then we would get some Puppet scripts to install that baseline WebLogic. And then we're using this custom-made solution for the more, I would say, application-level configuration of WebLogic. So what does the application need? What are the different deployment artifacts that are being installed, and what are, for example, queues, et cetera?

From a perspective of the short feedback loops, we do have daily build and deployment. So this is something that we're very happy with, considering the complexity of the system. But we do have some challenges that are preventing us from taking this further into the scenario where a developer makes a change, and then that would be triggering build and deployment.

Those challenges are some of it on the build not being as fast as we would like, and that's something that we are collaborating and pushing forward. And also there are some intricacies of the deployment that actually require us to at least restart the WebLogic servers when we are deploying. So if we would have build and deployment for every time there's a change, we would run the risk of having an environment that is more often being restarted than actually up. And that wouldn't really help with the short feedback loop part.

Amine Boudali

So continuing on our journey, here are the top, so to say, challenges.

So as we spoke a little bit about this long environment provisioning, as we are at the beginning of the program, there is a lot of proof of concept. So that needs a lot of environments to that in very short time. And as Jose was saying, if we have to organize and orchestrate a lot of unit in order to provision that environment, it takes a lot of time. So here we are talking about months or weeks in order to provision one full-fledged environment.

Couple that to the complexity of the deployment and orchestration, which the product requires downtime to bring that up. So talking about these feedback loops and shortening them, we would have a lot of challenges when it comes to the frequent deployment and the frequent development on top of this platform.

Talking about fragmented environment configuration, there is no full control of the whole stack, which means that provisioning from this unit might lead to different outcome dependent on the time, the humor, or even the state of that person who does the work.

Also, when thinking about a core banking system, this is not your system of engagement kind of application. This is the core where everything ties in. So you have upstream integrations coming from your channels. You also have downstream integrations to your accounting flow, data warehousing, and things like that. So it's really the heart of the body in there. And there is not only how we can stand that up, but how can we simulate and also mock these integrations.

Talking about shortening feedback loops to developers and testers, here I want to focus on this is not only to develop and test the current state, but also the future. I'll talk a little bit more about that when we come to how we ease the pain.

So taking the first one, long environment provisioning, we went from weeks and months to under one hour. This was amazing for us. One hour. Here, we are not talking about a database provisioning. We are not talking about an application server provisioning. We are talking about a full-fledged environment. One hour by using this solution. We're pretty happy with that.

Thank you. Thank you. Appreciate that.

On the proof of concept kind of activities that we are having here, we had a better lifecycle management. Standing up an environment and then decommission it, you would need to go again to these units to decommission it or reuse it, in which case you would need to bottleneck or to sequence your proof of concept. While with this, we could actually spin up environment like this, use it for the proof of concept, no dependencies between the proof of concept, and then decommission that when we want it.

Complex deployment and orchestrations. Here, for the first time, we are able, by using this product, to do live deployment. This comes from one hour of downtime or a few hours of downtime to zero downtime. Of course, this is not production ready. We still have small things to iron, specifically from a business perspective, but we would be able to do live deployment.

And here, when we are talking about live deployment, the product comes with its own modules, and then we are adding actually our own development and customizations or configurations on top of that. And that needs downtime whenever we need to deploy that. By this solution, we are able to basically swap the customization part in no time.

Fragmented environment: this is a full infrastructure as code. So when we are talking about an environment, when we are talking about the developers and testers and also the teams that are developing on this platform, they are also able to help us to improve the environment. So they can actually put some pull request or merge request, and then we review that, and we take them in. So this is more to bring them into the world of how we provision the environments.

How to integrate. Here, what we are using is we are combining the service virtualization technology. Who is here not aware of service virtualization? Just show of hand. Yes. It's basically mocks. It's just, let's say, context-aware mocks.

So what we are doing here is not to trigger the mock, build the mock, and then throw it away. It's more of reusing those mocks in order to build that end-to-end environment. So now we are not talking only about the core banking system, but we are also talking about mocking all the surrounding systems as to bring that end-to-end visibility and end-to-end feeling to the platform.

This here, you can think about it as taking a heart, making that heart pump, testing it, and then plugging it under another body, and off you go for sprinting.

Shorten feedback loops. With the ability of providing that end-to-end integration to the core banking system, we were also able to do frequent deployment to the development environment, but also concepts such as time travel.

This concept, what does it give us? It gives us the possibility to basically do the end-to-end, the end-of-year reporting, the end-of-month reporting, or, for example, interest accumulation ahead of time. So we don't need to wait for that time to do it. We are able to basically fast-track the time from today and do those type of testing in order to ensure quality to the product.

This is where we started. This is where we are today. Of course, there are a multitude of improvements to the platform, but also to the way we are working with this platform. And here, Jose will walk us through the platform itself.

Jose Quaresma

So right now, some of you might be thinking, "Okay, this looks really awesome, but how did you do it? We want to know that part." I hear it exactly here, so very good.

So our answer, and here we're not by any means saying that it's the answer, but it's the one we found and we came up with, it was to use Red Hat's OpenShift platform to start this transformation. We had a team of around five people, or I would say two teams, because some of the people were more focused on installing the platform. And then we had two, three people more focusing on migrating the, I would say, old setup of the application into the new platform.

And here, the migration was very much a lift and shift migration. We didn't want at this time to be thinking about which technology stack we should really be using in this platform, but more: let's grab the technology stack that we have right now running. Let's take this one, let's get it up and running in the platform, and let's see what does that give us, and then take it from there at a later stage.

So this took us around half a year. And what we ended up with...

Was a remote that doesn't work. Okay. That's all right.

Was a setup where we have a project, so a core banking application project in the OpenShift sense, or you could also call it an environment, is mostly contained of three containers.

So you'll have an Oracle Database container, and that one is attached to persistent volume to persist the data. You'll have a WebLogic container where the Temenos core banking application is running on, and then you'll also have an Oracle Service Bus container that is interacting with the T24 system, in this case, with the WebLogic container, to trigger the integrations and the services being used there.

Currently, we have around 30 of these projects running in our development and test environments, and they're used both by development and test teams to do, for example, test some features that they just want to play around with, or as Amine was saying, the test teams use it for the end-of-year reporting or testing and using these time travel features.

But we also have a lot of projects being used by, for example, part of our database team or in our side that are focusing on the database performance, and they're using one of these environments to really play around with different settings in the database and see how the performance, what are the performance outcomes of that without actually disrupting the other teams and the development and test teams.

So here on the bottom right, this was just to illustrate the live deployment part. And here we are combining the built-in features from OpenShift with the application. And what you have is that this is a deployment in progress, and the one you see on the left is the container that is currently running, and that's where the traffic is being routed through. But then you have a container on the right that is being deployed with, it has the update, and that one is starting.

So what OpenShift is doing here for you is that while the container on the right is starting, the traffic will go through the one on the left, the old one. But then once OpenShift sees, and you can code that, when it sees that the new one is ready and deployed, then it will actually shift the traffic to the new container and kill the old one. So this is the way we went about to achieve live deployments and really be able to be disrupting less the teams.

Okay, so one thing to note is this is not... It's our answer, but it's not the final answer. So we do have new challenges, and we're actually happy for that because if there were no more challenges, then it would be kind of boring, right?

So there are new challenges and things that we are now starting to focus on, and one of them is that there was a certain lack of awareness within the different teams of the platform. So the teams are very busy working on the different features and the things that they have to release. And then we came in and we have this platform with all these new features, and it's not an easy way of like, okay, so how do we communicate with the teams? How do we inform them of this new platform and the things that they can do with it?

Another very important thing is this big paradigm shift, and this was something that I also felt myself was that, and I was the one very excited about this, so I'm guessing that the other teams that are more not so familiar with the technology might feel it as well, is that it is a big paradigm shift to go from treating your servers as pets to treating them as cattle, to steal the famous analogy.

So here, it's a lot about if you're doing some manual changes in the servers, they might be gone in half an hour. They will be gone whenever you do the next deployment. So really having that mindset shift, it is really important. It is something not to forget or to overlook.

But the good thing about this is that it does provide more focus on having fully infrastructure as code, because if the developer needs something and it's not coded in the repository, it will not be there. So maybe it's there now, but then once we do a new deployment, it will be gone. So that was something very important and very useful for us as well.

And the third thing that we would like to mention is that, yes, we are pretty aware that we do have some heavy containers running on the platform. So it's not like the system is heavier than before, when it was running in a more standard, let's say, VMware kind of environment. But it is heavy containers. It doesn't allow us to take, I would say, full advantage of a containerized environment, where you have very quick way of loading environments, and then it's easy to do rollback and to bring up a new container.

Here, we do have the containers that are slow to load, that are quite heavy, and I would say their size is also fairly big. However, we are using the layering capabilities of Docker to really focus on reusing as much of the base configuration as possible. And then the changes that we have done more often, for example, by the developers, from a containers perspective, they are able to reuse the base configuration. So we're not rebuilding and we're not using so much space in general.

Amine Boudali

Moving on. This is potentially some food for thought. I know that pronouncing food while we are waiting for lunch is not maybe the best, but please bear with me here.

What we are looking, going forward, what we will be looking for is basically how we manage our users, customers, but also people. When do we actually say that this feature is ready for you to use? We've had some challenges, and that is back to the awareness situation that we had.

But also looking at processes and tools. Currently, the platform is in a dev and a test environment. What we would like to bring it to is a production-ready by 2018. Yes.

But there, we will need to tackle some of the security auditing compliance processes that we have and see how we can manage those in order to either fit the platform into them or find a way to basically bring them at par.

When it comes to tooling, for example, we talked a little bit about these heavy containers. We would like to talk about more different components using different web apps or different databases, but also experimenting with new ways.

For example, we are going into a three-data-center setup, and we would like to use the platform to simulate the delays that that can cause and what impact does it have to the application. Also, by using the technologies of service virtualization, we are able to simulate delays that will impact the core banking system and how that will impact the customers.

And of course, bringing this all together...

Maybe.

Yeah. Is also how we balance this. As you saw, the team that is focusing on this platform is five people, as Jose was mentioning. Of course, this team is embedded in a larger stream, but still, we would like to see how we can balance this to have that efficiency in providing the platform for core banking system.

We are open for feedback. Please, this is our first steps in using the platform and also experimenting in the industry and the component with using the core banking solution. So if you have any feedback, please come forward. You are more than welcome.

Jose Quaresma

Yeah. And getting back to the production-ready in 2018, we do hope that we can come back next year and tell you all about how we did that and how we achieved it. So that would be very good.

Amine Boudali

That'd be great.

Q&A

Jose Quaresma

So do we have any questions, comments? Please reach out to us. If you see that we've done something wrong, we would be really happy to hear about it now. If you have some suggestions on how we could be doing things better, we're really happy to learn and share. If you also have questions about how we did some of these things, if you want more of the nitty-gritty details, come to us as well. We're happy to talk about that.

Thank you.

Amine Boudali

Thank you.