Stefan Simenon

I'm Stefan Simenon from ABN AMRO Bank, a Dutch bank. I'm going to share with you our DevOps journey: why did we start, how did we start, what did we do the past years, where do we stand now, and what is our way forward?

I'm head of the IT Tooling and Software Development department. We are responsible for making sure we have the right tools in place and the right pipelines. We're helping the agile teams in improving their software quality and security. We have coaching available for agile teams in becoming more mature in CI/CD, and we are starting an initiative about cloud-native development.

ABN AMRO is the third largest bank in the Netherlands. About 22,000 internal employees; including external employees, I think about 50,000. Headquartered in Amsterdam, little presence abroad. We have 5,000 employees working in IT, and we have transformed into an agile organization. We have about 300-plus agile teams.

How did we start? In 2015, we became aware that we needed to do something. We recognized we were slow. We did a CI/CD pilot, and with a successful outcome.

We used to be a waterfall organization. We had lots of waiting time at that point in time. It took a long time to start up projects, took a long time to get resourcing for projects, and once the resourcing was completed, it took a long time to build and test and deploy the software.

We had multiple discussions with the management team about this issue, and at some point in time, we made clear to the senior management how long it would take to define and design and implement a Hello World application. And if you would follow all the processes, it would take more than six months to get it into production.

At the same time, we saw our competitors going faster and faster. We saw more fintech companies popping up, and also fintech companies dealing with banking products. So fintech companies became competitors as well.

So we were slow. Other parties became quicker and quicker. We said, "We need to start a continuous integration, continuous delivery initiative."

So we started a project. We implemented a small pipeline based on Jenkins, Bitbucket, Nexus Repository. It took about three to four months to get it done. Also, this was done as a project. It took a really long time to start a project in this phase.

We had five eager agile teams, Java and front end. They wanted to experiment. They wanted to do new things. They were experimenting with the pipeline.

And once we had the pipeline, once they had the applications onboarded, we saw tremendous benefits. They were much faster. They had much more transparency in what they're dealing with, into software quality.

And after this, we showed the benefits to the senior management. The teams showed the benefits, and the management was very impressed by it. So the management said, "Okay, we need to do this across the whole bank."

In 2016, we started a CI/CD program. CI/CD was so important that it became a strategic theme, and we continued improving our pipelines.

At that point in time, we had four themes.

The first one was agile. We were transforming into an agile organization. It took around three to four years to get the transformation completed. Last year, we completed it with a reorganization. So everybody within ABN AMRO, they needed to apply for their existing or new job. Also in the selection process, a lot of focus was given upon the agile mindset, and that was very important.

In the new organization, we have much less project management and much less project overhead, and there is much more focus upon making and delivering software.

The second theme was service-oriented architecture. We have a complicated IT landscape within ABN AMRO, and we want to make it more standard. We want to make the systems of record loosely coupled from the systems of engagement. Therefore, we need a service-oriented architecture, including an enterprise service bus. That was the second agile theme, to make sure that we have the right application in place to make the front end loosely coupled from the back end.

The third theme was cloud. We had procured a private cloud within ABN AMRO, also for standardizing our infrastructure and getting the cloud ready. Getting the applications to this private cloud was a main focus. And in that year, we also started thinking about a public cloud, about a hybrid cloud situation.

The fourth one was continuous integration, continuous delivery: making sure that everything gets automated, making sure that the right pipelines are in place. That was our fourth theme.

So what did we do? We started a CI/CD program, which was conducted in two ways.

"Pave the way." That is making sure that all the technical prerequisites are in place to enable the teams to do CI/CD. So making sure we have the right tools, making sure we have the right pipelines, making sure we have the right infrastructure prerequisites. We believe that it's 20% to 30% of the CI/CD journey.

But the most important piece is the "make it happen" piece. That is all about the changing mindset, behavior, making sure that processes get simplified, making sure that the organization changes the right way. Because the way how you're going to develop in a DevOps environment differs totally from the way how you develop in a waterfall environment.

We believe this is the most difficult piece of CI/CD, and we also believe that you need the right tools as a prerequisite to do the organizational change.

We started small. We started with CI in the lower environments. We started with front-end Java technology. Once we became more mature there, we started with other technologies, like Microsoft, like BPM TIBCO. Once we got more mature over there, we started with technologies like mainframe and mobile banking.

The idea is to enable the teams to do automated production deployments from pipeline as code, zero-touch deployments. It's up to the agile teams themselves whether they're going to do it. It's also, of course, dependent upon their own objectives, but we would like to enable the teams to do it fully automated.

We had lots of focus upon the awareness. Especially the management, they were made aware about CI/CD. We organized a whole CI/CD summer program with all kinds of events. There were IT leadership programs with CI/CD as a focus item on it. We had external speakers and parties to make the people aware.

It's important to have the senior management buy-in. And the senior management, they do not only need to say it, they also need to show it. They need to advocate it. They need to spread the message continuously into the organization.

What we also did do is that we have set up various communities. We have Java communities, we have front-end communities, we have mainframe communities. In these communities, people come together to share information, to share successes, to share failures, to learn from each other. And we also try to get external speakers within ABN AMRO to show the message across the whole organization.

Very important is also coaching. We have agile coaching, but we also have CI/CD coaching, because teams are not always aware. They need some help. They need some coaching to get more mature.

And we also have all kinds of e-learning modules that the people can learn. Very important is focus on learning and focus on improving your IT environment.

We say 70% of an engineer's time should be handled by making business functionality, 20% should be freed up to improve the IT for IT environment, and 10% is reserved for learning. We believe that the DevOps environment is a continuously changing environment with continuously evolving new technologies, and people need to learn. People need to improve themselves. Ten percent of ABN AMRO people's time is available for learning.

What did we do in 2017? We delivered mature CI/CD pipelines per development technology. We included security in our DevOps pipelines. I will show it later. We set up an IT for IT organization to support the tools, to enhance the tools, and we implemented a hybrid cloud situation.

What are the tools we have implemented? For agile, we have implemented Jira, Confluence. We grew from 2,000 users in Jira to 10,000-plus users in two and a half years. In the beginning, we had some stability problems with Jira, also because of a non-standard way of working. So we standardized the way of working, and therefore Jira got more stable. And a few months ago, we moved Jira to AWS, and now Jira is very stable.

We have all the tools in place for CI/CD. I will show a separate sheet on this later. We have the tools in place for software quality and for secure coding and for managing our open source libraries, and we have all the tools in place for test management.

Meaning that we have the tools in place does not mean that teams are fast from an automated point of view. They need to automate the test cases themselves. But we have the tools, the pipelines made available to enable them to automate.

In 2017, we also did a selection process for release and deployment management. We selected the tools from XebiaLabs, both XL Release and XL Deploy. We implemented the tools. We implemented OTAP environment for XL Release, XL Deploy, and we implemented standard CD pipelines for Java WebSphere, open banking, and IIB, which is the enterprise service bus I was mentioning about. And we are continuing enhancing our CD pipelines by XL Release, XL Deploy tools.

In the meantime, we have more than 1,000 XL Release users, more than 100 applications onboarded for automated deployments. Some of them already deploy to production, but some of them only to UT. They keep on improving.

The benefits of XL Release and XL Deploy are, first, that with XL Release, you can define templates so that you can define all activities in the CD phase. You can see all manual activities, so you get good overview what you can automate, but also what process you can get rid of.

We see improved security because of automated deployments. There's no manual deployments required and no high privilege access on OT and production environment.

And we see a decrease in release lead times achieved and improved time to market.

So meaning that we have automated pipeline with XL Release, XL Deploy means that teams can do zero-touch deployments, but it's dependent upon their own maturity, upon their IT landscape, how to make use of it. So they need to refactor their code. They need to automate their test cases. They need to get rid of redundant processes. So the tools are only a help for them to become more mature.

This is how our mid-range pipeline looks like. We have Slack, Jira, Confluence for agile management. We use Jenkins for managing our CI. The output of CI will be a deployable artifact, which we store in Nexus. And from Nexus, XL Release will pick it up, and XL Release will trigger the change management process. They will trigger the functional testing. They will trigger the deployments.

And once the deployment is done, the CMDB is also upgraded in an automated way by interface between XL Deploy and ServiceNow.

We use Splunk for logging. We use AppDynamics for application monitoring. We use HP tools for testing and DataMaker for test data management.

So as you see, all the tools are available to enable the teams to fully automate their pipelines and to do zero-touch deployments.

The sheet I showed, that's for the Java, for the mid-range, for the front-end pipeline. We also have pipelines in place for mobile, for BPM TIBCO. For Microsoft, we're using VSTS. That's a Microsoft SaaS pipeline. For commercial off-the-shelf, we're going to implement Ansible. We're also investing in a mainframe pipeline based upon Compuware tools.

We have a Docker pipeline to check whether Dockerfiles are secure, so that Docker containers can be moved to production in a safe way. And we're also making a Python pipeline. It's not yet on this sheet.

I already mentioned DevSecOps. Security, that's a key focus for ABN AMRO. We are a bank. We want to be reliable. So we also want to make sure that our software is reliable.

We have implemented SonarQube, Fortify, and Nexus Lifecycle in our pipelines. SonarQube, that's for improving the code quality, for improving the time to market. But Fortify, that's a tool we use for secure coding, making sure we detect issues like SQL injection, of course, cross-site scripting, very dangerous things.

And Nexus Lifecycle, we can check whether our open sources we use are vulnerable or not. If we encounter severe issues, then Jenkins will block the pipeline, and the developer is enforced to fix the issue before being able to continue. And only if the code is secure, then a deployable artifact will be generated, which can be taken up by XL Release to move further.

This pipeline only helps for applications that change. We also have security issues in legacy applications which do not change. And we also have made them visible, and we're starting a program now with senior management attention to fix these issues.

So security, very important for ABN AMRO. And to see that DevOps is helping us in that journey. We can make the issues visible in the early stage, and the developers can detect the issues in the early stage and fix the issues, even in the IDE.

We have set up a program, and we have the pipelines in place. We have the awareness in place. So many people within ABN AMRO think, "Oh, we're ready. We're ready with CI/CD DevOps."

But that's not the case. In order to do automated zero-touch deployments, we need to refactor our code. And that means that we need to deliver small changes in a frequent way instead of big changes in a non-frequent way. From the waterfall method, we have monoliths in place, big pieces of code, and we need to refactor them into small pieces in order to facilitate team autonomy and in order to make sure that teams can do automated deployments.

So refactoring this code, doing the decomposition, doing trunk-based development, integrate quickly and often, these are key items for CI/CD, for DevOps. People were not aware in the past, but after they got more mature, they got aware. And now our mobile teams, our internet banking teams, they're putting lots of effort in decomposing their code, and you see them becoming more and more fast, so to say.

And also testing, very important. Not only to automate the tests, but also to do a shift left, to do automated unit testing and to practice test-driven development. Also, this is a major change compared to the old way of working.

What did we also do as part of the reorganization? We have set up a separate IT for IT organization. So we eat our own dog food. These are DevOps teams. We have teams for Jira, Confluence, Slack. We have an application deployment support team. We have software logistics teams that maintain the CI tools.

These teams take care of the tooling upgrades. They implement new tools. Once there are incidents or once there are questions from the users, they will help the agile teams.

Very important is the pipelines team. They develop standard pipelines, which can be used across the organization.

We say to the teams, "It's your responsibility to have a pipeline. We offer standard pipelines. You can use them. If you don't want them, you can make your own pipeline, but make sure that you build in version management. Make sure you build in security. Make sure you store the artifacts in the right tools."

Many people make use of the standard pipelines, and we develop the pipelines and enhance the pipelines together with the community. We take the feedback from the programmers to enhance the pipelines. It's not that this central team is making it themselves and telling the people, "You need to use this pipeline." We do it together with them according to an inner sourcing model.

The fourth point was the hybrid cloud. We have an IBM private cloud. We see that this is an improvement compared to the old situation, but we also see that public cloud facilities are missing. So ABN AMRO decided to have public cloud in place, Microsoft Azure and Amazon Web Services.

The platforms are approved by the ECB, by De Nederlandsche Bank, by the internal cloud approval board. More and more applications are moving there. Jenkins is running on AWS. Jira is running on AWS. Our Tikkie app, for the Dutch people, very widely used app, is also running on AWS. And many Microsoft applications and a few important finance applications are already running on Azure.

We want to scale next year. So we have set up a program to get further prepared. And in this program, we're also looking: what skills do we need? How do we retain and maintain the right skills? How do we deal with cloud-native solutions, microservices, serverless? These kinds of things we are currently checking and implementing in order to get prepared to get the important applications to public cloud.

What are the benefits so far? We see tremendous benefits. We see an increased time to market. We see improved code quality.

You see some examples of benefits in the agile teams. Mobile banking, internet banking, they went to two-weekly releases, while they had four releases per year in the waterfall period. And they are planning to release every day, every two days.

We see a team which doubled their velocity after one sprint of IT for IT only. And we see a team who reduced a build from five hours to five minutes. These kinds of examples we see.

Within ABN AMRO, you see teams who make good progress. You see teams which are learning, but we also see teams which are not yet mature, which need to pick up the journey. So also these teams we're going to help.

What is advice for growing up? Senior management commitment and involvement, I already mentioned about it.

Invest in reducing technical debt. That was the refactoring I mentioned about, the decomposition of the monoliths.

Very important is to create a safe environment in which failing is okay. We deal with partners from India, TCS, Infosys. We used to have penalties in place if they delivered projects late. We said, "We get the penalties out of the contracts, and we're going to work together as partners." So we allow them to fail, we allow them to learn. That's very important, that you create a safe environment for each and everybody.

Pitfalls. Do not focus on tooling only. It's important, but not the most important.

It's a very complicated journey. More complicated than many people think. And what we also say: do not focus on long-term plans. Do not focus on presentations describing what you want to deliver in one year or two years, because the environment is quickly changing. You get things on your way you don't expect.

So every quarter, we are going to check and decide what we're going to deliver next quarter. And based on learnings, based upon experiences, we do this kind of planning.

What's next? We keep on automating and improving our pipelines. It's a continuous journey.

We will further transform into DevSecOps. We are checking how we're going to set up our credentials management, how we're going to store our keys and secrets, and we're also going to build it in our pipelines.

Improving way of working. Mindset and behavior is a continuous journey you need to conduct. You need to spread the message continuously. The management needs to do it continuously, and the DevOps need to be spread as oil across the organization.

Facilitate increased team autonomy. You see that teams are still dependent upon central teams to do all kinds of things. So we want to give more autonomy to teams. We want to make sure that they have the right access, the right tools to deploy a component on their own without being dependent upon other teams.

Focus on public cloud, I already mentioned about it.

Very important is database automation. We have done big improvements, but the database is a little bit a forgotten area, also within ABN AMRO. So now we're doing a proof of concept with DBmaestro, and if that's successful, we're also going to do the database automation via our pipelines in an automated way.

And we invest in CI/CD metrics. We want to have the right metrics per team, per area, so that we can take decisions based on real data instead of feelings, so to say.

So you see a lot of things achieved, but still lots of things to be done. We're not yet ready within ABN AMRO.

And this was my last slide.

Q&A

Q: Hi. I'm interested in your use of XebiaLabs. I'm interested about what size is your application deployment team and how long did it take to onboard your—

A: Our application support team consists of nine to 10 people.

How long does it take to onboard? That is dependent upon what you want to achieve. But what we are striving for, that the teams start experimenting with XL Deploy, XL Release, and that together we will do small steps to improve.

If you want to have a fully zero-touch deployment, including all security items, then it will take longer than if you do it step by step, so to say. And my view is that we should do the last: improve step by step. Start with XL Deploy, start in lower environment, then enhance to ST, enhance with XL Release, build in security things. I think this is the way how we should do it.

Also in automating your CI/CD cycle, you're never ready. There is always room for improvement.

Q: What options are you considering for managing keys and secrets?

A: We're going to check the cloud-native tools, and we're going to check CyberArk Conjur, and we're going to check HashiCorp Vault.

Q: You said one of the things that you're now focusing on is to extend the automation to the database. Does your database development happen in separate teams, separate from one—

A: Yeah. So what happens, the definition of the database happens in the teams itself. But once a column or once a change is required, there is a separate database management team, and there are all kinds of procedures in place to ask them to do something. And then they make all kinds of scripts, which is run on a need basis. So there is a separate team for doing all this stuff.

Once we have automated it, that team is not required anymore. Yeah.

Further questions?

Okay. Thanks for your time.