Jan-Joost Bouwman and Leon Janson

Leon Janson: Welcome everybody. We are at DevOpsDays, and we are from a bank, from ING. We are going to tell you something about the risk-aware engineer: a practical example of how we are trying to make our engineers risk aware. Before that, we should introduce ourselves.

Leon: I will introduce Jan-Joost. Jan-Joost and I have worked together for quite some time, since around 2010, in different roles. He goes to many conferences and now speaks at them: first about agile, then DevOps, and in his last role he joined me as a risk manager. I challenged him to talk about risk in relation to DevOps. He used to talk about ITIL, but he cannot talk about ITIL anymore. You can follow him on Twitter; he travels a lot, goes to places other people do not go, and likes birds, especially exceptional ones.

Jan-Joost Bouwman: Thanks, Leon. Leon has been my manager several times within ING. He is probably the only person within ING who can sort of manage me.

Leon: Trying.

Jan-Joost: All others have failed horribly.

Leon: But he is still here.

Jan-Joost: I am still here, so it must be their failure. Leon is a family man with a wife and two kids. He enjoys spending time with his kids at their sporting events. His son plays soccer and his daughter plays hockey, and he coaches both of them, which would be complicated for me because they have different rules. He has a long time of experience in risk management in different functions. For the past couple of years, we have worked together in the risk and compliance area we are in now. He is also the person within ING who invented the Risk Awareness Days, which we will talk about today.

Leon: Before that, let us introduce ING.

Jan-Joost: ING is a financial institution. We are the biggest consumer bank in the Netherlands, actually in the Benelux. We operate in over forty countries. A lot is wholesale, but we also have retail banking activities in the rest of Europe and in Australia. We have about 50,000 employees, 38 million retail customers worldwide, and about 500 DevOps teams within the Netherlands working agile DevOps. First I want to talk about our agile transition.

Leon: We are based in the Netherlands, and we focus on the domestic bank Netherlands. That is where our agile journey started: in the domestic retail bank, quite a big part of ING and a big part of the profit. That is where agile and DevOps started, and we are even doing BizDevOps.

Jan-Joost: Our agile transition falls into three phases. We started in 2011, when developers started working agile with Scrum, sprints, and a method of work. That helped us deliver software faster and deliver much more software. But it did not help us get software into production quicker, because there were still handovers to ops before we could get value out of the software.

In 2013 we started working DevOps. We made dev and ops people work together in joint teams under one manager. For the first time, dev and ops people had a shared goal. They were responsible for maintenance, development, and production. For ops people that was new, and dev people were also responsible for production and incidents.

Leon: Both of us originate from the ops side of the organization.

Jan-Joost: Although DevOps gave us faster release to production, we were still lacking business involvement. IT management promoted agile ways of working to business people and took them to companies like Spotify and Google. Suddenly the business said, everybody should work agile.

Leon: Let us do it in one big bang, like a big program, like waterfall. That was not the start we envisioned.

Jan-Joost: But still they started. In 2015 we started working BizDevOps. The third picture is basically the Spotify model, with changed colors. We have a tribe structure where people with a shared common goal work together, for example mortgages, business lending, or savings. In those tribes there are squads of people working on a single proposition for clients. The squads consist of customer journey experts from the business side, ops people, and dev people. In general the product owner is from the business side. Direction comes from the business perspective, but the squad is also responsible for reliability and security.

Leon: The benefit of putting these people together in one team is faster communication, faster development, faster generation of ideas, and better understanding. In a big company one team may work on one part and transfer it to another team without enough communication; that creates delay and waste. We get that out of the way by making small pizza-sized teams, maximum nine or ten people, preferably fewer, working together on one topic to create value. It really worked.

Jan-Joost: Horizontally we have chapters, where people from different squads work together to share knowledge, for example chapters on security, multi-factor authentication, databases, Linux, or other topics.

Leon: Next we talk about shift left. Jan-Joost has been going to conferences for years, so he is ahead of the game.

Jan-Joost: When you have an agile transition, you want to deliver fast. In a traditional V-model of testing, testing is done at the end of the cycle, and you never have enough time to test. It helps to shift testing left and start working test-driven development: design your software with tests in mind. When you work agile, that becomes even more important because you have short two-week life cycles. You have to think about how to prove your software works before you build it, otherwise your sprint is already gone. Testing at the end of the sprint does not work.

Leon: I use the same story from a risk perspective. Twenty years ago, testing often happened at the end of a project, or in production, or not at all. Then test processes came up and we brought testing step by step to the left of the cycle: from production signals, to UAT, production acceptance, unit test, component test, and now test by design. Think about risk in the same way. Five years ago, especially in ops, when did we look at risk requirements from a regulator? Often at the end, or in production, or after a security incident. That is too late. We promote following the same structure as testing: do it faster and bring it more to the left. If you think about risk in your design, it is cheaper, faster, and gives you a better chance to be in front of the hackers. Hackers take advantage of common mistakes: easy openings in applications, firewalls, or vulnerability patching. If you do not take that into account in the beginning, you will always be too late. Hackers are more numerous than your IT people and have more time.

Jan-Joost: Within ING, we call shift left security by design. We emphasize to new employees in boot camp that they have to think risk by design and security by design before they start coding. Automation is essential because, if you do not automate testing, you can never do enough. You can automate secure code review and even penetration testing, but automation is only as good as the input you give it. You have to continuously update it with the latest vulnerabilities and hacks. It can only work if you have a risk-aware mindset in your engineers.

Leon: That risk-aware mindset is the struggle, because risk awareness and mindset are culture. Our auditors, external auditors, and the European Central Bank said the underlying root cause of CAAT findings and IT risk challenges was risk awareness. We had to meet a KPI to be within risk appetite, and we asked how we could ever meet that hurdle. Culture is hard to measure. You can measure how many vulnerabilities are patched or how well you control vulnerabilities, but culture is hard. If you want risk by design and shift left, you need to be in engineers' minds. Engineers are the builders of your house. They know the weakest spots in their software better than anyone. They know where they took a workaround, shortcut, or forgot something under time pressure, especially when they built it yesterday.

We could not measure culture, but maybe we could measure effort. Together with the CIO of the domestic bank Netherlands, we decided to organize Risk Awareness Days. One day is dedicated to risk for all engineers. The CIO led by example and said that if you want to be a good engineer, risk needs to be part of your craftsmanship. If it is not part of your craftsmanship, you cannot become an expert engineer.

We had to organize 2,000 to 3,000 people to work on one topic in one day. We had two and a half weeks to organize it. We started the day with a webcast, using the big screens teams already had for monitoring and progress. Then people worked at their desks on risk topics, with guidance. We added games, including Kahoot, to create energy. We also bribed them with orange T-shirts, food, and drinks. If people saw the orange T-shirts, they knew they could get food somewhere. At central locations, the orange T-shirt people had to ask: what kind of risk or security item did you solve today? If someone answered, they could take candies and drinks.

Jan-Joost: Two mantras are important for Risk Awareness Days. First, celebrate failure, because only if you identify failures can you improve. Second, celebrate your successes by doing things right. Celebrate failures, but do not discuss them on the train home. We are a bank and we like open discussions, but not too open.

Leon: At the second Risk Awareness Day, we had a webcast in the morning and another after lunch to bring energy back. During the CIO webcast with an external expert, I was behind the screens and got an email from him with his name spelled incorrectly and a strange message. I received signals from other people and realized he was spoofed. A team had taken on the challenge, found a gap in mail security, and thought it was funny to send a mail from the CIO mailbox saying the Risk Awareness Day had ended, to 3,000 people. That was not in line with the disclaimer. Fail big, but share small. They were mentioned in the webcast. I put a question on the screen so the CIO could read it, and he said, we have some enthusiastic people, well done. But they had to report to him that afternoon and got a different speech.

Jan-Joost: Key success factors: it is a dedicated day and all engineers take part. Since this year we also include the business side, so all people within Market Leaders Netherlands, the domestic bank of Netherlands and Belgium, participate. Clear ground rules: celebrate failure and celebrate successes. Specific topics are very important. Tangible, actionable items help engineers because if they can engineer the problem, they understand more quickly why it is important and what it means to their application. It also helps if they can connect it to their own application or personal life, such as handling a personal iPhone with privileged ING information or passwords. Make it concrete, understandable, and actionable, not fuzzy or high-over.

Leon: Concrete topics were key. At the first Risk Awareness Day we were more afraid of failure than success. The risk discussion with the second line was always about unclear guidance, paperwork, and documents nobody thought added value. We did two things. First, we challenged teams not to discuss the rules and guidelines of the second line or regulator. Instead, ask: how can I make this bank more secure today? Patch one vulnerability, close one firewall, reset one password. Second, we made it concrete: credentials, meaning passwords. In ING's internal wiki we searched for the word password. You expect to find the word in instructions, but not the real password together with a login or admin password on a Confluence page anybody within ING could access. It may say it is a UAT password, but it gets worse if you can log into production as well.

The CIO broadcasted in the morning that he had a list and, if he were a manager or product owner of one of those teams, he would look at the applications because he found a password and logged in that morning. People started working because the CIO was pointing them out. He also challenged the Git team and security team to ask how many passwords were embedded in code repositories in clear text. Some people started scripting, and by the end of the day they had a script scanning repositories, pulling out passwords, and emailing people personally that they had passwords in their repository. Then we built a process around it, and it became part of automatic code check-in, giving a signal if a password is in there. If you do not solve it quickly enough, you can report to the CIO again.

The more concrete you make it, the better it works. Sometimes topics are closer to world peace, and the energy goes away. In pizza-sized teams, people often said their team did not have passwords and that another team did. Then the quiet introvert in the team said maybe they should look at their own code. That gave the quiet person a voice and showed where they were failing on passwords. That made the bank more secure.

Jan-Joost: Key ingredients include learning: a webcast, content guides with information and instructions, workshops where experts speak, and walk-in sessions where people explain how to do things. It must have fun elements, such as Kahoot quizzes with silly answers, games, gamification, coding competitions, and hack-the-box competitions. We wrap it in a standard program: a webcast, two hours of work, lunch, food and drinks, re-motivation after lunch, a code risk quiz, two more hours of work, and then a retrospective to get topics for the next one. We have organized twelve in total. The first ten were IT-focused, and the last two were cross-border Netherlands and Belgium, business and IT. That is a challenge because some topics become world peace again, making it hard to keep energy in it, but reviews are still good and the target audience is shifting.

Jan-Joost: Quick wrap-up. Short life cycles of DevOps and continuous delivery mean security by design and shift left are essential. Security is just as important for the DevOps team as coding or testing, or it should be. The role of risk managers is changing toward coaching about risk rather than checking risk items. Automation is an enabler, but you need a risk-aware mindset. Our approach is Risk Awareness Days, with relatable and relevant topics, webcasts, workshops, actionable items, and actual user stories people can complete in a day, reducing ING's risk that day.

Leon: Most important: make it fun, keep the energy in it, and bribe them while you can. Thank you for your attention. If you have questions, look for us in the speakers booth.

Closing Video

Video speaker: A great day where people are truly focused around their craftsmanship, about playing Champions League in a bank, IT in a bank. Getting everything up on the table and seeing how we need to improve.

News clip: Companies across the globe now racing to recover after a massive cyber attack. Credit reporting agency Equifax says the data breach involved names, addresses, Social Security numbers, birth dates, and driver's license information.

Video speaker: It is the IT craftsmanship and the one way of working. In the end, it needs to be proactive. The question is, could it happen to us? I think the honest answer is yes, it could happen to us. What do we want to do? We want to prevent this stuff, right? That is something I really want to make clear. Thank you. Have a great day.