Robert Scherrer

My name is Robert Scherrer. I'm from Switzerland, as you might hear in my accent, and I'm responsible for the whole application development and test at SIX.

So what is SIX?

I'll give a brief introduction about SIX out of an IT perspective, then go into what DevOps is at SIX, and then come to the main topic of my speech, which is regulatory requirements versus DevOps discipline, giving you concrete examples, and then have time for questions.

So what is SIX?

We are Swiss Exchange. We have ultra-low-latency trading: 35 microseconds from trader to us and back to the trader, which is a world record. So our specialty is low latency.

But that's not all. We are the Swiss central securities depository, storing all kinds of values physically in a vault as well as electronically. And we are the Swiss franc clearing system of Swiss National Bank. So our specialty is secure transactions.

But there's more. We provide data from hundreds of stock exchanges, bringing incredible throughput to our systems. We have maximum throughput, and this throughput is growing faster than Moore's Law. So we cannot just wait for the next generation of hardware. We have to improve all the time. So our next specialty is high throughput.

And there is even more. We process payments around the clock. Highest availability, highest reliability.

So think of not just being available, but being available and processing each and every payment once, and exactly once. Twice is not good; not processing as well. So think of having to provide a change in database structure while guaranteeing consistency of data while being online. If you've tried this already, you know that there's some difficulty with it.

With all these specialties, we are the IT of SIX, and we are only 700 engineers and operators, and we are extremely proud to achieve so much with so few people.

SIX in figures: we are here for a while, almost a century. We are in 25 countries. And in short, we store huge amounts of assets, and we transfer an even more ridiculous amount of money each day.

So that's SIX.

Now, what's special here? We are not only a bank, we are a financial infrastructure. So we are not only regulated by financial authorities, we are also in strong cooperation with financial authorities. And we are located in many countries, so we are regulated by financial authorities in Austria, in Switzerland, in Luxembourg, and so on. And we have to comply to securities requirements from various standards organizations. And this makes the business a bit tough.

Now, why DevOps in this environment?

Our business guys are quite happy, actually very happy, with our stability, with our reliability. But we are convinced that we have to be more flexible and faster in delivering new functionalities, and we are convinced as well that DevOps is the way to do it.

So last year, we started our DevOps movement. And from the beginning on, we focused on inspiring people and not on technology and not on tools.

So we put together a small core team of a dozen employees, a dozen key players, and we started. We created a vision. We defined our principles. And last but not least, we wear the absolutely coolest T-shirts in the company.

And from starting with this core team, we inspire larger and larger communities of employees to scale up. And of course, we invest in tools and technologies. But that follows the cultural change, and it does not lead it.

Originally, we had the ambition to have 10% of our organization converted until end of this year. But already now, not even a year after the start, we are way beyond this goal.

If you look at this, two-thirds of the developers already wear pagers. One-third of the teams are truly cross-functional DevOps teams. So we are quite successful and fast from the beginning.

If you look at the left-hand side, you see a very simple KPI spider where we assess teams, where we try to raise the ambition to take part in the DevOps race, and where we want them to win the DevOps awards.

So actually, it's about inspiring and keeping and getting the momentum in the teams and not about commanding DevOps.

And of course we are in technology and tools. So a first container-based deployment to production is already in place. We have many applications, and don't talk about, let's say, public websites. It's about processing applications that have more than 30 deployments a month. So we are really in this transition to DevOps.

And the first pioneer value stream managed to incorporate business into the team. So we are not having DevOps anymore. We're having BizDevOps, or BizDevSec, DevWhateverOps. We have the whole chain in the team.

And when you look at not only the speed, but as an example, a number of SLA breaches within this application, you see a tremendous improvement because quality gets better when you're working together closely.

But with all this progress, we don't see us at the end of the journey. We actually see us at the beginning of a long, long journey of continuous improvement.

Or, being in good old England, as Winston Churchill would have said: "Now, this is not the end. It is not even the beginning of the end. But it is perhaps the end of the beginning."

Well, that's about our DevOps movement. And now to the main topic of the speech, which is how to bring DevOps on one hand and regulatory requirements on the other hand together and make them successful as one.

I start with the logic, how audits actually work, and then go to specific requirements, how we handle them.

Now, the most obvious, simple, straightforward way of audits is you have a requirement, you fulfill the requirement, the auditor comes in and looks for evidence that you fulfill the requirement. That's easy.

Unfortunately, in many cases, that's not how it works.

Compliance officers like to have internal directives that cover the requirements of the regulator. So you have to follow the internal directive.

So what happens in the audit: first, the auditor checks whether the directive actually covers the requirement, and after that, he checks whether you follow the directive.

So I definitely prefer fulfilling requirements directly.

Why is this so important? Well, one story is this one. Internal directives usually overfulfill requirements. Good employees tend to improve what they do, which is a good thing in general, but when a compliance officer improves regulatory requirements, it's much harder to fulfill them.

But that's not only the compliance officer.

If you look at our company, where we have many regulators, you have IT solutions that have to comply to a regulator. Now, if you want to have one internal directive covering them, you add up the requirements of all the regulators, and each and every system has to fulfill the requirements of all the regulators, which makes it unnecessarily hard.

And here comes the first aspect of DevOps into the play.

We in DevOps, we like scalable infrastructure. But that does not help.

In theory, in an ideal world, the auditor would accept a shared platform as a full-blown isolation between applications. Now, out of my experience, that's difficult to impossible to achieve.

While older technologies like LPARs on IBM Z series might be seen as mature enough as an isolation layer, new technologies like OpenShift or Cloud Foundry are not.

So you have the choice of having separate infrastructure pools or to implement measures to improve the isolation, and I think that it's much easier to separate the infrastructure pools than to do the other thing.

And here we are, for the first time, not only ticking off requirements, but here we are already in negotiation, in cooperation with the regulators, because you discuss what's allowed, what's accepted, and what you can do so that it works for both.

And this is the main takeaway from my speech. It's always negotiation with the auditor.

We at SIX, we are involving these guys early. We involve them even before we start changing things, and of course, way before the next audit is scheduled.

So the auditors can take the opportunity to follow our thoughts, to bring in their input, and to see a common solution grow, and that's what leads to positive audit reports.

So negotiate, cooperate with the auditor. Do not just try to tick off the requirements.

Now from this abstract layer, one layer down to the concrete requirements.

I start with segregation of duty because this is maybe the most popular one.

The laughs seem to give me...

Well, in the old world, we have these functional silos. We have the operation silo, the development silo, the architecture silo. So access to production is the job of operators. It was easy.

And now with DevOps, with cross-functional teams, that's not so easy anymore. This changes dramatically.

And, well, first, we looked at where comes this requirement from in our company, and of course, it comes from an internal directive. You shouldn't be surprised after my speech before.

And what can we do now? So we are looking for where this requirement could come from. Is there an external source to this? And we haven't found one so far, to be honest. We haven't stopped searching, but we haven't found one so far.

So what's the next step? The next step is to amend the directive. We are now looking for a new formula how to put this into the directive that it's not so harsh anymore. And maybe we can also negotiate a special treatment for DevOps teams.

But what's the reason we can negotiate? And the reason is DevOps brings massive improvement to auditability.

And here we can start. When we replace manual interactions, even manual interactions done by people from separate units with separate roles, when we replace them by automatic mechanisms, it's a huge improvement to auditability, to traceability, to whatever auditors like.

Even more, when we have immutable servers in production, so that they get replaced by a new immutable server when we have a software change, we can reduce manual interaction into production to a minimum. Another thing that auditors absolutely love.

So stay confident that DevOps brings so many advantages to auditability that we can stick to it also in a regulated environment.

The key mechanism is the following. We accept, and we fully accept, that there is a role of dev and ops. But we take the freedom to define, at any given point in time, who is to take over the role of ops.

So technically, this works the way that all the employees with the necessary skills have an inactive admin account. And like in case of fire, everyone is allowed to call the firefighters, all these employees are allowed to push the button to release the break-glass procedure and activate their user profile.

Now, and this is something auditors love as well because it's not all ops guys having access to production anymore. It's only the one guy in the ops role in case of accident that has the rights to go to production.

So another thing we improve with DevOps.

And now to something much more sophisticated.

PCI, I don't know whether everyone knows, PCI is the organization representing the payment card industry like Visa or MasterCard, and the objective of PCI is to prevent misuse of the card payment infrastructure.

So I've taken the requirement of code review. What does this mean? This means you must not deploy code without a prior code review done by another engineer.

Now, in DevOps, we want to have continuous deployment, so what do we do?

Now, even here, DevOps helps. If your tool chain is automated, you can include vulnerability scanning into your tool chain. And there actually is a market for tools that find vulnerabilities.

Now we are done. Fine. Let's go on.

Unfortunately not, because I cut the scroll on the right-hand side a bit too early. The full text makes it clear that automated checks cannot find malicious code in all cases. Actually, it doesn't find code in many cases.

Now, how do we handle this?

Our application development process has now an opt-in clause, which means a change in a critical part, like a booking system, is marked as to be reviewed. And then, as you can see, it has to be reviewed.

Saying that, if we don't mark a change as to be reviewed, we have continuous deployment. It works.

Now, of course, no auditor would accept this if we could not provide evidence that we actually mark critical changes as to be reviewed, if we could not show evidence that we actually review the code. So it's of course a balance, but it works, and we have continuous deployment also in card processing.

Now, coming to the conclusions. How to survive if you want to have a DevOps initiative in a regulated environment.

I have three key takeaways.

First, involve the compliance officers, auditors, whoever, early in your plans. Discuss with them. Get their input. Accept their input. Work out solutions that work for them too, and then you will get a positive audit report.

Second, track down internal rules, directives, controls, whatever, to their true external source, if there is any. And it's not always any. And amend them if necessary. It needs a bit of courage and it needs backing from the management, but you can amend these things.

And third, and maybe the most important, be confident. DevOps works. DevOps is good. DevOps brings many improvements to auditability, to security, traceability, quality of your systems. So DevOps is good and you will have resistance, but never give up.

And being in a country that once was part of the European Union, and to take a quote from a former prime minister: never give up on something that you can't go a day without thinking about.

If DevOps is an initiative you drive by heart, then don't give up. There will be a way to survive.

So, coming to my point where I could need help from the community. This is how can we show the value of our IT? How can we show the value of software development and not only the cost of IT to top management?

So if you have any ideas, feel free to contact me, and thank you for your attention.