Log in to watch

Log in or create a free account to watch this video.

Log in
London 2017
Share
Download slides

Backbone of the Swiss Financial Center Goes DevOps

As the infrastructure backbone of the Swiss financial center, SIX provides a comprehensive set of critical services with utmost reliability, among them the Swiss Exchange and the Swiss Franc system and is highly regulated.


As the company definitely had not waited for a major cultural transformation, choosing an ideal business line to start, involving doers with leadership skills and managing the stakeholders was as critical as implementing technology and processes. The transformation journey at SIX shows the cultural challenges, where true leadership and trust replaces command and control behaviour and teams take over responsibility while clear roles of the individual disappear. But exactly within these changes lies the key to tackle ever moving constraints and waiting times to deliver faster and more effectively aligned to business needs.

Chapters

Full transcript

The complete talk, organized by section.

Robert Scherrer

The backbone of the Swiss financial center. Maybe not the place you would choose to start a DevOps journey. Maybe yes, with some touch of madness, maybe.

So, Aristotle is famous as the tutor of Alexander the Great, but he's not so famous as a tutor of lean practices and principles. But he already said you need some touch of madness.

SIX, we are here for a while, and our customers, the banks, usually were founded as banks, or at least not as a tech company. Even Amazon was founded as a bookstore, not as a tech company. SIX was founded as a tech company. So you could call us the first fintech startup in the world.

So it was our original destiny to provide technology to banks. The technology available, which was telegraph, which was TV broadcast, which was online terminal to mainframe computers, satellite uplinks, whatever was available, we used it to the benefit of our customers.

So in the last 40 years, we more and more switched to straight-through processing, and we are still the only place in the world where stock exchange trade is cleared and settled finally within seconds.

SIX in figures. I always like this half trillion of francs flowing through our Swiss franc system. It's a number just exceeding my imagination.

So we are not only the Swiss franc system, we are Swiss Exchange. We are an international securities depository. We are processing cashless payments all over Europe, and we are providing comprehensive information about financial instruments on a global scale.

And you heard maybe this morning John Smart talking about 19,000 tech guys. I'm extremely proud that we do not only do one business, that we do all these businesses with only 700 engineers and operators.

And continued success is our legacy, but the message from our CEO is absolutely clear. We have to adapt faster. We had to cut lead times.

Now, it's not so easy to get faster in such an old and process-oriented company as ours, but there was hope.

So Dave, who is our head of ops, and I, as the head of dev, we built a small core team. And when we did this, this was during the 2015 Rugby World Cup. And so we were impressed by the All Blacks, and we are impressed how they show this dedication, this team spirit in the haka before all their games. So we've chosen the haka as our motto. We wear these black polo shirts. So every time you see a black polo shirt, it's one of the core team. And we show this dedication, and we tackle whatever we can when we see which is not yet DevOps. So that was our start.

Already in the beginning it was clear for us that DevOps is about people. It's not only tools and technology, and people with the right mindset and attitude.

So where to start? Of course, we had to do infrastructure, tools, technology. But this was the fun part. Everyone wants to do this. We had to change architecture. But it was from the beginning our goal to cover all the existing systems and not only try DevOps on a fancy new web portal.

It's definitely harder when it comes to processes because processes are just there and you have the people that think you shouldn't change them. But nobody really knows why they are there and where they actually come from. So we find the origin, the original reason, and then we try to adapt.

For the organization, we wanted to change the mode of working first, and only after that adapt the organization in a way that it stays out of the way of delivering.

And last but not least, we have to build up skills, and that's not possible without people with the right mindset and attitude.

But how to bring this to the people? How to educate? There are nicely designed e-learning lessons in the intranet. Do you have e-learning lessons on the intranet? Do you like them? You don't?

Well, our tutor said you have to touch the heart, too. Is your heart touched by e-learning? No.

Well, you remember what Aristotle said about the touch of madness? We were mad enough to write a theater play and to bring the core team as actors on stage in front of hundreds of employees.

That's where I really experienced what stage fright means. You have to imagine a dark room. You see nothing. Sold out to the last seat. You hear the murmuring of the people, you don't see them. And there was only me and Dave on stage. Spotlight went on, and we started squabbling about whether dev or ops brings more value to the company.

And so the storyline developed. And in the end, there was the whole core team, you see the black polos, on stage transferring this one message: We build it, we run it, we love it.

And believe me, in our company, we talked about DevOps for days and weeks after this theater.

And we never abandoned this on-stage approach. Still today, we show our DevOps achievements every few weeks on stage in front of audience. And we still attract audience, and we attract more audience than written status reports attract readers. So this is our way of showing how we progress.

Now, spirits were raised. Okay, but does this already help?

Well, our tutor told us to search for excellence in repetition. And you can see it with the All Blacks. They would never have won two World Cups in a row without repeated daily training. No sports professional would go to the most important tournament of the year without daily training.

But that's exactly what IT professionals do when they go to their yearly release. That's exactly what they do. So that's not what we wanted to do.

Now, the answer is simple: go towards a continuous mode. Now, this is easy said but difficult to do.

So what we really did, we have chosen one existing product and one greenfield project. We involved them in the core team and started supporting them in automating release, in automating testing, in containerizing, and so on.

Now, you can say one product, one project, it's not a lot. But we used a trick. We started a competition so every team can win a HAKA Award for DevOps achievements. And the best winners are invited to DevOps Enterprise Summit in London. Big applause for them.

So now what happened? We converted DevOps adoption into a race. So before our sponsored project had container deployment into production, another team had. So all these things came up out of nothing to win these awards.

And I remember when Oli, our system engineer, he had to provide a base image for Red Hat Linux with Docker, more or less for one team first. And shortly after that, he called me to his place. He showed me his screen, the Kibana dashboard, and there was this counter of Linux machines already running Docker, and it showed the number of 80. And we had no clue which teams are doing what with these 80 servers with Docker, but we know the race was ongoing.

So now we scale up. We build up an OpenShift scalable platform. And if you look at this schema, you see if you bring something not so complicated as OpenShift to a very old and grown enterprise, it gets immediately complicated because you have to blend into the existing network zones. You have to be able to restrict some ports to run only on certain hardware. So it gets complicated, but that's what we do.

And now we are starting migrating applications to OpenShift.

Now, migrating applications, sometimes it's only reshuffling of components, maybe adapting, logging, and so on. But sometimes it's migrating from legacy platforms like COBOL and Tandem to Linux, Java, whatever.

This picture here shows our newest ATM platform. And ATM is a business that we have run for decades on Tandems. And it's a continuous business. It never stops. It's highly reliable. It's great security, encryption, and so on. And you see now hundreds of containers running with smaller services scaling on a container-managed platform.

But that's the fun part. Technology is the fun part.

With our technology, we now reach the limits of our control framework, but as well of our organization. And there you need definitely more courage than in technology to come over this resistance.

SIX is in financial business in several countries and therefore has to fulfill requirements, comply to standards in several countries from several organizations. And it's not that DevOps makes things worse. On the contrary, I'm absolutely convinced that we are bringing more transparency, that we bring better auditability, that we increase security.

But that does not help if the compliance officers, the auditors cannot understand. And they are people that are interested in the sustainable success of our company, so that's not the evil ones. So we have to explain, we have to cooperate with them, and we have to go forward.

What we found out, it's not the external requirements that restrict DevOps, it's the internal interpretation of what we have to do to comply which restricts DevOps.

So as an example, we had to rewrite the security framework of our company. You know that's a thing, it is owned by the chief security officer and it's approved by the board. So that's where we needed not only the courage, but the good relationship to the CSO to be able to do that.

It's even harder to tear down the walls of our silo organization. You know that's not a joke. That's our organizational chart. That's me, that's Dave.

So it's not only silos. We have separate line management, of course. We have conflicting goals. We have process descriptions that actually prevent collaboration.

So how to deal with this? What we did, we just started the first cross-functional DevOps team. So through all these silos, we have taken the teams together, physically together in one room, and started in one DevOps cluster, as we call it, to deliver faster.

Now, the question was where should we start? And we've chosen the business with the most functional changes per year and with the least regulated business. So we expected better acceptance, maybe more value, less restrictions.

And the success was really, really great. We tripled speed more or less immediately and business was absolutely in our favor of this.

Now, what we did then, more or less a half a year later with the experience of this first DevOps cluster, we defined more DevOps clusters in the same business line. And now this year we expand to more business lines and we define more and more DevOps clusters.

But that's not the end. So what we actually do now till end of the year, we completely tear down these walls and have a new org chart, which stays out of the way of delivery. A pure value stream organization, at least for the IT part of our company.

Now, the IT part, that's not enough. So for me, it's the start to a new organization of our company where business is docked directly to IT. And we have already first value streams where we have co-located product management, business analysts, and IT, and that's the way I envision the future of our company.

But org charts do not help if people stay with their mind in their silos. So we need the specialist know-how, but we need the attitude of a generalist and we need the ambition to learn additional skills.

Now, question was where do we find this and how do we promote this? We could take the intranet, make a page “Our Culture” and say, “Be T-shaped.” Do you have intranet page as “Our Culture”? Not? Yeah. Does it say, “We are innovative”? Does it help? No. Okay, it doesn't.

So I'm absolutely convinced that culture consists of the behavior you reward, of the people you promote, because this is visible and people see what's the thing they should do, and then culture becomes true.

But where to find these people? And it was interesting, we found them. We really found them in our company. We found the older employees that worked that way in the past, never really understood why they shouldn't do anymore. We found young employees that haven't learned to be so restricted to one role.

And my favorite, because of my age, we found these renegades, these holistically thinking renegades that never wanted to be restricted, and now we can give them the freedom so they can perform. But the learning is we found them, but we reward them, we promote them to build the culture of the future.

Now, remember the message of our CEO. Did he ask for cross-functional teams and value streams? No, he asks for speed.

So is this enough? Well, back to our tutor. He thinks not. And I like this one: “What lies in our power to do lies in our power not to do.”

And the most easy way to gain time, to save time and money is just to skip unnecessary process steps. And we have lots of unnecessary process steps.

I always think of this anecdote, and it was true, where the release manager asked the order manager, who asked the application manager, who asked the engineer about an estimation for a small change.

Well, the engineer came back about four hours later and told that he was absolutely confident that he's able to do this change within four hours. And how did he get this confidence? Well, maybe he did it.

Now, the application manager went back to the order manager, who went back to the release manager, who tried to plan this four-hour change into his release plan and found out it will not be able to do it for the next change window, which he signaled back to business, who was confused when it was live after the next change window.

So we really have these unnecessary process steps.

But when you have skipped all these steps that you don't need, it's time to go back to basic lean management mechanics, which in our case is value stream mapping.

So we pull people from the customer representative to the IT operator together in one room. We analyze work, we analyze the flow, and we start removing waste.

Only last week, last Tuesday, we ended a four-day value stream mapping, and we were able to take four weeks out of a major release plan. And even though it's now shorter, we can give the customers earlier access to test systems just with reshuffling, with removing waste, with reordering work, which wouldn't be possible without value stream mapping.

So what we are doing now, we are teaching internally people to be able to do value stream mapping, and we do more and more value stream mapping workshops because this is the way to go forward.

And it's important to focus on small improvement stories that can be done. It's no use to have a workshop where everyone says, “We want to change the world.” Next day, they are working and everything is like before. It's important to have these small improvement stories that are done within a week, within two weeks, and then the next retrospective to do more and more continuous improvement.

Now, I'm already almost at the end of our story. So I think we are on a very successful DevOps journey. We have this shared responsibility. And there, we reveal the secret that should frighten every head of a silo, be it Dev or Ops or whatever. We reveal the secret.

Developers care for stability, and operators care for features if they get accountable for it.

We have hundreds of server containers running. We have this OpenShift platform. We have more CI and CD pipelines, which actually fits into OpenShift perfectly, and we transform organization into a value stream.

But that's an inside view. That's our company that we transform. But we want to do more.

If you look at this picture, see the black polo shirts. This was on the DevOpsDays Winter Tour in May, this May, where people from our core team had a speech. The response was so huge that the organizers had to do an unplanned breakout session for 50 people just to be able to ask more questions, to get more knowledge about how we do our DevOps journey.

But we are not only on conferences. That's one thing. We invite companies, banks, insurance companies, public service, railway, whatever, to lab visits to show what we do. Because our original ambition was to transform our company. Our current ambition is to make Switzerland a better place for IT professionals.

And what I always wanted to say, our aim was never just to try it out on the web portal or mobile app. We always wanted to transform SIX into something faster and better.

So thank you for listening.