Host Intro (Gene Kim)

Andrea Hausmann is Head of Program Strategy and Engagement for Credit Suisse, and she is presenting with Duncan Lawie, Head of Strategy and Governance. They are both part of the DevOps and Development Practices Group, responsible for helping elevate the state of engineering across the enterprise.

They are talking about their surprising revelations as they decided where to focus their efforts, and their refusal to be trapped into arguments about whether this tool is better than that tool, or even about tool standardization strategies. They reached some startling conclusions, which I think will be relevant to everyone in this community.

Personally, I am so delighted by where this journey took them. Please welcome Andrea and Duncan.

Duncan Lawie

Hello. To provide a little context for our conversation today, I would like to give you a brief introduction to Credit Suisse.

The bank was founded over 160 years ago and has about 48,000 employees today. We have 1.5 trillion Swiss francs in assets under management, as our headquarters is in Zurich, Switzerland, as you can see in the picture here.

We serve our customers around the world through three regionally focused divisions: the Swiss Universal Bank, International Wealth Management, and the Asia Pacific division, along with specialized investment banking capabilities from Global Markets and Investment Banking and Capital Markets. Behind that, we have our shared services divisions, which includes our central IT organization.

Andrea Hausmann and I, Duncan Lawie, are both part of that central group, with Andrea covering program strategy and engagement within the DevOps and Development Practices Group, which is part of the IT Strategy and Architecture organization, whilst I work in the same area, covering a broader space of strategy and governance.

Andrea, over to you.

Andrea Hausmann

Thank you. This presentation is about patterns, mostly cultural, recurring over and over again, irrespective of the job or the industry we talk about. We want to highlight the importance of critical thinking, as well as keeping our minds sharp and always being aware of such behavioral challenges.

There is not the solution to any of that. But we at Credit Suisse have tried to come up with a few best practices we would like to share with you today to overcome some of them. Unfortunately, we do not have the solution to all of that yet, so if you have something to share with us afterwards, we are happy to learn. What is important behind that is that we know these challenges exist rather than really having the solution to everything.

The storyline today of this presentation is: among my rather short career so far, there are actually two things both Duncan and I have worked in in the past that we both enjoyed very much. One of these is DevOps, obviously, and the other thing is working at the bar.

Very at the beginning of my professional career, I was working in Zermatt, which is a beautiful mountain village in Switzerland. You can actually see the Matterhorn depicted on the picture. We were at that time reestablishing the bar from scratch. We had to order everything from beer to wine to spirits, but also bar utensils, like what cash system do we use? What glasses do we buy? At some point, what corkscrew, for example, do we need to use as a barkeeper team?

Suddenly, we found ourselves in a discussion of barkeepers discussing which corkscrew is the best and easiest to handle. Everyone had their advantages and disadvantages in terms of stability, price, sustainability, ease of usage, probably how long they are going to last.

We could not come to a very easy discussion after all. But if you think about what really is the value which derives from a corkscrew, it is about opening a wine bottle, pouring it into the glass of wine, and bringing it to the customer. That should optimally be quick, and there should be no leftover from the cork in that glass. Very good wine nowadays does not even have cork sometimes anymore. And many people, because we were not solely a wine bar, ordered beer or gin and tonic, so this discussion is probably not that relevant in the end.

The ultimate goal we must keep in mind is to give the best service possible to the customer while we enjoy ourselves.

Fast forward a few years. I was graduating from university in business mathematics. I enjoyed numbers. I loved to calculate those big complex equations manually by hand. Unfortunately, I was born a few years too late, and this kind of job does not exist anymore. So I got myself a job as close to that as possible, doing the same thing but with a computer. I was hired as a data analyst in Credit Suisse, where I had been analyzing IT operational data, for example defining and making predictions about how healthy a server is and seeing when there probably was going to be the next outage.

Once I had my mathematical model verified, we had to give that to the customer. Obviously, my mathematical proof would not help the customer understand very much what I was trying to say in that proof, nor is it very user-friendly. All of a sudden, we had a discussion in the data analyst team: how do we bring that now to the customer? Discussions started arising from whether we are using Tableau or Power BI, or should we even build our own GUI? They all have their advantages or disadvantages.

Because we could not decide on whether we go with Tableau or Power BI, we decided to go with both. Me and my colleague started building in parallel. I went for the Tableau solution, and he went with Power BI. We had both been developing, and this process was actually very fun because we had been challenging each other and seeing how we progress, and we were seeing disadvantages and advantages of each of the tools. But what it ultimately meant is that it definitely delayed the shipping of my valuable information from mathematical proof to the customer, and it definitely cost way too much because we had two people building on two dashboards, even though just one was needed.

Do not get me wrong. It is important to think beforehand about what functionality my dashboard actually needs, and probably also think from a user perspective, not just from me as a developer perspective. But it is rather about what functionality the dashboard needs to have than which tool we ultimately use.

In the end, again, the end goal is to provide the best service possible to the customer, that in this case he can make informed decisions, and we should enjoy while doing that, and we should not delay the shipment of the dashboard.

This has been two examples about such behavior, but not in terms of DevOps, actually. DevOps was my last big change in career direction, and this is actually where Duncan and I met. When I joined the team, Duncan was, at that time, co-running the enterprise-wide DevOps working group.

The DevOps enterprise working group in Credit Suisse was a group of senior leaders thinking about the future of Credit Suisse in terms of DevOps. As it is an enterprise-wide, cross-divisional thing, each division was able to send their delegate to this working group. Each of them had sent their probably most senior or most technology-aware person to that working group.

And guess what happens if we put more than ten such leaders and technologists into one room? The discussion about tools gets even larger, probably more aggressive, and definitely more political.

Duncan Lawie

As Andrea said, when we met, I was already co-running the enterprise DevOps working group. But that was a journey that was a long way into my career journey. With over 20 years of IT experience, I was quite confident that I knew where my career was going. I had a great deal of skills, and it seemed to me that when the term DevOps first came up, it was yet another one of those terms that consultants want to whitewash the world with, and it does not make a real difference at all.

But when I spent a bit more time looking at the topic and trying to understand it, I realized there might be real opportunities for me in my career, for better success in the teams I work with, and possibly even a bit more happiness for our organization. I think it is important for us to remember where we have come from as senior leaders, and to have empathy for people who have not yet made that much of the journey, and to think about what will actually bring them with us towards the things that we think DevOps can really do for us.

As Andrea said, when we got together, the easiest conversation for us to have was about technology. Which is better, Jenkins or TeamCity? What is the right way to use Jira? Of course, these are the kinds of lively conversations that we love to have. There was plenty of heat in the room, but sometimes perhaps not so much light. It really gave us a feeling that we were talking about the important things in the world, but it did not solve all of our problems.

So we found that we also needed to talk about processes. Processes in an organization like Credit Suisse follow Conway's Law, unsurprisingly. Complex organizations, multiple divisions: we found that our processes were also shaped like that, and that the control handed from one part to the next across the organization. With the older processes, there were things that actually reflected previous iterations of our organization and were even more difficult to understand.

We did work within individual divisions around this, primarily around value stream mapping. But one of the other challenges we found here was that people's egos are built into these things. I had worked on how we did our database provisioning process. It was very important to me that we had learned this, we had delivered this, we had built something that was better than ever existed before. But actually coming back to that, taking my own ego out of it, and looking at what we could do next was a significant challenge. We need to recognize, again, that people have built their careers and feel that that is what they need to have and to show off forever.

So there was not a lot of low-hanging fruit in that space.

We also talked about people. In the people sphere, the group in Asia Pacific who had first started working on DevOps had made organizational changes. They had put people into dev- and ops-focused groups. They had also focused on how they manage their hiring process and how people should be rewarded in a technology organization. But that was not necessarily simple. Whilst the group of Credit Suisse as a whole understood we wanted to make these changes, the process of making those changes, the effort required, was slow, which meant there were not a lot of quick wins available there either.

Andrea Hausmann

Going back to the statement I just said before, that the discussion about tools got larger, more aggressive, and definitely more political. The next few things I am going to mention now are things you probably have experienced your colleagues act against, and probably even yourself have had as behavior in the past. These are typically human flaws, and probably it is not even flaws, but just how we as humans function in order to survive.

For example, people are getting motivated by different reasons. Some people are getting motivated by cognitive reasons, some people are getting motivated by altruistic reasons, and some people are getting motivated by power. If someone is getting motivated by power, they tend to make sometimes irrational decisions just because they know this decision will probably help them grow their personal impact they are having. Or it can get even worse: people might tend for a solution or making a decision because they know this decision will not shrink their power impact area. I like to call that garden.

People are liking to look after their garden, and this is a very human behavior. Those people normally do not do that because they are evil or anything like that. It is really just how humanity in the end works. It is a very thin line for the companies and for decisions to make sure we take care that this is a human behavior, and we must take care of people behind that behavior and decisions, but also that we try to minimize those personal biases as much as possible in companies. Also, if I think about how many people I personally know who would put the company's success before their own success, I think there would not be too many.

Another big pillar in complex and big organizations are politics. Politics might, again, play into someone's card because you know this person will repay you at some point, or this person might even help you progress in your career. This behavior is very toxic, and is very not what we should have in companies. But such a transformation can take years, and depending on what kind of senior leadership you are having, will not ever even happen.

You attending today at this conference, you are the middle management and senior management who can actually change that kind of behavior in your organization.

Duncan Lawie

One of the critical things for us to do to make those changes is to move away from the HiPPO, the highest-paid person's opinion, and move towards being more of a data-driven organization. Now, that might sound like a slightly tired term two or three years later, but it was just what we needed to think about at the time.

We looked at what data was available and what we could do with it. To begin with, the data that was available to us was the information that had been important to us in the past. Quite often, these were things where either we had chosen those measures because they made us look good, or we had worked out how to make ourselves look good against those measures. They are not necessarily vanity metrics, but they are the kind of things that help to make sure that you get a nice green chart on your own list, and therefore there was some resistance to changing those measures.

As we moved forward, one of the key things that made a difference to us was that we needed to see how we could use the light where it was, a little bit like looking for your keys under the streetlight, not necessarily because you lost them there, but because that is where the most light is. The next thing for us to do was to try and spread that light out.

To be honest, a critical thing there was actually having enough people in our organization having read this book. Once enough people had read "Accelerate," we could start talking about outcome measures. We could start talking about a common industry-standard set of metrics, and we felt that we were all talking about the same thing, that we had a common understanding of what we were talking about.

But even once you have agreed on those metrics, actually measuring them can still be a challenge. Lead time to change is quite easy to define, from the point of commit to the point that something is actually released into production. But if you do not have a single chain of control through that process, it can be very difficult to actually measure that amount of time. Some of our more advanced groups were using pipelines and processes that let them actually do that end-to-end measure. But if we just measured those teams, we would be actually losing a lot of our opportunity for growth, our proof of improvement of moving towards pipelines.

Another example from the other side is that if we look at change-related failure, some of our divisions are very latency sensitive. They care a lot about small issues. They will shout as soon as they see something. If you compare that with another group who are more interested in long-term measures and have less sensitivity to moment-to-moment changes, they may appear to have a lot smaller change-related failure rate. That brings us to a place where the organization is afraid of the comparison between divisions, because these guys look like they are doing a lot worse than those guys.

Here, again, we need to make sure that both are on the trajectory of improvement rather than trying to make unreasonable comparisons between divisions.

All this combination meant that we felt we had a compass, but we did not know if it was a perfect compass. Clearly, the important thing here was that we had a rough idea of where north was, and it was important to start moving, to look at the fact that it might not be compass north, it might not be true north, but we could get towards our targets. We could start to make real progress rather than waiting for a perfect compass. As we get closer, we can improve the quality of our measurements and narrow in on our focus. Indeed, that is the work we are doing now: to continue to improve the quality of our measures.

Another thing that we wanted to look at was staff satisfaction. My favorite definition of DevOps is still the one-liner from Jon Smart: "Better value. Sooner, safer, happier." We are talking about a lot of the aspects of this through the conference. But talking about happiness in a big corporation can be difficult. People have different ideas of what happiness is. People have concerns about actually asking people in their own jobs, "Are you happy?" But my feeling is the best way to do this is to actually ask people.

I had created a challenge there myself, because when we first started talking about DevOps, we did not have enough measures. We sent out surveys. We asked people, "How DevOps are you?" It turns out that on a sunny Friday afternoon, people felt that they were more DevOps than on a wet Monday morning. The critical thing here is that we should have been using telemetry, instrumentation to measure those things. But some things are about people, and the easiest way to find out what a person thinks, I believe, is to ask that person.

From there, I ended up having to talk to associates across the industry to understand what they were finding when they attempted to do this kind of thing. One organization, a leader of 1,000 people said, "I know what my people think," and he did not need to actually run a survey. Well, potentially, he is a genius. Potentially, he is misinformed. In either case, surely you can get better quality of data, even to confirm his hypothesis, by running a survey.

Another example was an organization where the leadership team, frankly, were afraid to ask their staff, "How do you feel?" because they felt that their staff were not happy, and they did not want to be put into a situation where they would have to ask them and then do something about it.

Bringing that back into Credit Suisse, we thought about the way that we wrote our satisfaction survey and what the questions should be, so that we could work out what we would do with the answers. We used the results of our satisfaction survey to prioritize our ongoing activities. This made a significant difference to the quality of our outcomes. Of course, we also made sure that there was room for comments, for individual experiences and responses that could guide us about the things that we did not already know.

This comes under the term of engagement through dialogue, of working across the organization and building our own shared vision.

Andrea Hausmann

Through such a shared vision, a few years back, a common toolchain enabling DevOps end-to-end got developed in Credit Suisse. This toolchain really enables everything from collaboration tool to code reviews, search functionalities, pipelines, build, test, deploy, security analytics for technical debt. It is a huge thing.

Duncan and I actually joined that team a few months back. When it was originally built, this toolchain was living in a business division and serving this business division. Suddenly, it was realized how great and important that would be for the whole organization, so it got moved into the central IT function.

As it is living now in the central IT function, it has to serve all the functions and all the other businesses. With that, it suddenly has more than 20,000 IT employees to serve. With that, you definitely have 20,000 different opinions about what you should do next.

As soon as this was moved into a strategic place, this discussion got huge because everybody had the feeling that their very own division probably would need a tiny bit differently, so that they can leverage what they need to do the best afterwards. If you think about it, we really have different balance sheets for those divisions, so they ultimately have different angles in mind.

This was not easy, and soon there were competing solutions coming, trying to challenge our approach of the central toolchain. Do not get me wrong here: we think challenging the status quo is very important, and we think entrepreneurship is the right thing to do, but we must do it in a sustainable and reusable way.

As money does not grow on trees, not even in banks, and neither do our resources, and we definitely do not want to build endless complexity into our ecosystem, we must be careful what we do and have clear eye on that. If you think about it in game theory terms, the plan is not that one player gets from it everything and as much as possible, but that everyone, all the players, profit from it as much as possible.

This central toolchain is now a huge success in Credit Suisse. We are having now even business users on our central toolchain using a few parts of the toolchain. With that, we have more than 40,000 users and more than 28,000 daily users.

Surely, there will always be a few complaints and a few concerns from users, what could be changed and whatsoever. But ultimately, two things helped us getting the toolchain into a state everybody can accept and live with and really support. One thing is we are 30% financed externally by our customers, well, internal customers in Credit Suisse, and that helps us always stay customer-focused.

The second thing is we are living an inner-sourcing approach. That means that everybody in Credit Suisse can put their development and extensions into our toolchain if they wish. As long as they do not break anything or harm anything else which is already built, they are free to put any extensions into our toolchain if it helps them to grow something better.

Duncan Lawie

Where do we get to here? My favorite term for this is to talk about balance. Perhaps balance is just another way of saying systems thinking, that first way of DevOps. There is a balance that we need to find between the individual and the team, between the team and our business customer.

It starts with the individual. One of the responses we got from the feedback survey was from people who were concerned that they were being asked to do too much in too many different dimensions, and to somehow abandon that deep skill that they had. Now, that was definitely, and is definitely, not our intent. People like me, who have got 20 years of experience in a space, or whatever the necessary time is, should be able to concentrate on where they are technically excellent.

One of the things that we do here is we have started doing practitioner-to-practitioner sessions so that those people can share their expertise, and we can build the knowledge of the whole organization.

But equally, people need to be able to think about what the rest of their team is doing and build out a broad understanding of the rest of the organization. We are doing that primarily through communities of practice, which will focus on broad topics like DevOps and Agile and so forth. Therefore, they can learn a little bit about what other people do.

Equally, if you are early in your career, perhaps what you spend more time doing is looking at that broad opportunity and thinking about where exactly you want to actually root down into deep skills. The result is you end up with the T-shaped skill set that people talk about. The edges of those Ts join up to give you a team that can work together across the topics that we want to cover.

That, in the end, gives you that DevOps result. You keep your team in synchronization. You have your balance between the development goals and the operational goals. Again, those DORA metrics have two measures that focus on the kinds of things developers primarily care about, speed of delivery, and two measures that operational people primarily care about, the stability of your operational environment. Balancing those goals lets the whole team improve, and also helps them with the empathy for each other and the separate goals that they might have otherwise.

Beyond that, of course, the purpose of IT in a bank is actually to deliver to our customers, whether that is the end customer. In Zurich, Switzerland, for example, we have a major high-street banking operation. Globally, perhaps we are focusing more on relationship managers who are helping their customers, who are helping with investment processes. Whoever is actually the consumer of our technology, we need to think about what we are delivering to them as the primary focus.

But still, there is a balance needed. If you focus only on delivery to the customer, then you are tempted to put in hacks and shortcuts, create manual work to get deliveries done faster. But doing that puts you in a place where you are creating an unsustainable situation for yourself, and the team needs to spend some time on self-care, on doing the things that make sure that we succeed and continue to succeed as a team. Otherwise, you lead to burnout and eventually the failure to deliver anything to production. Focusing on that balance, on reducing manual processes and toil, also helps with the customer and continuing delivery of success to the organization.

Andrea Hausmann

Last but not least, I would like to talk about a very delicate topic. As Duncan mentioned before, the ultimate value we create as IT to the banking is either serving our end customer or shipping good code quality to our front-facing employees. The pure value creation lies with the engineer.

If you think about the core versus context framework, the core actually is the engineer and the context is everything else. We must make sure that the core part gets bigger and the context part stays as small as it necessarily needs to be.

What we did in our division to make sure we are following this approach, which is not an easy one, is we put a different mechanism into the hiring process. If a hiring manager would like to hire an engineer, it is quite easy. There is a shortcut through it. There is not much to do, not much to justify, and they will get their spot approved and can put the offer to the market.

But if you would like to hire someone else than an engineer, for example someone like me, the hiring process is tiring. There are many obstacles we built in on purpose to make this process painful. So the hiring manager really thinks about: is this what we want to do? Is this the way forward?

Surely, engineers alone will not rule the world. There is, like with everything in the world, a need for diversity. As we discussed before, this pattern with a discussion about tools, if you tuck in ten engineers into one room, we can guess what happens eventually. There is this need for diversity that, for example, someone in the room probably at some point will point back to the original problem saying, "This is probably first we need to discuss the process behind it," rather than how do we resolve that with which tool in the end.

Ultimately, always, we need to think about the end goal. If we think about this conference we are holding here at the moment, for example, the DevOps transformation, which we are all very much interested in, we do not transform because it is funny or because it grows our impact. We do transform because we ultimately want to reach a goal, and the goal is to be generating the value better, sooner, safer, and happier.

Duncan Lawie

So are we achieving that? Yes, I believe we are. We are having an impact, and some of these comments include our customers across the organization and other parts of IT. I would like to focus on the one from Laura Barrowman, twice mentioned in the Fintech News list of the most influential women in finance in Europe. She is clearly influential in Credit Suisse. As the group CIO, she has 12,000 staff, so what she says matters.

On this topic, she said, "The DevOps program has served to ingrain a common set of goals across Credit Suisse IT, while igniting a transformational shift in our organizational culture and mindset." So we are on the journey, but we are far from finished. We have more mountains to climb.

The questions we would like thoughts from you on, particularly at this point, are: how do you focus on core engineering activities over context? How do you talk about happiness in your organization? And how do you test whether the message is reaching through the whole company? Because the more people we have together with us on the journey, the easier it is to reach our goals.

Thank you.