Kenneth McLeish

Hi, we're here from JPMorgan Chase.

My name is Kenneth McLeish, and I run Equities Technology. This is my colleague, Danny Myers, who runs Equities Production Management, and also runs the DevOps community of practice in London. Or DevOps Danny, as we call him.

Let me give you a little bit of background about JPMorgan Chase. Hopefully, you know us. We're one of the biggest financial institutions in the world. We manage assets of $2.6 trillion, and we've got a history going back over 200 years. We operate in over 100 countries, and we employ over a quarter of a million people.

It's a massive organization that ranges from everything from retail banking, ATMs, big mainframes, all the way through corporate finance, investment banking, asset management, and things like high-frequency trading, and a wide range of technologies there.

In doing that, we have over 50,000 technology employees and spend $10 billion a year on technology. Looking at it in that kind of dimension, we are, in many ways, a technology company. We've been doing technology for a very long time, and it has always been core to what we do.

If you even think back to the '80s, we were electronically trading even before there was the internet. But many of the ways of building software at that time were not modern, and there are new techniques that have been learned.

It's really come as an organization that, from the top of the organization, people have said, "There must be a way that we can be developing software better. We can learn from others. We can elevate the principles that we apply in building software, in developing software, to be more modern."

Right at the top of house, there was a framework that had been created to try and help facilitate the adoption of new modern development techniques and DevOps. That comes through things like learning and development, new career paths for people, bringing in new tools, looking at additional metrics that we can track the capabilities of different teams, and then creating communities of practice where we can share knowledge and try and encourage others across the various groups to do more.

We'd now like to take you on our team's journey and highlight the impact and contribution that our team has had within the organization.

We work within asset management, and we're focused within the equities trading technology team. It is our systems that generated, in 2017, $1 trillion worth of trading activity, which is an equivalent of $4 billion daily volume. We trade across six trading desks, which is across 70 financial global markets. We're operating at very significant scale.

Our journey started in 2013, where we had many monolithic and legacy systems that we were operating. In equities trading alone, we had 13 systems that were mostly a combination of regional trading systems and reporting systems.

Releases at the time were very infrequent. We actually had very much of a structured plan, build, and operate model that we were working within. We had to have lots of hand-offs between the different teams as part of that cycle.

Releases didn't come without issue either. In fact, they were troublesome, dramatic events where they typically caused incidents. As you can see, there were 10 that year. As a result, we had incidents from releases and system outages. No one really looked forward to them.

Our code coverage at the time was only around 36% across our systems, so there was a lot of manual testing involved. It also took us around a week to ship our code to production. Excuse me.

Our technology team was very much split at the time, so we had a separate dev team and a separate ops team as part of the segregation of duties model that was linked back to the plan, build, operate model.

In the ops team, we had eight people. The development team was made up of 36 people. Now, that wasn't solely developers. In fact, it was only 50% developers. The rest were manual testers, business analysts, and project managers.

What we'll do now is take you on the journey in more detail and cover off some of the challenges and principles that we applied.

I want to start talking about some of the principles.

The first challenge that we faced was how to get started. Given that we had such a critical business that we were supporting and had such an archaic platform that we were looking after, the real challenge was the developers felt that they were just desperately trying to keep the car on the track. It had some dents and bashes, and yet what we were wanting to do was completely modernize the vehicle that we were operating while it was still driving. That was a huge, big challenge.

One of the first principles that we then looked at for the team, and this is something that some of you may have come across, was the sphere of influence and control.

The principle states that there are some things that are in your sphere of control. For instance, what you're wearing that day. You completely decide what you can wear. There are some things that you can influence, and there are many things that are in your sphere of concern, but you have absolutely no control or no influence over.

Often, you'll find people will spend a lot of time worrying about trying to change the things that are outside of their sphere of control or influence. They end up getting frustrated. They end up becoming the complainers, and people start listening to them less, and ultimately their sphere of influence and control starts to shrink.

But for those that focus on the things that are within their control and being good at that, then over time, the things that they get given start to grow, and their sphere of control and influence starts to grow as well.

Looking at that principle, one of the things that we dealt with was our change process. The change process that we had was very onerous. You can see on the screen here, there were multiple parts that developers had to fill in before something went into release. It was time-consuming. It needed lots of different teams to get involved, and it was something that the developers wanted nothing more than to get rid of.

But ultimately, given we're a highly regulated business, this is something that was outside of our control to get rid of. The key part was then reminding the team, "Okay, what do we have that we do control that we can apply to make this better?"

The first thing that we looked at was actually tools like these within siloed organizations can end up becoming the communication mechanism between teams, and that is massively inefficient.

You'd have developers write up all the details about their software, and then they would click submit, which would go into a queue, which would end up going to another team that would then review that information, come back with some questions, hit reject, and that would bounce back. That was one of the killer parts that was slowing the whole process down. But that was in our control to change.

We really changed how we were engaging as teams. First of all, we would communicate more often and earlier in the process. We would start thrashing out some of those issues earlier on, even to the point where we could actually start sitting teams together. Sitting support and development side by side had a massive impact in making this more efficient.

This really started to transform, even within the piece that we control, how efficient this was for us. Then over time, as we started gaining momentum and getting faster in our changes, we were also invited by other teams to help them. That ultimately led to us being involved with the change group in terms of choosing a new tool that will ultimately completely streamline this process and automate it.

Danny Myers

Our second challenge was with tooling. As our two teams moved closer together, we realized that there were a number of tools that we were using that were producing the same result. We had to assess those tools and consolidate them to choose the best tool that would fit the job.

After looking at the tooling aspect, we were able to reduce the tools and make sure that we had the right tool to do the job. By doing that, we could then become much more streamlined in what we were doing and how we were operating.

The principle that we applied was called nudge theory. Nudge theory is a principle about how you can encourage behavior by making it simple to follow and nudging people in the right direction.

From the image, you can see Richard Thaler, who authored the book called Nudge, and he came up with a number of behavioral science concepts that can encourage people to choose the right behavior.

This quote is taken from the UK Behavioural Insights Team, who conducted a lot of investigation into how you can nudge and encourage behavior through government. They coined an acronym called EAST.

You can encourage behavior through making it easy, so if you remove a hassle factor from that. Through making it attractive, if you incentivize it or have rewards associated with it. Making it social through social norms or if other people are already adopting it. And making it timely by having the most appropriate time when people are receptive to that.

There is a funny example of nudge theory that is quite useful in everyday life, and I'd just like to draw upon that.

This is a men's urinal. Many of you will recognize this. Nudge theory has been applied here.

There was a study carried out in Schiphol Airport in Amsterdam, and they were trying to identify how they could reduce spillage in the men's urinals. To do this, they conducted lots of research because what they had to do was find the optimal place for where a stream could enter the urinal and produce the least splashback.

Once they had conducted all these rigorous tests to do that, they had identified that exact spot. I haven't actually got it on here. This is actually a football and a goal, but it's still a target. But on this example, they had stuck a sticker of a fly, and that became the optimal spot. What it did was it decreased the cleanup time in the toilets, in the men's toilets, may I say, by a whopping 80%.

Now, as I'm sure most of you can appreciate, us men are simple creatures. So if we see a fly, or a ball and a goal, some type of target that we can aim at, then we're most likely to do that.

Obviously, this is not a talk about toilets and anything else. We're talking about DevOps and nudge theory specifically. So how, in JPMorgan, did we apply nudge theory?

A tool called AppFit was developed, and the purpose of this tool is to assess each application's software development life cycle, and that's every application within the bank. We have 50,000 technologists within the bank, and this application is used to test every application's SDLC best practices.

There are various measures that it looks at that cover from how often you're committing, how often you're doing automated tests, the volume of automated testing that you're doing, is it above a certain percentage, and how often you're deploying.

The important thing here is to note there are pictures of robots on the screen with an AppFit. To determine how healthy your application is, you get an image of a robot graded against your application.

If you're ticking none or some of the boxes, you may have the broken robot. No one wants a broken robot. The power of the image is so important because previously we would have had maybe a scorecard that would've just been a checklist of how well we're doing. But it's the picture of the robot that is making the most difference here.

The broken robot, you're not really doing too well. The awkward-looking robot, you're doing all right, but it still looks like you're struggling. It's almost like that robot's in a straitjacket. Ultimately, what everyone wants to have is a picture of the happy dancing robot that's doing the jig, because this is showing that your application is following the best practice and is recognized as leading the way in terms of how you're applying your software development life cycle processes.

Kenneth McLeish

The next challenge we faced was one of technical debt. Every system has technical debt. We had lots. Some of our systems were 20 years old. Some of it was written in Visual Basic. It wasn't designed to be built for testing, built for frequent releases.

Again, for the development teams, that really felt like a mountain to climb, to ultimately replace many of those systems, many of those components, and start to tackle that technical debt problem.

The principle we looked at for this piece was one around marginal gains.

Marginal gains was a term that was coined by David Brailsford. Now, David Brailsford is the British cycling coach and the coach of Team Sky. What he said, and this is a quote from him, is you can achieve optimal performance by the aggregation of marginal gains.

This is what he started to apply when he became the coach in 2002. At that point, British Cycling was terrible. They'd won one medal in 78 years, so it wasn't doing well. But he was saying that if you can be 1% better at everything you do, that aggregates together to make you much, much better overall.

There's a huge number of examples on this. A couple that I really like: when a rider had a rest and recovery day, usually what the athletes would do, it just meant that they weren't training, so they would just go about their normal day-to-day business. They would play with their kids. They would go to the shops, whatever.

What he would change is, he said, "If you're on a rest and recovery day, you sit down or you lie down and you do nothing." That change actually improved their recovery, so the next time they went training, they were actually a little bit better.

It even went into other things, like when they were staying at hotels, they would take, first of all, their own pillows, then their own mattresses and pillows. That actually optimized the rest that they would get the night before a race, so that they could, again, improve their performance the next day.

All of those marginal gains, from 2002 to the point of 2008, the Beijing Olympics, the British Cycling team won seven out of 10 gold medals. A massive transformation. From there, they've gone on to become the most successful cycling team in the world.

So how could we take marginal gains and apply it to what we did?

One of them was improvement themes. Again, there's been documentation on this, but it's a great example of marginal gains. We would take things that we wanted to improve, and we would sit down, and the team would then brainstorm what would that definition of awesome look like if you were to be really good at this one thing?

Then from there, we would also, as a team, brainstorm what were all of the problems that we were currently facing. Then from there, we would then say, "Right, we're going to take some of these. Let's take a small set." Breaking it down into very small chunks of the things within our control that we could solve and move forward to making it better.

By looping that process again and again, and ticking these things off, then it's amazing looking back at how many of those things you did. It takes away that sense of overwhelm that you have by just focusing on small steps, which over time aggregate together, make things much better.

Our fourth challenge was gaining buy-in, not only from the team, but also from the business. How can you do this?

We used the power of stories. You may have heard of the anecdote, to change a culture, you need to change the stories within that. To do that, you need to be able to inspire people through having the story to help drive the change. You need to be able to have a purpose that people can buy into and relate to, to help drive the culture change that is also in line with the organization's values.

To do this, we came up with a number of catchphrases and mantras to help reinforce the messages that we could use again and again.

Here are some examples. For example, never see the same issue twice. We all accept that when developing and deploying software, you're going to see issues. We know that happens. But the main thing is to make sure that that exact problem does not reoccur again. To do that, fix a problem, build the right automated tests around it to make sure that particular piece of functionality can't break the same way again in future.

As we're working within a trading environment, we came up with the mantra of changing trading from an art to a science. The better we can monitor and measure our systems, the more systematic we can become, and therefore our trading itself can become much more systematic as well.

So much so that our traders have actually gone on to reuse this quote when speaking in the press. You really know that your mantras have taken good effect when they're used right back at you, and you can actually read them and hear them used back at you.

Code complete to production in an hour. This is the benchmark we set ourselves so that we could release quickly, either when fixing an issue or when reacting to a business change that had to be done quickly. We know that we can address the functionality and then roll out that change within a quick period of time.

Or from quarterly to daily releases. That really is the aggregation of all the marginal gains that we've gone through to be able to release on a regular frequency.

And to finish up: it's no fun if it's too easy. We all know change is hard. Probably all the people in this room have been on big change transformation journeys. They're not easy. You've got to persevere with them. They should be fun. They should be challenging. But the more you persevere, then you're going to see success at the end of it.

Let's look at the results over that five-year period. We've gone from having 13 systems and managed to tackle a lot of that technical debt to get to three. In doing that, we've modernized those platforms, so they're no longer the monoliths that we had. They're modern, microservice-architected platforms.

We've managed to change from having that kind of quarterly release process to releasing more often than once a day. But in doing that, and again, going to some of the other mantras, faster, safer, better, we've reduced the number of incidents.

Even though we're releasing more often, the number of incidents has gone down quite substantially. We've only had one critical incident this year. That is something that is really meaningful for us as a business, because when we have incidents, that can mean reputational damage or major financial loss.

A lot of that has been achieved through enhancing our code coverage. Again, these are metrics that we track. On our whole estate, including what legacy remains, it's 60%, but then we also monitor any new code coverage, which is much higher.

Then our metrics to go from code complete to production: it was a week. That's now under an hour, and is getting faster all the time.

The most profound change is really in the team. While it was two separate teams between dev and ops, we are now one team that's combined. We have had more investment because we have been giving more value to the business, so they've been happy to invest more in us. The team has gone up slightly.

But the biggest impact is the number of people that are actually engaging at a code level. While it was only 18 before, so that was half of our development team actually writing code, of the combined development and support teams, we now have 80% of the people actually engaging at a code level.

What that then means is that the business are seeing the value of 40 developers rather than 18 before, even though it's kind of been a marginal increase in overall headcount.

Looking at the business results, we have a successful business. Our trading volumes have been increasing over time. Over the five years, it's gone from $600 billion a year to over a trillion. But what could have happened was that in doing that, we would have had to increase traders. That would have happened if we'd not changed the systems.

But by changing the systems, the trading headcount has actually gone down. It's gone down by about 25%. Each trader is now capable of handling almost three times the volume than they were able to do before. That's because we've been able to streamline and automate the processes that they follow. Again, looking at the whole value proposition across not just tech, but the business as well.

There's no point in doing that if it makes the trading performance worse. That is something that we measure very accurately, and that has improved by 24% over those years since we started. Just before we started, it was going down. Since then, it's been continually going up. That benefit goes directly back to our clients.

Another benefit that goes to our clients is the trading fees. As we have got more sophisticated, we then use simpler services from our brokers. They then charge us less for those simple services. That has amassed a whopping 47% saving in brokerage fees that we get charged. Aggregating the financial value of that together, that is $1 billion in savings for our clients over those years. It's really substantial, the impact that we've been able to deliver.

Then there's actually the changing of the business. Christian West, who's our business sponsor and runs the equities trading desk, now thinks of technology differently. He was quoted in the press saying that the trading team has become a mix of traders, quants, and technologists, with the latter two becoming a bigger part of the mix.

That is a massive transformation that the business have felt. They're operating in a very different mindset than they were before.

In doing all of that, we've been awarded the best buy-side trading desk for the last three years. Something that we are collectively, as a team, very proud of.

Just to wrap up with a few key takeaways. Firstly, know your sphere of influence. Stay away from what you can't influence or from what you can't control.

Use nudge theory. Remember the sticker of the fly. That can help drive behavioral mindset change, and also nudges, when applied, can have a huge impact and be very cheap to implement.

Use marginal gains and improvement themes. Just by listing out and addressing each issue one by one, looking back over time can have quite a profound effect on the overall amount of deliverables that you're working on.

And remember the power of stories and mantras and catchphrases. Sometimes reality is too complex, and you need to break it down into a story to be able to get people to understand what it is that you're trying to do.

Just a couple of problems that still remain for us. In JPMorgan, we're a huge organization, 50,000 technologists. We've got many different technology teams moving at different paces as part of this DevOps transformation that we're working on across the bank. We're trying to find ways to make it easy for those teams to adopt through socializing those experiences.

We're trying to look at how we can socialize experiences in an easy way so that people from all parts of the organization can understand what it is to go on that transformation.

And we're always hiring, so we're trying to nurture and hire the right kind of talent so that we can build the teams of tomorrow as part of this journey that we're going on.

Thank you very much for listening.