James McLeod

My name's James McLeod. I'm a software engineering lead at Lloyds Banking Group. And I'm actually here today to talk about our journey into InnerSource.

Before I actually get started on the talk, I'd like to just get to know a few of the people here. Hands up if you're a software engineer.

Okay, great. We've got a lot of software engineers here.

Hands up if you're a business lead, or want to hear about engineering transformation and how to accelerate.

Okay, that's great. And hands up if you're from GitHub.

There we go, in the corner there.

First of all, who knows what this is? Just shout it out or put your hands up. Does anybody recognize the brand Monzo, or has anybody heard of fintech? I guess people have.

Within Lloyds, we've also recognized that there's a real need to accelerate the way that we do engineering. Our competitors now are very lean, very fast startup companies who get accelerating very quickly.

Quite recently, I was actually speaking to one of the co-founders of Monzo, which is a guy called Jason Bates. He came to us within the Lloyds engineering leadership community and was talking about how Monzo actually came up with their idea of providing debit cards, and got from concept to cash in a month.

For a bank like us, for a very large retail bank, coming up with a concept of a new product and actually getting it over the line in a month is unheard of. Previous to our engineering transformation, it would take us 365 days to get a line of code from developer out into the world.

And so to hear that there are fintech companies, and even tech titans like Amazon or Google, and even quick-paced startups and disruptors like Uber getting all of their products out in a very short production lifecycle, it fills us with the need to change.

As an organization, we know that engineering is the future, and we know that we've got to accelerate and catch up with the curve very quickly. But in order for us to become super agile or super lean like an organization like Monzo, we've got to do things very differently.

Because as you can see from all of the figures up on the board, we're massive. We have 26,000 corporate customers. We have 2,000 branches throughout the UK. We have 75,000 employees. We have 30 million customers. And we actually process 125 billion sterling per day. And so turning this oil tanker around is going to take a lot of effort.

But we need to do this because, as you can see, we've been serving customers for 250 years, but now we regard ourselves as a 250-year-old startup company.

In order to do this, Lloyds is transforming, and we've come an absolutely long way since we actually started our digital transformation. We're actually becoming a very engineering-cultured bank now, and we're actually pretty mature when it actually comes to DevOps.

But we see DevOps as more than just automation and more than just making sure that we're mitigating automated testing and upping our quality. We actually see DevOps as pivotal to the way that we change our organizational culture within engineering.

Up here you can see that DevOps is also about collaboration, self-service, automation, measuring, becoming lean, and changing the way that we operate in terms of engineering culture as well.

But in order to do that, DevOps is also delivering our ability to change at pace. It's keeping us safe. So as we become more nimble and more lean, DevOps is actually what's there in order to mitigate all of the shortfalls of moving fast.

It's there to help us unite our business and also with our engineering. So our product owners and everybody who operates within Agile can actually use DevOps and the way that we test to influence the way that we write our stories and measure the way that we're actually producing all of our products.

And best of all, within Lloyds, we enable our engineers to use the best-of-class tools. We federate all of our DevOps if we've got very mature teams, so our engineers can go out there and pick the tools that they need in order to solve the problems that they're trying to solve, or we also provide those services as well.

So our culture of engineering within Lloyds is actually transforming, and that's great, because we need to be able to move fast, and we need to be able to produce our products and our services very quick and in a very lean way.

However, what this has actually produced are various different product teams, which we call value streams, that produce code and produce products very fast. We have, within our organization, many different feature teams with very talented engineers who are able to produce products working together, within these red circles, very fast and very agile.

However, the problem that we have right now is that it's very difficult for those teams to actually work together.

As you know, if you're an engineer here, and if you align to open source, you'll be working within a system that allows you to look at all of your fellow engineers' code and raise pull requests or raise support tickets or maybe even come up with functional ideas about how to move forward within that.

Within Lloyds, unless you're working within a product team, it's very difficult for us to communicate. So if there's an engineer like myself and yourself, and we're all working on the same product, providing we're in the same repo and providing that all of our permissions are set up the same, we can collaborate.

But equally, if I'm sat next to an engineer who's working on a very similar product but isn't part of my product team, we might be able to talk over our laptops about what we're doing, but it's actually very difficult for us to actually share code.

And for me, that's an absolute problem, because a lot of quality isn't just about DevOps, and it's not just about the tooling. It's actually about working together and collaborating from engineer to engineer.

So the code that I actually write physically, although I might be able to get a DevOps tool that tells me whether I'm duplicating code or whether I'm linting incorrectly, it's actually me working with somebody else physically that actually ups my game and makes sure that the quality that I'm producing is actually the quality that we expect as software engineering leads.

And also, one of the outcomes of this is that each of our product teams are actually producing individual releases that go into the same release cycle. It's very likely that one product team will produce a library or something that is very similar to another product team.

But because we can't see the work that each of our developers are doing, we can't tell if the work that I've actually produced is being duplicated with another product team. And so the amount of efficiencies, and also the amount of code duplication, means that although myself, as a feature team, are moving fast within my own product area, as an organization, we can actually become a lot more efficient if we're able to see what each other's doing and collaborate in a way that removes that duplication of code from each other as engineer to engineer.

So what we're doing within Lloyds: we've noticed as an engineering team that there are actually a lot more efficient ways for us to work. And part of our engineering culture has actually meant that we're now enabled to come together as an engineering community.

Rather than being a very hierarchical organization which is being driven by product owners and from the top down, we've actually moved into becoming an organization that's actually very much led through communities and guilds and forming our own working groups.

And as part of that move, engineers together put their hands up to say, "There are better ways that we can be doing this." It's really great that we can use DevOps tools that we say that we should be using, but actually, as a developer, there are really efficient ways that we can collaborate together in code.

And so we formed a working group. We went together as a full array of engineers. So it wasn't just the type of engineer. It wasn't just me and my friend. It would've been engineers from all across Lloyds Banking Group. And we set out on a journey to find a way that we can collaborate in code.

And so it just so happens that we ended up looking at GitHub Enterprise, but actually, that entire journey took us through looking at GitLab, looking at Gogs, looking at Git, looking at Gerrit.

And what we decided as a working group was that we would actually like, as engineers, to be able to work within a system that we recognized on the outside as well.

Because one of my other passions outside of engineering within Lloyds is actually I run a meetup. I run a JavaScript meetup called React London: Bring Your Own Project, and a lot of the engineers that come to that are also open-source collaborators on the outside of the organization.

They formed part of this working group as well, and they said, "It would be really great, not just for us within Lloyds, but also for us to be able to communicate with people outside of Lloyds as well, to be able to collaborate using the same tool as we recognize on the outside of the organization as well."

And not only is that good for us because we're working within the environment that we know, but it's also good for engineers outside of Lloyds to recognize that we're actually using tools that they recognize as well.

It's kind of a very good attractor of engineers into our engineering teams, because all of the open-source community who recognizes this tool will recognize that we use it within the organization as well.

For us, not only is collaboration on the inside of the organization really important, but also bringing the right engineers and the right level of talent is also very important as well. Because believe it or not, even though our brand is on the high street and you probably walk past one of our branches every day, whether it's Halifax, whether it's Lloyds or whatever, we are actually an engineering company, and we do have many thousands of engineers.

So making sure that we have the right tooling and we actually give out the right engineering message is actually really important.

Not only did we produce an outcome to start pioneering and using GitHub Enterprise within the organization, but what we actually found was that GitHub Enterprise itself allowed us to create engineering communities virtually within the organization as well.

If I look back at this model, a lot of the product teams that were working together were sat next to each other. So it'd be literally us over here and you over there, separated or working together in your groups.

But as soon as we introduced GitHub Enterprise, we actually found that we were getting lots of different engineering collaboration happening, not just within engineering groups around the table within feature teams, but we would have engineering groups working together across locations.

And actually, it also started to cross boundaries outside of your regular software engineering teams. We found that a lot of our creative team would actually get involved as well, because design is actually becoming a lot more creative technologist. And so we found that we were having a different and new relationship with our creative teams, too.

So what we actually evolved this into: we brought GitHub Enterprise into the bank. We started working together around code in a very different way, but we wanted to take it further, and we wanted to explore different ways of collaborating.

And so the baseline for that is to create an engineering community. So based around internal workshops, brown bag sessions, meetups, and guilds, and also the engineering community within the GitHub Enterprise system as well.

We wanted to make sure that engineers could collaborate in code, so people contributing to very common libraries, and also just taking a look at what each other's doing. Engineers are quite nosy. They like to see what people are doing under the hood.

And then also use the system to improve ourselves. So it's not just a system that allows people to scrutinize each other's work, but it allows us to be more open in the way that we operate. It allows us to keep all of our documentation together within the same repo that it relates to.

And it also allows us to be a lot more open in the way that we communicate around code and actually have code as the fundamental building blocks around our discussions now, rather than PowerPoint or talking about what we would like to do rather than being able to show how we're actually doing it.

And all of these three arrows or building blocks are working towards something that I don't think it's just us who call it this, but I've heard it across PayPal plus other banking groups that I work with as well. It's actually a model called InnerSource.

So it's that whole internal open-source model that allows engineers plus others to collaborate and also share their work.

That is both through the ability to read and also improve each other's documentation, the ability to fork off of people's repos and build upon it yourselves, the ability to open pull requests and for others to review those and accept them.

And then also the ability to use systems like Nexus as an internal NPM registry and push your own JavaScript or other libraries to an internal registry that other people can pull as well and also contribute to.

What we also wanted to make sure that we had within Lloyds is a decentralized system. And this was really important as well, because Lloyds has been known for in the past as an organization that provides the tooling that other engineers should use. It's gone from being a very centralized organization to a very decentralized organization where federation is actually promoted and engineers have the opportunity to choose what they work within.

Now, because GitHub Enterprise is seen potentially as a centralized system, because you basically provide a tool that you're asking everybody to join, we needed to find a different way of selling this to our engineers because we didn't want people to think that we were trying to force our engineers back into a central system.

The way that we actually overcame this was by providing a mechanism for everybody to get involved in the system and for the system to be engineering-led rather than led by the organization.

And so what we formed when we realized that GitHub Enterprise was the direction of travel, we formed a working group called the GitHub Enterprise Working Group, which is basically, as you can see in the triangle, formed of three sides.

So we have our engineering teams, who are actually our customers. We've kind of flipped it around. Our engineers are the GitHub Enterprise Working Group's customers. That's what GitHub Enterprise is there to fulfill, and they're the people who we listen to when there are issues, or they're the people who we notice when they say, "What are the actual benefits of us joining this?" They're the people who we need to educate.

And then we have the operations side. So we're lucky enough to be working with our IBM partners who are actually the people who are providing this and operating this on our behalf. And we've invited them into the working group as well.

And I remember the first time that I actually invited my fellow engineers from IBM in, they were saying, "Oh, why do we need to do this? We operate this as a service. The whole reason why we're here is so you don't have to worry about this type of stuff."

But actually, what we found was that by bringing our IBM partners in, we were able to look under the hood. All of the engineers who didn't like a centralized system suddenly realized there is an engineer behind the scenes looking after this stuff.

We were able to look at CPU utilization, see how the RAM was actually being used, know what the upgrade path was for the system, and also speak to our operations team within IBM about all of the difficulties they were finding.

And so we did actually have one scenario where one of our engineers was continually polling the system, and that was causing a spike in our CPU and RAM. And because of the GitHub Enterprise Working Group having engineers and operations engineers within it as well, we were able to quickly shut that down by just coming together in the working group.

Whereas before, the mitigation for that was to upgrade the system or upgrade the RAM or upgrade the CPU. But we were able to close it down by coming together with the engineers and saying, "Do you mind not doing this? Can you find a different way of using webhooks rather than polling?"

And then we also have our support team as well who are there to make sure that our engineers are happy as well.

Our support team are the people who are creating self-service mechanisms for our engineers to join. We have many thousands of engineers who would love to join GitHub Enterprise, but we need to actually find a mechanism for them to do so because we can't just open it up.

We've got to provide a mechanism for our engineering leads to say, "This is my team. These are the guys who I'd like to join." And so we need the people on the forefront who are getting those requests and producing the tools that allow everyone to join.

And so this operating model is actually really good, because what it means is that everybody within the working group is getting to understand how these systems should form. It's almost like having a small business. And so we're communicating with each other in a very different way.

And we're becoming very mature in the way that the organization runs and the way that these platforms should operate. So I'm actually hoping that the members of this working group will also grow in their careers, and they'll really understand what it means to be running these tools, especially when you're scaling into an organization of thousands of engineers potentially.

Right. So the way that we're actually moving right now: believe it or not, within Lloyds, although our maturity curve over our 250-year-old history means that we've basically been there since the dawn of engineering, we are actually very progressive in the way that we actually want to get involved and collaborate in code.

Just bringing GitHub Enterprise into the bank and having something there which is not locked down and enables people to see each other's code is actually a massive step forward in the way that the bank actually operates.

Because as you can imagine, banks are very bound by security and regulation, et cetera. And so for every issue that somebody's had previously within an organization like that, there's been another safeguard. And so everything has been tied down and locked down.

But because we're actually working with the GitHub Enterprise Working Group now, we're facing all of those security stipulations that we've layered on top of all of our repos. So only the people who need to see code should see code. We're starting to remove that. We're starting to work with the regulators and the people who do our auditing to teach them that actually engineers are okay. They can work within code. You do not need to lock them out. You can trust us.

Which means that we're really moving towards that InnerSource model now. We're really starting to collaborate and build up that trust with the business and within the organization.

And more engineers are starting to learn the benefits, and they're starting to really use the language as well. It's really crazy. You come up with an idea, and you plant a seed, and then all of a sudden, language that you've been speaking to others starts coming around from directions that you don't expect it.

And so when you actually notice that, you know that people are starting to believe in this, and that's where we're at at the moment. Our engineering teams are starting to feed that back into us.

Now, the next step of our journey is actually into open source now. And this is a real interesting point because Lloyds, as an enterprise bank, has a huge brand. I remember each time I go to the cinema, there's a Lloyds advert. Each time I walk down the high street, there's a Lloyds branch or a Halifax branch, and people know who we are.

And so we need to be really aware that as soon as we start swinging the lighthouse around, so we start contributing into open source, all of a sudden we've either got code or documentation or we're letting our IP out that other people will start scaling across their organizations.

And so we're starting to have the conversations now about, okay, we understand what collaboration is now within the organization. How can we be more collaborative outside of the organization? So we're starting to take that journey outside now.

You've got me speaking at--I've come to this conference. I'm starting to speak more at other conferences as well. We've got meetups that we're running. We've got a flagship branch of Halifax in New Oxford Street in London, which has started doing coding sessions for children. I'm going to be running a meetup from there as well.

And so it'll be taking JavaScript into the high street. Who would have thought you were going to Halifax and learn how to code JavaScript?

And so we're starting to take those steps into open source. And I'm starting to bring my fellow engineering leads together to say, "Okay, so what would that next step be if we wanted to make that journey?"

Because as soon as we step out into open source and actually start delivering code to other people, we then need to support it as well. And so we're talking about how we can make all of our documentation that is shareable, shareable.

We're also talking to our GitHub friends about what have other people done that allow them to move into enterprise code sharing.

And so very soon, I would imagine, and this is a challenge that I'm setting myself, we will start producing something that we will share maybe in GitHub, which for me would be like that next step outwards.

Anyway, I'm James McLeod from Lloyds Banking Group. I'm very open on social media. So if anybody wants to reach out to me, I'm on Twitter. If anybody wants to connect to me, I'm on LinkedIn. Or if you want to ask me questions after this, I've just been told we've got five minutes, and so I'm more than happy for people to ask me questions now or at least come and find me after this presentation when I'm happy to talk more about what we're doing at Lloyds, and also get your ideas as well so I can take that back, too.

Thank you very much, everybody.