Roderick Randolph

Hello, everyone. Thank you for joining this session with us today. My name is Roderick Randolph, and I'm a distinguished engineer at Capital One. And I'm Arthur Maltson. I'm also a distinguished engineer here at Capital One. We're super excited today to tell you a story of how an internally built CI/CD pipeline framework grew to become a major InnerSource product at Capital One.

We'll share with you the lessons we learned that shaped our journey, and it can help your organization harness the power of InnerSource inside your company's walls. We'll start at the beginning. Many years ago, Capital One followed a waterfall approach to software delivery. We had a lot of monolithic applications, and they all ran in our own data centers. It was a painful process to release software into production, and the deployment steps themselves were very manual and error-prone. It literally took months to get a release out the door.

If we fast-forward to today, we operate a lot differently. We have agile teams following continuous delivery practices. We're all in the cloud, which enables our business to be more nimble. And more importantly, we've automated our deployment processes so that they are safer, predictable, and more frequent. But how exactly did we get to where we are today? Our digital transformation didn't happen overnight.

It took a lot of hard work and effort from everyone. The CI/CD pipeline framework we built was an important part of that journey. If we hop into a time machine and go back to 2015, Capital One had publicly announced at Amazon's re:Invent developer conference that we were moving to the public cloud. And while our digital transformation was already underway at that time, the public cloud announcement was a big moment for our company. And software engineers, including myself, were super excited about where we were headed. Around that same timeframe in 2015, across the Great Lakes in Canada, in a relatively small office near downtown Toronto, Capital One was standing up a brand-new technology organization of what we call a software studio, and this studio was going to transform the way the Capital One Canada business operates.

The new studio in Canada had a startup feel to it, and we were going to do all the transformational goals our tech leaders had inspired us to do. We were going to automate all of the things. And so on a more personal note, I saw the new studio as an exciting new opportunity and challenge, so I packed my bags and I moved from Virginia to Toronto to join the team and be a part of the exciting journey. But as we got started on our first few business initiatives in Canada, our developers quickly ran into a few roadblocks as we worked. We had hired a lot of new developers. They literally needed to learn all of the things.

They needed to learn how to build and migrate applications into the public cloud for a bank. They needed to learn DevOps principles and practices. They needed to learn how to run containerized workloads. They needed to learn how to use open-source software safely and securely, and how to automate manual processes. Each one of these on their own is a challenge, but tackling them all at once is very hard for the average developer. And oh, did I mention that we're also a heavily regulated bank across three countries?

That means there are many compliance requirements we need to go through for everything that we do. We soon realized that getting from point A to B, which in our case was getting an application deployed into production with automation, was going to be filled with twists and turns, and our developer productivity would suffer unless we developed a solution that made their jobs easier. So with that startup mentality I just mentioned, we decided to build a CI/CD pipeline framework that worked for the Canada business, and we used open-source software. We thought about the entire developer experience and how we could reimagine our software release process to make developers' lives easier. We also made all of our new framework source code available inside of our company's walls so that other developers in the office could see the code and could contribute features as well. This was an exciting and fun time.

I recall a moment where a software engineer, who I won't name, but they happen to be a co-speaker of this talk, they committed a bunch of new code changes to add a brand-new feature to the framework. And immediately after the code change, we had developers running over to our desk complaining that their build pipelines were failing. Now, it turned out to be a simple syntax error, and we were able to quickly fix it. But this was an example of how quickly we were developing the framework and working directly with developers to gather their requirements and pain points. Developers were our customers. It wasn't long after we built our first MVP that our Canada business launched its first microservice API on the public cloud with a fully automated pipeline.

This was a huge milestone and accomplishment for the team. We knew it was a great start for the business, with many more automated deployments to come. And to give our framework a brand, we gave the framework a name, and for this talk, we're going to call the framework Bingo. We designed a cute little mascot for Bingo because what developer doesn't love mascots, stickers, and free swag? The news about Bingo quickly began to spread across other parts of the company. Developers started talking to one another about how they enjoyed the developer experience and how it solved the problem that they were facing.

We knew word was spreading because we had a dedicated Slack channel for Bingo, and a key indicator we used for adoption was how many people had joined the channel. We would celebrate every milestone from 25 new members to 50 new members to 100 to 150. We were just seeing natural and organic growth, which was exciting, but we didn't know what was coming next. So as the framework continued to grow and take on new users and new use cases, I decided to pack my bags and actually move back to the US. But throughout this transition, we started to see a huge growth in Bingo. We saw hundreds of developers starting to adopt and use the framework.

Bingo was reducing the amount of time it took teams to release software. It automated away a lot of developer toil, reducing the amount of time it took to deploy new software from months down to weeks, if not days. But we soon noticed that there were other similar pipeline frameworks being built across the company. At first, I was surprised and bewildered at how many frameworks that had been built. What felt like duplicate effort turned out to be developers trying to solve a common pain point. And to put this into perspective, like any large enterprise, Capital One comprises of multiple lines of businesses.

And given the size and scale of the company, it's easy for duplicate efforts to occur, even within the same line of business. So case in point, the Canada organization falls under the credit card division, and we were seeing multiple pipeline frameworks sprouting up in just the card division. So within card, we began to naturally ask ourselves: Why do we have so many duplicative pipeline frameworks? Can we consolidate to a single framework? Can we pool all of our talent together and solve the problem once and move on to more challenging problems? So this led to a healthy conversation between the various framework owners within the credit card division.

And we came together in a community event we call barn raising. And as a community, we worked together to figure out which framework we would standardize on for the credit card division. And after much deliberation, the community decided to standardize on the Bingo pipeline framework that had been built in Canada. A key decision from that barn-raising event was that we were going to turn Bingo into a formal InnerSource product. And for those who may not be familiar with InnerSourcing, the main idea here is that you take many of the concepts of building open source software and apply them inside your company's walls. So think about public GitHub issues, pull requests, project maintainers, along with some other processes and governance, but bringing them inside your company and optimizing them for how your company works.

And there are a lot of great benefits to InnerSourcing that can unlock collaboration and productivity inside your company. So benefits include code reuse, knowledge sharing to help knock down those silos between teams that can often occur in a large enterprise, promotes higher code quality because more developers are looking at the software being used inside your company, and overall, just better innovation because everyone is focused on the same goal and feel a part of the solution. If you're new to InnerSourcing, I would definitely recommend checking out the O'Reilly book on how to get started with InnerSourcing. Now, returning to our story. What happened after the barn-raising event? Well, you can probably imagine the excitement of how Bingo was going to become the standard for the card division, but also there was a lot of panic in our faces after we learned that Bingo was going to be leveraged by the entire credit card business.

A framework that had initially been built for a few hundred developers in a small office in Toronto was now going to be used by several thousand engineers and hundreds of applications in Canada, US, and the UK. This was definitely an exciting time, but also there was just a lot of pressure to make this work. And at this point, we had to fasten our seat belts and scale up our operation, very much like a startup that had just hit the jackpot. We quickly realized that we needed to replicate the Bingo team and spread knowledge on how our framework works. So among the first things we did was we conducted large-scale boot camp and in-person training sessions, and this enabled a couple things. It helped educate more developers on how to use Bingo for their business application.

But more importantly, it taught those who wanted to contribute to Bingo how Bingo works under the hood. So this education helped teams adhere to many of the core principles that went into the initial development and success of the framework. We also ramped up our mob programming sessions where we had multiple developers sitting physically next to each other or even virtually between the US and Toronto, developing new features and capabilities in real time. We also found that there were other areas where we needed to make significant investments, including improving our documentation and expanding our office hours. Things were just moving very fast. And then we got hit with yet another curveball, which brings me to act three.

When a process or product has been found to work well, it's normal in a large enterprise to figure out how to scale it up, especially the good parts. So remember when I said that Capital One has multiple lines of businesses? Well, just like within a division, there were multiple frameworks, pipeline frameworks. Across divisions, we had even more pipeline frameworks. And the question was asked again, why do we have duplicative frameworks across the company? What if we had just a single pipeline framework for the entire enterprise?

Imagine the reduction of duplicative efforts and the time saved by engineers if the divisions just collaborated and worked together. Just imagine that. Well, that's exactly what happened. All of the divisions got together to rely on a single framework for the enterprise. And collectively, the organization decided to take the framework that was built in the credit card division, Bingo, and bring it to the enterprise as the base for all other divisions to build upon. And one of the good parts about Bingo was its InnerSource spirit.

We needed to take many of the lessons that we learned from InnerSourcing card and quickly apply them for multiple divisions. We thought scaling Bingo in just one line of business was difficult, but boy, were we wrong. While bringing the divisions together, I remember a lot of conversations that went something like this. "Hmm, our divisional pipeline doesn't solve the problem that way. We solve it this other way." And so working through all those differences and aligning on a path forward was probably one of the most challenging parts of bringing the divisions together. But the more uplifting moments were those aha moments where, "Oh, you do it that way?

We do it that way, too. We should work together to remove the duplication." And we had a team in retail bank that needed a new serverless capability into the pipeline framework, and that capability didn't exist, and their response was, "We would love to contribute that feature. How do we get started?" So those types of open conversations is really what makes InnerSource super viable in a large organization. And to help foster more of those collaborative conversations, we established what we call a pipeline council. And the council is a group of leaders from each division to represent their division's business needs to influence the direction and design of the shared enterprise framework going forward. It's a great forum for everyone to align on priorities, use cases, roadmaps, architecture patterns, et cetera.

Just really enhances the collaboration between divisions. And like any popular open source product, the number of GitHub issues and pull requests against a successful InnerSource product will feel endless and become very time-consuming. Additional contributors are definitely needed to really scale up. We had a community contributor who got to the point where they were consistently contributing multiple high-quality pull requests into the Bingo repos. They would also review other people's pull requests and provide feedback on their pull requests as well. And to reward contributors who were actively engaged and consistently participating, we elevated them to become a trusted contributor.

A trusted contributor is someone who has achieved a special status in our repos, where they are trusted to review and approve pull requests from other contributors. So having a path for users to become a trusted contributor has really enabled us to bring others in the community and give them a sense of part ownership in the product. There's so much more to our story, but where are we today? While I'm not allowed to reveal exact numbers, I can say that today we have thousands of developers in our Slack channel, which is a far cry from the small handful of engineers we had initially. We have hundreds of contributors and contributions coming in the framework across the entire company. And the framework powers thousands of automated pipeline builds every single day.

Bingo has come a long way, and it's amazing to see how it's grown. But what lessons can you take away on perhaps your journey to growing an InnerSource product?

Arthur Maltson

Thanks, Roderick. Now everyone knows my embarrassing syntax error story. But as Roderick was saying, our InnerSource journey was a winding and sometimes fraught journey. But we wanted to share it with all of you because we'd love to inspire all of you to take on your own InnerSource journey. And to help you along the way, we would love to share and summarize some of the key focus points we think helped make our InnerSource journey successful and highlight some of the pitfalls we fell into so you can avoid them yourselves. We feel that what we want to share is valuable regardless of whether you're just starting out in your InnerSource journey, you've started to gain traction, attention, and users, or your product has gone enterprise-wide.

Regardless of what phase you're in, we believe there's three key focus points for a successful InnerSource journey. The first one is to look at your InnerSource project as actually a product. Something that has continual upkeep, a roadmap, and many of the product principles that many organizations understand and practice, to use those on the InnerSource product as well. And one of the key aspects of a successful product is user focus. The difference with an InnerSource product is that the user is a software engineer, so developer experience becomes key to the success of an InnerSource product. And finally, as Roderick said, InnerSource was all about bringing the concepts of open source to an organization.

One of the most important concepts of an open source product or project is focusing on the contributor and making their experience really great. So let's dive into one of these key focus points, product. From a product perspective, in the early days, what you really want to look for is ideas, concepts, MVPs that scratch your own itch or scratch somebody's itch. If a specific team or part of an organization finds something to be painful, manual, not really a great experience, it's very likely that many other people across your organization feel the same way. As Roderick described, our journey to cloud and that automation was something that we built to scratch our own itch via Bingo, but ultimately scratched the itch of many other parts of the organization. As your product starts to gain traction and attention, you really want to focus on setting down some core principles that serve as the pillars or foundations for your InnerSource product as it continues to grow.

In our case, for Bingo, we really cared about developer experience, so we would write down specific parts of that experience as core principles. For example, make sure that error messages are well-written and guide the user on how to fix them. And as you start to get to an enterprise scale, it's often possible that changes can happen to the InnerSource product without a good discussion about the long-term implications, what the downsides might be, and the overall impact of said changes. This is where we took learnings from the open source community and brought in the concept of RFCs, request for comments. Many large open source projects like Rust language make good use of this, and we've tailored it for our needs. And as you start to also get to an enterprise scale, not losing sight of the details is one key focus point that's also important because as you get to that scale, it's easy to lose sight of the individual in the metaphorical hills.

But focusing on that individual and making sure that their experience is still as great as it was in the early days is what's going to keep your InnerSource product healthy and usable and sustainable into the long term. So just to quickly summarize from a product perspective as a key focus point for a successful InnerSource product, look for products that scratch your own itch. As you start to scale, create some core principles, and as you get to enterprise scale, consider introducing RFCs and making sure you don't lose sight of the details. Let's move on to users. As I said, one thing that's unique with InnerSource products is that they're focused on the software developer. So early on, what we found was key to our success was making sure we provide that white glove support.

This helps build that foundation of advocates who not only had a great experience because we sat with them and troubleshot with them, we also gave them some insight into how the product works so that they become better advocates in the future. As you start to scale, what we found is the manual touch points, the places in our product that required manual intervention, start to really grind on your ability to scale. One story that comes to mind is actually our onboarding process required 20 to 30 manual steps that engaged both the developer trying to use Bingo and us as the core team. So we took the time to automate that entire onboarding process, and it became a Slack app that took very little time for the developer. They had a great developer experience and took no effort on our end. So we were able to focus on new features and bug fixes.

And as you start to scale to the enterprise level, when I mention not losing sight of the details, one of the big details is actually user happiness, and the way to gauge that is to use an industry practice known as Net Promoter Score, or NPS. This will start to help you see how do others perceive your product and can give you the ability to narrow in and focus on specific individuals or groups and have focused empathy interviews or other mechanisms to get more details on why they feel a certain way. So to summarize, focusing on the user is critical for an InnerSource project's success. What we found is that providing white glove support in the beginning is a really great way to build up advocates. As you start to grow, automating away manual touch points so that the developer experience is maintained and is even better, and you free yourself up. And as you get into enterprise scale, running quarterly or twice a year NPS surveys.

Moving on to contributors. As I said, contributors are the lifeblood of an InnerSource product, and the way to engage contributors early on in your product is by building it in the open, being transparent. Doing everything in GitHub or your internal source repository, but making it public, like Roderick was mentioning. Having those discussions and code reviews public as well. I can still remember early on, we would have random software engineers from Canada join in on a code review, and they would provide really valuable feedback. So it's a really great way to engage contributors early on. As you start to scale, as Roderick mentioned, the in-person training and boot camps, we found those really valuable.

Not only giving them the technical details, which we can kind of do that asynchronously, but instilling the principles and the culture of the InnerSource product, that was the most value that we saw from the in-person training and boot camps. And as you start to get to an enterprise scale, providing a trusted contributor model, a path to move up the ladder and take on full ownership of some parts of your InnerSource product or ecosystem. This lets you as the core owner move on, but still feel comfortable that it's left in good hands. And as you get to enterprise scale, recognition, which is important in all stages, becomes even more important. And recognition doesn't have to be complex. Sending an email with a thank you to the contributor, but CC'ing their manager goes a long way.

It's not open source. We all have bosses, and when our bosses know we're doing a great job, you feel great as the employee and you want to do it again. So focusing on contributors as a key focus point early on, develop in the open. As you scale, using in-person training, mobbing sessions, boot camps to onboard new teams, and at the enterprise scale, introducing a trusted contributor model and not forgetting about recognition. So those are the three key focus points that we think help make a successful InnerSource journey. But of course, along the way, you might bump your head and if Roderick and I could jump into that DeLorean at the beginning and travel back in time four years, some of what we would tell ourselves, whether we'd listen to our future selves or not, I don't know, but what we would warn ourselves about remembering and focusing on and making documentation a first-class principle in your InnerSource product.

Make it your definition of done. Just make sure that documentation is not only written, but kept up to date on a regular basis, because this comes back and haunts you in the future. As we grew to a larger scale, started to gain traction, we'd remind ourselves to always keep up with the latest version of whatever tool we're using, not to put off upgrades, and to keep track of the code quality and not forget that the health and quality of the underlying code is really what forms, along with the principles, the foundation of your InnerSource product. And as we started to get to enterprise scale, we would remind ourselves to be open to different contribution models, that maybe what works for one line of business doesn't work for another, and just be more open and empathetic there. So these are some of the kind of pain points and pitfalls that we ran into. So more documentation at the very beginning and really throughout, reminding ourselves to keep up with code quality, keep up with versions of the tools that we're using, and being open and willing to experiment with different contribution models.

And of course, this wouldn't be an enterprise conference if there wasn't something that looks like an Excel spreadsheet. So here's a quick summary of everything we just talked about. Thank you again for listening. We really enjoyed having you here. I'm Arthur Maltson. Feel free to reach out to me on Twitter or LinkedIn if you have any questions.

And I'm Roderick Randolph. You can find me on LinkedIn if you have any additional questions. Thank you again for listening to our talk. And here are some additional resources. Of course, the slides themselves you can find there, but also linking to a lot of really great knowledge that's been built up around how to run and introduce InnerSource to your organization. Thank you again.