Topo Pal

Glad to be here for the fifth time. I started speaking at this conference back in 2014, and over the years through the transformation I have lost a lot of my hair. So, my name is Topo Pal. I'm senior director and senior distinguished engineer at Capital One. Basically, I'm a developer, DevOps evangelist. I'm also a member of our open source steering committee, and I'm the creator and core contributor of our flagship open source DevOps dashboard project called Hygieia.

Jamie Specter

For those I haven't had the pleasure of meeting, my name is Jamie Specter, and I am very excited to join Topo today on stage to talk to you all. Part of my role at Capital One is open source counsel. That means I get to partner with Topo, our open source program, and our development community on all things open source. That includes use, contributions, and external projects.

I've been with Capital One now for six years, and I've had the incredible opportunity to partner with people all across the enterprise. One thing I love about this conference so far, this is my first time here, is the consistent theme related to the value of partnering with others. I've seen a lot of value come from that, and you'll hear that theme today in our presentation.

Topo Pal

Many of you may know Capital One is a credit card company. We're actually one of the largest credit card companies in the US, with over 70 million accounts. You may also know Capital One is one of the largest banks in the US. But what you may not know is that Capital One is a founder-led technology company. In fact, we're about to celebrate our 25th anniversary from going public next year. Our youngest competitor is actually 108 years old, so in that sense we are a startup. Because we're founder-led, we have that entrepreneurial mindset.

From the day we were founded, Capital One has always tried to be disruptive. It's part of our DNA. Open source and DevOps have contributed to that success despite the regulated industry in which we operate. We are different. We have a different DNA because we build our own software. We build on public cloud. We deploy our software on public cloud using microservices architecture, and we use a lot of open source. We are an open source first company. And we strongly believe in continuous delivery, especially in a regulated industry. Our belief is that if you are not doing continuous delivery in a regulated industry, you are not "compliant enough."

About six years ago, we were mostly outsourced, 100% waterfall, manual processes all over the place. We were slow, as you can imagine, and only commercial software was allowed. We also said no to open source. That was six years ago.

Six years of journey to today took us from mostly outsourced to mostly insourced, and I think this is very important for our digital transformation. It took us from vertical silos like ops, dev, QA, and so on, to a product-centric approach where the product team includes everybody that needs to be there to do everything related to that product from soup to nuts. And as I said, from dev, ops, QA, RM, we had these different roles, your developers, your operations, and all that, to we are all engineers. Developers write some code. Operations write some code, infrastructure as code. QA engineers write some code, testing and all that. Even release managers have become release engineers.

We spoke about that. In 2014, I talked about building out our automation steps. In 2015, I talked about how we tried to scale DevOps and open source and cloud innovation across the enterprise. In 2016, we talked about how we are concentrating on measuring, improving, and maturing our DevOps success. Then last year I talked about better governance via continuous delivery, and that is very core to us because we are regulated. We cannot do things that are not allowed. So we had to be very careful about our journey, and we decided, and we believe, that continuous delivery is the way to go.

This year, when I was thinking of the topic that I'm going to talk about, I found a State of DevOps report from DORA, the Accelerate State of DevOps report. For the first time, they included open source in the study. I thought it was a natural progression to talk about open source and DevOps. That's the thing we are going to talk about.

As I said, about six years ago, we said no to open source. When we say back in the days, say no to open source, I also talked about many enterprises like us, larger and smaller. They all say no to open source, but use this: code in Java, use Spring Framework, code in Eclipse IDE, use Apache Ant, but say no to open source. I don't know if that makes sense or not. But what it means is that everybody used open source, but they were in denial mode or something.

Back to 2012, when we first started our continuous integration journey, we started with this basic tool set on commercial tooling. Then I brought in Hudson, because Jenkins was just born at that time and there were a lot of things going on between Jenkins and Hudson. We decided to stick with Hudson. We brought in Maven, SonarQube, Nexus. And then it blew up. It didn't work.

That is because the commercial source code software didn't have enough capability to kick Hudson build jobs. We went back to the vendor. The vendor said, "Oh, the version that you have doesn't support that. You need an upgrade, and that costs money." And we kind of said, "Okay. We are not doing that. We are bringing in Subversion." Today we don't use Subversion, we are on GitHub, but back in the day we started using Subversion, and on day one it worked. Magic.

But that was a big deal, moving from commercial source control to Subversion, because: free tool? Are we going to use a free tool in our critical business needs, such as the thing that holds all of our software? That's risky. What if it breaks? Who to call? I don't know. Are we going to give our IP away because we'll be putting source code in that open source code? Does that mean we have to open source our code too? What if somebody steals our code, meaning some hacker puts something in Subversion that sends every commit to some cloud version or somewhere in the world?

So we went through all these problems, and we talked about them. Finally, we came to a realization that to get Subversion into our organization, we at least had to buy a subscription to some support. We did that for the first year. We paid less than 1% of what I would have paid to upgrade my commercial software. And guess what? In one year, we never had the opportunity to call anybody, because nothing broke.

So we moved from commercial SCM software to Subversion. The other big thing was Maven. We used Ant, and everybody had their libraries in their source control, and they had their Ant scripts. It took days to even stand up a project on individual developers' laptops, and building was a nightmare too. So we decided we were going to use Maven. We were going to put Nexus in and put all the libraries in there, and resolve all the dependencies through our CI pipeline against Nexus, an in-house Maven Central, if you will.

Now that posed another problem: how do you get the libraries? Before Maven, this was our way to get libraries. This is an open source acceptance form. You fill this out. You first print it out, and then type your name, your title, the date, and identify the open source software. Each and every library used to have one of these. Actually, there are multiple pages. I just printed the first page for demonstration purposes.

Then there are some pretty important questions asked from you. Are you going to modify this software, yes or no? And if you said yes, you are in trouble. Do you intend to distribute this software along with your source code outside of Capital One, and all that? We printed that form, got it signed by your immediate manager, your VP, and all that, and then we did this.

So that's the fax going. Once the fax went there, the legal department looked at it and said something. But the question is, why fax? Why fax? Because back in the day, and I think even now, our legal department was in a part of a building that is kind of secluded. There's a huge glass door and you need a special badge to even get in there. That's the first reason. Second reason is you could not reach out to the legal department directly. You couldn't call them. Your emails were not answered because, in fact, they were busy in something else.

Once I was walking past that building, I saw one of my colleagues standing in front of the glass door and knocking on the doors. He was carrying a big bunch of printed sheets. I said, "What's going on here? Why are you here?" He said, "I have these open source library approvals to be obtained before I go to release, and my release is coming tomorrow." I said, "Why do you need approval now?" He said, "If I don't get approval for these, the CAB, Change Approval Board, won't approve my release." He was kind of knocking on the door. He finally got approval. Just by looking at this piece of paper, somebody would have approved it anyway.

Then he and I decided this doesn't need to be this way. There has to be a better way. No engineer should go through this. We decided that we'll set up meetings with our legal department to talk about the open source approval process. And we learned a very important thing. First of all, they didn't know why we were going to them to get approval for open source. And we didn't know what the risks are. We had kind of never talked to each other. That's where we sat together and understood what they mean by approval, and what we need open source software for. That whole realization came to a kind of success, which Jamie's going to talk about: our open source intake process and all the risks.

Jamie Specter

Awesome. First, let's think about why we're here. What is the DevOps Enterprise Summit? The goal of this conference is to give leaders the tools and practices they need to develop and deploy software faster and to win in the marketplace. What I'm here today to do is provide you information to have these discussions about using open source. What I'm not here to do is provide you specific tools and processes to implement in your companies. You are the expert for that. You know what works best for you. But equipped with the right knowledge, you can actually take advantage of the value that open source brings.

Something for this group to think about: we take great care in writing the code for our company. We do extensive due diligence in procuring our commercial software. So why wouldn't we treat the open source that we use in the same way? We want our applications to be healthy, and we want to make sure we're protecting our companies no matter what we're bringing into our environments.

First, we need to understand how much open source we're using. Being in Vegas, I'd like to make a bet with everyone here, and I bet that you are all using open source. I'm not usually a huge gambler unless I have some kind of inside knowledge. Thanks to Black Duck, and also the DORA report that Topo mentioned earlier, I have a little bit of inside knowledge that I'll share with you.

This data is based on a Black Duck survey of over 1,000 code bases in every industry you can think of. They found that 96% of those applications contained open source. Of those applications that contained open source, 57% of the code base was actually open source.

The visual here drives home that, yes, you can think about going out to a repository, downloading the open source, and bringing it in, but open source is actually coming into your environment from a number of different avenues. The takeaway from these two slides is: basically all applications contain open source, and more than likely, on average, your applications contain more open source than proprietary code. So it's really important to understand the risk.

Before we get there, let's think about how we think about open source. Open source is free, right? It's free meaning there is no initial monetary cost, and it's free as in freedom. You can use, distribute, and modify the source code without going to ask the author's permission. Something else you may have heard is that open source is free as in puppy. Who doesn't love puppies? They're cute, they make great friends, they make you smile. But having that cute companion that makes you smile can actually come at a high cost, meaning they require a lot of work. You have to feed them. You have to clean up after them. But many people still have dogs despite the obligations because there are ways to make it work. There are ways to make the process manageable. Open source is the same way. Understanding those risks and obligations is important to ensure that you're prepared, that the right people are involved, that the right tech is used, and that the right processes are in place.

I'm the lawyer, so let's talk about the risks. The main thing I want you to take away from my part of the presentation is that there are real risks to using open source. Someone in a legal or risk function who is not as familiar, or even in a business or technology function that's not familiar, may have some hesitancy. It's easiest always just to say no versus taking the time to understand. But hopefully, after today's presentation, all groups are ready to have that conversation because they are equipped with the right knowledge.

These next few slides have more information, and I'm not going to cover everything. The idea is that you'll get these slides afterwards and can use them as a reference deck. We'll talk about two buckets of risks today: security and legal.

Jumping into security, the takeaways are: one, your developers are the first line of defense. They are the ones bringing in the software, so they should be the ones managing it. There needs to be that you build it, you own it mentality embedded into your culture. This is consistent with the DevOps mindset. Also, unlike commercial applications that push updates for you, your developers are responsible for pulling those updates. Since vulnerabilities can be found anytime, and we do see breaches continue to increase, continuous detection and remediation is necessary. You do not want to be part of that team that has to answer to your CEO and executive team because there's been a breach because you've waited over 1,500 days to resolve one security vulnerability, when really remediation in a security role usually means upgrading to the most recent version.

There are also real legal risks. Some things to keep in mind: you need a license if you're using open source. That is your legal permission. That is the rules. Authors of software actually have over 2,000 different licenses from which to choose. When we think about open source licenses, think about two general buckets: permissive and copyleft. With permissive, generally the obligations are that you want to give attribution to that author, give credit. The other end of the spectrum is scarier for organizations, and that's the copyleft licenses. In certain situations, you not only have to distribute and share the source code of the copyleft-licensed software that you're using, but you also have to share the source code of any code that it touches. It's a very use-case-specific analysis.

I also want to point out that you're using the open source that you're bringing in as is, meaning you have no recourse against the author from which you got it. That's also a tricky area to explain to your legal and compliance functions, because with commercial software, you do have those reps and warranties.

When we talk about remediation for security, you upgrade the version. It's a pretty straightforward approach. With licenses, generally licenses aren't going to change throughout the course of the project, so remediation is different. It's usually replacing it with an open source alternative, something you write yourself, or a commercial alternative. Because obligations are usually conditional, meaning if I use the open source this way, then this, you can work with your counsel to get approval if the use case actually mitigates the risk that those obligations will take place.

Sometimes, if you don't know what the license is, instead of requiring removal, it's really easy to get on GitHub and request confirmation from the author. That's all you need. Also, if you want to see if it's a license that you can't use internally for whatever reason, based on your organization's risk level, you can get online and ask, "Hey, let's maybe add a permissive license to that." And last but not least, we're all busy, so sometimes you just have to submit a pull request with the license that you'd like them to adopt, so it's really easy for them to add.

Topo Pal and Jamie Specter

Topo Pal: Can I ask you a quick thing?

Jamie Specter: Sure.

Topo Pal: Is that your picture on there?

Jamie Specter: I'm sorry?

Topo Pal: Is that your picture there? Is that you?

Jamie Specter: Oh, yes. That is my picture.

Topo Pal: You are a lawyer, right?

Jamie Specter: Last time I checked.

Topo Pal: And you work at a bank?

Jamie Specter: I do.

Topo Pal: And you have a GitHub account?

Jamie Specter: I do. I have my very own GitHub account.

Topo Pal: And you also do pull requests.

Jamie Specter: I sure did.

Topo Pal: I don't care what people say about you, you are my hero.

Jamie Specter: Topo's too kind on stage. Thank you, Topo. I've generally had a really positive response when reaching out to developers. It's been a really good conversation. I may not always disclose, or ever disclose, that I'm an attorney, but that's only because I don't want to stop the conversation before it starts. This is just evidence that good conversations can happen.

Jamie Specter

Speaking of the attorney, what's the worst-case scenario? We understand the high-level security and legal risk, but so what? These are just some general categories to think about. I'm not going to spend too much time. If you want more details, come find me after. But these are the kinds of things you want to tell your leadership so they have an idea of why it's actually urgent to get this stuff in place.

The first thing is trade secret disclosure. Your code is your secret sauce. If you have to open source your competitive advantage, what's left for your company's worth?

Devaluation of patent portfolio. This is for companies that have patent portfolios. Some open source licenses will have patent grants, meaning you're giving out a free license in certain situations that covers the same functionality that your open source project does. From a policy perspective, this makes perfect sense. If I'm throwing out open source software for the community to use, everyone adopts it, and then I have a few patents in my back pocket and sue everyone for patent infringement, we want to avoid that, and that's what the licenses do. But you should be aware if you do have a patent portfolio, because the value of patents relates to your ability to exclude others from using it. It will devalue the patent portfolio.

Mergers and acquisitions. For companies looking to buy other companies, or companies hoping to get big enough one day to sell, it's very important to be in compliance with your open source obligations. Open source compliance issues can delay a deal, devalue your company, or kill a deal altogether.

Security threats and potential fines. I have Heartbleed and Equifax right there. You don't want your company to be part of a national breach and get that much media attention.

Reputation risk. Your customers, current employees, and potential hires don't want to work for a company that doesn't have a good reputation for risk management, especially in an industry like ours with financial services institutions.

IP infringement. I know there are some network people here. IP is intellectual property, not Internet Protocol. Typically, litigation in the open source space has been from open source foundations, and they're motivated by making sure that people comply with their licenses. But we have seen an increase in another category of people that bring litigation, and those are motivated by business and more business-related financial motivations.

This here is an order from a recent court case that recognized that the copyleft license is both a license and a contract. What's important to keep in mind is that this opens up additional remedies that a plaintiff can bring against the defendant they're suing. Something to think about: what happens if, because of pending litigation, you cannot use your mobile application or browser extension, and that's your main offering? Or what happens if you're forced to comply with the copyleft licenses versus substituting it out, and you have to open source not only the open source you're using, but also the proprietary code? That goes back to the trade secret stuff we talked about. Both scenarios can have a big impact.

So we understand who's using it, basically everyone here, and we understand some of the high-level risk. How do we tackle those risks? This is a framework to think about as you start the journey, depending where you are. Make sure you know what you're using and assess it. There are solutions that can map different security vulnerabilities, for example, too. There are resources out there, and I'm happy to talk in more detail if needed. Think about it as identify, analyze, and remediate, and then there's always going to be continuous monitoring and auditing, which can fit very nicely into your DevOps pipeline.

Then mitigate with a governance strategy. Capital One has actually implemented every one of these. I'm happy to connect afterwards to talk about our specific journey with that. I spend a lot of my time on the last one, education and training. I also included two links at the bottom that are helpful resources to reference no matter where you are on your open source journey. In addition to these resources, Topo's going to help us understand how DevOps can help.

Topo Pal

Right. As open source helps DevOps, DevOps helps open source too. The first reason is you can automate. Now, do not automate a bad process, because if you automate a bad process, then you get a bad automated process. Instead, look at your process and determine which ones are important to you or which ones make sense, keep those, and automate those.

Shift left right from the beginning of your pipeline. Start looking for open source license violations and legal violations, and mitigate accordingly. Frequent releases: if we need to patch something, we can always frequently release that patch as soon as possible instead of waiting six months or 1,500 days. Smaller batch size reduces the risk. You can release only that particular library update version to production. Collaboration and transparency are critical. Without that, you cannot go further, and you cannot do it alone. You need to bring in engineers, legal, security, risk officers, or anyone else you may need along your way. Without the collaboration, there's no way you can get there.

From here, I say this: just using open source is not enough. The reason I say that is because, remember I talked about our CI journey? We put all the libraries into our Maven repositories. While doing that, we found an incredible thing. There are libraries that look like open source, but they're not really open source in the sense that we modified them. Meaning that on that form, when you said, "I'm not going to modify," I actually lied. Now I'm in trouble. Oh, sorry.

Those modified libraries are in those source code repositories. What do we do with those? We cannot contribute it back, but we cannot use it, because now we are stuck in time. There has to be a way. So I started proposing this idea that we should contribute back to open source, and that was pretty disruptive.

To make it more manageable, I thought disruption is bad, so what did I do? I posted in our internal Pulse site, a social network within Capital One, a blog post: "Should Capital One contribute to open source community?" The response was huge. It is against our policy. Well, let's change the policy. Why should we give our IP away? We don't want to do that because Capital One pays my bills. I don't want to give my IP away and do damage to myself. What are the legal implications? Don't know. Let's ask Jamie. Who will use your software? How does it matter? Maybe no one, maybe not even us. Who knows? Are you going to write another payment framework in open source? Maybe. I don't know. It depends upon the developers. What if we get a bad reputation? Wait for a minute. We write code to handle people's money, and we cannot write code to log something in the log file? Give me a break.

So we had a war room at that point. The whole idea of the war room was that everybody comes together, all the decision makers come together, and try to resolve this issue that developers are not allowed to contribute to open source. In that war room, we basically decided that yes, we are going to make contributions, and that was a good day for Capital One engineers.

Since then, four years, we have contributed to open source: 90 developers, 193 projects. Most notable are Angular, Ansible, Hadoop, Log4j, Spark, and so on and so forth. And we didn't stop there. We have our own open source projects also. We have been open sourcing our own internal projects for three years now: 31 total projects. Most popular are Hygieia and Cloud Custodian. Many of you already use these tools and contribute back. And we are not stopping there.

This is our open source page. There's another thing that we are doing. We are actually working with many enterprises on what we call intersource, the next version of open source. What that means is basically enterprises getting together and solving their own problems without having to rely on commercial vendors. We are calling that intersource. We have partnered with companies like Verizon, Walmart, and American Airlines, forming a community around DevOps practices.

For us, being open source first results in a lot of benefits for Capital One, and some are listed here. But the one I want to call out first is the culture of collaboration.

Jamie Specter

Perhaps the most valuable observation I've made while being at Capital One and working with Topo and the rest of them in the open source space has been how much culture can influence behavior to do the right thing: the right thing for you, the right thing for your company, the right thing for your customers. It's not an easy journey. There are a lot of questions, multiple risks to manage, and a lot of lessons learned, which we're sharing today. But with the right knowledge, the right team, the right technology, and the right processes, it's been an incredibly valuable journey so far.

The value is recognized beyond our development community. You hear me standing up here talking about how valuable it is, and even our business partners understand this as well. We have a quote here from John Smet, a product manager in our card line of business, and that drives it home. He's actually spoken at this conference in London earlier this year. It really is a great community that we've established.

Topo Pal

No matter what you do, you will have an approval process. But here's my appeal: make it easy, streamline, automate as much as possible, not the bad processes, but the good processes. Collaborate often and create a good engineering experience.

I'll end with a few of my favorite hashtags because hashtags are important. I call these hashtag DevOps hashtags. The first one is YBOYO: you build it, you own it. Very important for your DevOps success. YBYS: you build it, you secure it. Thanks, John. MVC, I learned it yesterday: minimum viable compliance. That was awesome. GTGF. Any idea what that is? Get together, go faster, which is this conference. And the last one is my favorite: JFDI. I'm going to say this in front of you. I'm not offended. It means just do it.

Thank you. The reason is my final goal is to get this T-shirt: all of Chuck Norris's change controls are full cycle, and they're always approved. I still don't have my T-shirt. I need your help. So if you have any ideas as to how to better manage open source contribution and consumption in your enterprise, please let me know. Jamie and I will be very happy to share our experience. Thank you very much.