How Comcast Embraced Open Source, the Key to Success in DevOps Adoption
Comcast is a leader in media and technologies and serves millions of customers with broadband, streaming media, smart home technologies and content. The scale and complexity of Comcast with 1000s of engineers as well as its history could make it difficult to roll out new cultural changes and technologies. Yet, Comcast embarked on the journey to become a technology company with leading edge software development practices like open source, DevOps and security practices. I will share our journey into becoming an open source contributor and leader and some of the key elements that helped with this transformation.
Nithya A. Ruff is the Head of Comcast's Open Source Practice. She is responsible for growing Open Source culture inside of Comcast and engagement with external communities. Prior to this, she started and grew the Western Digital's Open Source Strategy Office. She first glimpsed the power of open source while at SGI in the 90s and has been building bridges between companies and the open source community ever since. She?s also held leadership positions at Wind River (an Intel Company), Synopsys, Avaya, Tripwire and Eastman Kodak. At Wind River, she led a team of product managers in managing a world class embedded Linux distribution and was a key member of the Yocto Project advocacy team on the board. Nithya is a Director at Large on the Linux Foundation Board and represents community interests on the board.
Nithya has been a passionate advocate and a speaker for opening doors to new people inOpen Source for many years. She has also been a promoter of valuing diverse ways of contributing to open source such as in marketing, legal and community. You can often find her on social media promoting dialogue on diversity and open source. She has spoken at multiple conferences such as OSSummit, OSCON, All Things Open, SCALE, Grace Hopper, OpenStack, VMWorld, OS Strategy Summit and Red Hat Summit on the business and community of open source. In recognition of her work in open source both on the business and community side, she was named to CIO magazine's most influential women in open source list. She was recently one of 4 people to win the 2017 O'Reilly Open Source Award for exceptional contribution to open source.
Nithya graduated with an MS in Computer Science from NDSU and an MBA from the University of Rochester, Simon Business School. She lives in the San Francisco Bay Area and is a proud mother of two daughters. You can follow her on twitter @nithyaruff
Chapters
Full transcript
The complete talk, organized by section.
Host Intro (Gene Kim)
I'm excited to introduce our next speaker for reasons that I'll tell you more about in a moment. Nithya Ruff is the head of the open source program office at Comcast, which is the largest network provider in the United States and owns brands such as Sky News that operates in Europe. In the past, she's helped manage the open source practice at SanDisk, Wind River, and has served on the board of the Linux Foundation for the last three years, responsible for the custodianship of some of the most important open source projects in the world.
I also know her because I used to work with her at Tripwire in 1999, when she was our director of marketing. She helped us open source Tripwire for Linux, and we worked with Red Hat, Caldera, and VA Linux to bundle it in their distributions. Nithya is a person I've known for nearly 20 years. I'm so delighted to see all the amazing things that she's worked on.
I'm also excited because just last week she was nominated and then was elected to become the chair of the board of directors for the Linux Foundation, the first woman to hold that role. With no further ado, congratulations and welcome to Nithya.
Nithya Ruff
Thank you so much, Gene. Good morning, everyone. It's going to be a slightly different kind of talk today. I'm going to be talking more about open source rather than DevOps, but I think open source is such an integral part of DevOps tools, DevOps experiences, and when you launch products, it's often open source-based. I think it makes sense to understand how open source influences and disrupts companies and how companies adopt open source. Thank you, Gene, for that lovely introduction. As you can imagine, it's especially wonderful for me to be here because I've known Gene for such a long time, and I have so much respect and love for Gene.
My journey in open source itself began almost at the beginning of open source, in 1998, when Silicon Graphics, as a server company, moved from being a proprietary server vendor to an open source company. It started shipping x86-based servers using Linux rather than IRIX, which was the operating system. As a company, we really had to understand what open source meant to our company, how to adopt it, how to work with the community, how to price our products, et cetera.
Then I got to work with Gene at Tripwire, and we had an opportunity to really embrace the open source community again by open sourcing Tripwire for Linux, working with distribution vendors. Then I had a different kind of experience at Wind River, where I had an opportunity to lead a product management team which was creating a commercial embedded Linux distribution, if you will, that would help telecom companies, industrial companies, automotive, mobile, et cetera, build devices based on open source Linux.
The last experience was at SanDisk, Western Digital, actually running an open source program office, which was very interesting, and I'll talk more about what an open source program office does. But today, I really wanted to focus on two things. One is my life at Comcast, because it is where I am today, and I'll share a little bit about that. Then I'll talk about the Linux Foundation and why institutions like the Linux Foundation are so important to making it easy for enterprises like us to work with open source and to work with communities.
This is Comcast. We assembled all of the logos that I could find into one single slide, and it's chock-full of logos, as you can imagine. The first side, to the left-hand side, is what you would in the United States know as Comcast. It's the Xfinity, xFi business, which is our internet business, the high-speed internet business. It is X1, which is our platform, which delivers both IP as well as traditional video. Then you have Xfinity Mobile, which is mobile telephone services, and Xfinity Home, which is really home security.
A big part of the Comcast products is also business services, where we actually work with small, medium, and large enterprises, serving both phone services, video services, as well as high-speed internet services. Comcast Spotlight is the online advertising part of the company. FreeWheel is an advanced advertising company which helps us do algorithms and other online stuff. Comcast Spectacor is actually less known. It's more about ventures and, if you will, running sports venues, sports teams. Esports is something that they're getting into, which is quite interesting. Then Comcast Ventures, which is the traditional venture investing arm of Comcast.
The second piece of the middle bucket is NBCUniversal, which comprises both the TV side, cable TV, live TV, like NBC, CNBC, MSNBC, and then you also have Universal Studios, DreamWorks, and other content creation side, as well as the Universal theme parks, which is all over the country and the world, actually. At the bottom, you see some digital properties which you're probably familiar with, like Fandango or Rotten Tomatoes.
On the right-hand side, very proud to welcome Sky as part of the Comcast family. In fact, my colleague Michael and I were at Sky on Monday, at Isleworth, and it's just a magnificent company which has all of the components that traditional Comcast has: both cable as well as content creation, as well as production, et cetera. The size of the company: about 175,000 people worldwide, almost $100 billion in revenue, and we are about 55 years old as a company, just to give you a sense of the scope.
Let me talk about the Linux Foundation, where I spend a good deal of my time as well. The LF started off more than 25 years ago as really a protector and kind of a home for Linus Torvalds, who created Linux. It was to help enterprises use Linux and open source and help them find a neutral home to create collaborative projects. As you can see from this chart, we've really grown into one of the most well-known and central institutions in open source. It includes everything from platforms and application projects to Hyperledger, which is blockchain and blockchain-related projects, to cloud and cloud automation, to embedded automotive, IoT, and certainly networking and orchestration. Practically everywhere that open source has won and is disrupting industries and is disrupting technology, Linux Foundation has a project that leads in that space.
The mission of the Linux Foundation is really to create sustainable ecosystems around open source projects. The belief is that there is a virtuous cycle, as you can see, in that projects often start with a maintainer who has an itch to scratch, a problem to solve, and then he or she builds a community around that project. Successful projects under the Linux Foundation umbrella or with support often end up being used in products by enterprises, which then creates profits, which then gets reinvested back into the project. The notion is that projects need to be valuable, useful, and usable, if you will, by companies and by enterprises and businesses, such that there is a reinvestment back into that project.
It's important to understand the history of open source, though, because you're probably saying, boy, when open source started, it was really a very fringe movement. It was started by people like Richard Stallman, and it had some very ideological beliefs. It was so far from how companies think, how your CFO thinks, how your CIO thinks. In the '80s and '90s, when open source first started, there was a tremendous amount of fear in the enterprise. Can I actually use this? And they're talking about free, and does that mean I have to give my products away? What is the GPL license going to do to my company?
The left-hand most, the goat, is really GNU tools, which Richard Stallman, who was the creator of the GPL license, created. The middle element is FSF, Free Software Foundation, which was started in those days, and then GPLv2, which is a copyleft license that enterprises really didn't know how to deal with and feared.
Then the next wave of open source adoption in the enterprise starts with the birth of Linux. Linus Torvalds, a student in Finland in Helsinki University, actually put out a little kernel, a microkernel, and said, "Does anyone want to work with me on this?" Taking the GNU tools that you saw on the previous page and the kernel that Linus put out, you actually had the beginnings of a good operating system that people could start using.
Then there were institutions like the OSI, which is in the middle, which protects licenses, which kind of reviews licenses, which makes it acceptable to know which license to use, which not to use. Then OSDL, which is the Open Source Development Labs, they were all started in the '90s. The objective of OSDL, which was started by companies like HP and IBM, was really to make it enterprise-grade and to make Linux usable by companies. You start seeing also the beginnings of Linux and open source being called open source rather than free as in free freedom rather than free as in beer, because remember, corporations and enterprises were afraid of the word free because they thought that meant that they had to give things away.
Continuing on with the second wave, you start seeing more mature organizations coming into being, like the Apache Foundation and the Linux Foundation. Then you start seeing companies like IBM, Sun, SGI, and HP adopting Linux in their product lines, shipping product lines based on Linux, actually endorsing Linux. I love this campaign, IBM's open source campaign, where they talked about peace, love, and Linux all in the same category. I think companies were trying to become more hip and cool and appear like they were also embracing open source while being corporate.
The third wave is something that you're all familiar with. I think by the time the third wave came around, open source had really dominated and won many, many infrastructure categories. It became the emerging standard, frankly, for cloud infrastructure components such as the operating system, such as server and compute and more and more storage and networking, et cetera. It wasn't surprising that companies like Amazon Web Services and Alibaba and Azure and Google Cloud are predominantly built on open source components, and cloud architecture is really very, very open source-oriented.
Let's really go to the meat of this talk, which is enterprises like ourselves, like Comcast, like Capital One, like Bank of America, like Target. Why are we going from just being passive consumers of open source or technology to actually creators and developers of technology? It really comes back to our journey as a company. In the beginning, even before 2006, frankly, we were like many other companies. We would buy technology from vendors, and we operated it, and we were basically operators of that technology. We really didn't know what was in the black box, and we were highly, highly dependent upon vendors for the features and functions and roadmaps.
Around the 2006 timeframe, when we started working on our X1 box, we said, "We've got to take destiny into our own hands. We've got to create our own products. We've got to create something that is truly innovative and provides features that our customers and our subscribers love." Around that time, we started creating original projects inside the company. We started using open source and consuming open source, and using open source methodology to create products.
Around 2011, I would say, we went from really just consuming to contributing back, where we upstreamed a number of patches into OpenStack, Cloud Foundry, Apache, and many other communities. Then I would say around 2014-2016, we also started creating original projects like Traffic Control and many others and open sourcing them. We had to create a structure inside the company to help us do that. Around 2016, we created an open source program office, which is inside Michael Winslow's organization under the CTO office, that really helps us strategically manage open source inside Comcast.
The phases of the journey started with being dependent on vendors, where we used to buy technology from vendors and work with vendors. Then we started understanding that every product we shipped had open source in it, so how do we make sure that we're in compliance with licenses and we're doing the right thing in terms of the open source licenses? Then we started creating open source inside the company, which taught us to create an open source advisory council. Then, eventually, we started coordinating and collaborating across the company in a more formal way through an OSPO.
Open source program offices, by the way, are quite popular these days and quite common, especially with large companies, both digitally native companies like Google and Facebook and Netflix, but also with other enterprises like ourselves. Capital One has one. Fidelity has one. Target has one. Indeed.com has one, and so do we. There is an organization called TODO Group under the Linux Foundation, where we all have an opportunity as open source program offices to work together across all of us.
There is a tremendous value in having a focused, dedicated open source office. It helps handle questions with the legal team, act as a translator between developers and legal, between marketing and engineering, and really help the company sort out its open source plans in a more strategic and coordinated way.
What has the impact of this open source journey been on Comcast? The first impact has been that it allows companies to really focus on their core competency and not on commodity or things that are already solved. What do I mean by that? We highly encourage people to start with open source in mind. It helps you cut costs. It also helps you understand where you need to put your limited development resources in terms of working on things that customers care about, like user interfaces and making everything easier from a customer experience perspective. Frankly, it's helped us reduce our commercial license spend, become more self-sufficient, become more competent from an engineering perspective.
Second, believe it or not, it has helped us really recruit talent and really attract good talent into the company. Especially as a traditional company, you're not known as a technology company, right? It becomes harder to recruit talent. Becoming an open source company has allowed us to, A, share our projects so people know what we're working on, and that becomes attractive to developers who then get involved in the project. We actually get to see the quality of their work before we recruit them. It becomes such a virtuous cycle. We are more visible in conferences, and we change our branding into more of a tech brand, into an open source-friendly brand. Frankly, onboarding also becomes easier because these are developers who know these technologies that are used in open source, and we are using common technologies that they've already worked on.
The second big benefit has been that it really helped us understand what open source we are using throughout the company, what our critical dependencies are, what sequence our product roadmaps should be based on, and the synchronization of open source projects with our product roadmaps. This has allowed us, frankly, to really encourage our developers to upstream any changes they make and patches so that we're not carrying huge technical debt. It's helped us sit at the table of key open source projects that we depend upon and make sure that we have some influence in the direction of that project. It's helped us also to make sure that projects that we depend upon don't die and have a healthy ecosystem around them, that we are actually contributing both money and also code and other forms of contribution.
For example, we work very closely with the Yocto Project because our set-top box depends upon it. We work very closely with the Cloud Native Computing Foundation because our journey as a cloud company and using cloud services really depends upon the likes of Kubernetes and Prometheus and tracing. We want to be at the table so we understand the latest developments, we can influence them, we can actually recruit from that community, work with other commercial ecosystem vendors there, cloud vendors, et cetera.
The third benefit has been, as you can imagine, we are such a huge company and we grew up by acquisition. We have so many silos and so many different cultures and so many different engineering teams, and we were really duplicating and reinventing the wheel. Frankly, I wouldn't be surprised if we had five of a certain kind of library or three of a certain tool. Open source, by using that methodology inside the company, has allowed us to create more collaboration inside the company and break down silos.
I'll give you two examples. There's a project called Vinyl DNS, which is DNS as a service under Michael Winslow's organization. This was a small group of people maintaining this service that was used across the company. We opened it up such that the users of that product can actually do pull requests and contribute to that project, thereby not waiting on the project maintainers to do all the changes, but actually to be empowered to make the changes themselves.
There was a second project called Helio, which helped us really bring two competing teams together. It's a lightweight IP video project, and we allowed these teams to be able to work better together by focusing on one single project and allowing both teams to do pull requests on that project rather than creating two duplicate projects in the company. That team also then opened it up and said to the entire company, "Hey, we have this project. If anyone wants to get involved, anyone wants to work on this project with us, you're more than welcome to." You actually create this collaboration inside the company, which helps you then when you get out and open source the project, you already have a built-in community. Frankly, it really, really helps the company break down some of the silos.
Let me tell you that it hasn't been without challenges. One of the challenges often is the executives get it because it's strategic and at a conceptual level, they understand open source is strategic, important. I think developers love it because they are now being trained in school and they come out enjoying working with open source. But often middle managers who have to deliver on budgets and projects say, "I don't have time to allow my developers to go work in open source or attend an open source conference or contribute back or upstream patches. It takes too much time. Frankly, they have to move on to the next product or the next project." Working with middle managers to really educate them on why it's important to reduce technical debt, to work on the latest level of the software, and to really be able to use the latest innovation and leverage group work has been a challenge, and I'm still working on it.
The second challenge companies often encounter is their legal teams. Legal teams, by their very nature, are all about risk mitigation, identifying risk, mitigating risk. Legal teams often don't want you to contribute back to open source. Executives sometimes who don't understand open source will say, "Why are we enabling our competition?" Or, "Shouldn't this be patented? Shouldn't this be protected rather than opened?" It is important to engage with them to make them understand the business benefits of open source and how the business benefits outweigh the risks. Frankly, one of the things we have done is to create an open source advisory council that comprises many different functions so that we can have a very multi-point argument and discussion about whether we should open something or keep it inside the company. The business really needs to make the decision. Legal's job is to tell you about the risk, but you as a business need to say, "This is important to the business. We understand the risks. We still want to go forward with it."
The last is open source as a whole, and here I'm wearing my Linux Foundation hat. Open source as a whole, sadly, is still not very diverse: diverse from the perspective of gender, diverse from the perspective of color and global, and there's a lot of work to be done to make open source more diverse. It's less than 10% women. It's less than 5% African American or Hispanic or other underrepresented communities. Some of the problem is that the projects tend to be very technical and not put enough thought into documentation, onboarding, code of conduct, and doing the right thing in terms of welcoming people, in terms of including people in the project.
The second challenge often has been that code is valued above everything, and not all forms of contribution are valued, such as documentation or legal or marketing and community building. As all of you know, open source will not succeed without all of those components being included. One of my charters as part of the Linux Foundation board is to work on how we include more people in open source because it benefits us all when we have different perspectives, but also it lifts everyone into this digital transformation that we're going through. If our world is getting digitized, then we need all kinds of people involved in the creation process, not just one kind of person.
The help I'm looking for from all of you is, one of the things I still struggle with, and I think the DevOps organizations are really fantastic at this, is: what metrics do I use to show that open source is making an impact in the company? The line of sight often is very long from, okay, I consume this open source project, but how did it help the company? It takes a couple of years to demonstrate that it's reduced the company cost by millions, or it's helped us get to market two years faster than we would have, or it's helped us get more to markets that we wouldn't have, that sort of thing. I'm still working on metrics. If you have any ideas for good metrics, I'd be happy to listen to you and talk to you.
The second appeal that I'd like to make to all of you is: how many of you actually use open source and contribute to open source? Quite a few, right? You can't avoid it. I would ask you, Hacktoberfest is coming up. You can do hackathons in your company. Encourage people to start working in open source. It's, I think, good for technical careers, but it also helps you keep in touch with projects that your company cares about and your company uses all the time.
The third one I would say is continue to push the envelope with your legal team. Keep educating them. Keep asking them, "Why does this need to be inside the company? Why does this need to be closed? Why can't we open it?" Not everything is a key differentiator, by the way, and a lot of things need to be on the outside. Why? Because all of us consume a lot of open source, and we should be giving back. That's one thing. The second is, frankly, you get the benefit of amazing thinking from across the community, which improves that product, which helps you share the burden with others and makes the world a better place. I would say not just projects you depend upon, but also things that are good for the common good, good for humanity in general. Let's give back and work on moving the envelope from an open source perspective.
With that, I want to close and I want to say, frankly, open source is at the heart of many, many digital transformations. As companies move to cloud, as companies move to doing more DevOps, they're encountering more open source-based products, open source-based tools, and understanding how open source works and how to integrate that into your companies and making that a part of your success strategy is so very critical. Thank you so much for your time.