John Mark Walker

Good morning, y'all. I'm John Mark Walker. I'm the founder and director of the Open Source Program Office at Fannie Mae. We do a variety of DevSecOps plus innersource, plus external open source community building. And I'm here to talk to you today about general transformation, but also how innersource can and should play a role.

And I have to tell you, I'm already a little bit annoyed from this morning, 'cause I didn't hear innersource mentioned one time this morning. Did anybody hear innersource mentioned? No. Okay. Does everybody here think they know what innersource is? Okay. You've heard the term innersource. Cool. Alright, so we're going to talk about innersource, what we're doing at Fannie Mae, and along the way I'm going to do some light agile bashing and DevOps bashing — but only a little bit, only as an insider who wants them to be better. And I'm going to talk about ways that we can kind of close the circle and have a complete transformation with the help of innersource practices along with agile and DevOps methodologies. So that's kind of the basis of this talk.

Why current transformation falls short

So let's look at transformation today. If you think of digital transformation in most contexts, most companies have been on this journey for decades now — at least since the early 2000s. The prime sort of start date for most agile transformation was the 2010s, I think you could say. And if you got most IT leaders in a room alone and asked them how it's gone, they would say there are some benefits, there are some things they've gotten, but it's just a little bit not quite what they expected. Maybe the complete value isn't entirely there. They wish it could be better. At least that's the sense I've gotten. I've been at a couple companies that have gone through substantial agile transformations, and you can certainly see the benefits, but a lot to argue is that we haven't entirely tackled the incentive blockers yet.

So let's go back to the Agile Manifesto and how it started. Essentially the thrust was: why don't we treat people better? Why don't we focus on software? Why don't we do the things that we want to do, as opposed to all these things that we're told to do that don't really add much value? One of the central problems I've noticed — well, before I go there, let me say that agile kind of takes on the spirit of religion. Once you get to a certain point, it's like, "Oh yes, now we've finally gotten agile right." Every good technology movement kind of breaks into a religious philosophy. Agile, DevOps, innersource, open source — they're no different. I just think it's good to call it out that what we're really talking about here is opinions.

But when you come to what has happened with agile and agile philosophy, a lot of it comes down to what I would call the "No True Scotsman" phenomenon. Whereas, well, agile didn't go as planned. Well, it was implementation that's the problem. And you end up with a lot of different implementations, and in many cases you end up with a scenario where what was created or called Agile ends up becoming a bureaucracy that replaced the bureaucracy that already existed. And I think the key phrase here is that even though the intentions were good, it never fundamentally achieved the desired outcome, because they never really got to the incentive structure that was responsible for a lot of the bureaucratic processes that we have already and still have today.

In other words: how can we really transform the way we work if the incentives are there to prevent that from happening? If we look at how incentives usually manifest in corporations, you end up with a very top-down driven structure. It's not really designed to have the kind of collaboration that we want to have. It ends up being, you know, the tail that wags the dog. And I think when you get to the heart of it, the original writers of the Agile Manifesto — again, their hearts were in the right place, very idealistic — but I don't think they really understood how corporate structures work. Or if they did, they didn't quite understand how that manifested in the way we write software and manage software.

And then we have DevOps, which is basically: how do we take agile and marry it with automation? I remember in the early days of DevOps, it was very much a philosophical approach about busting silos, bringing people together. It was very much a philosophy of collaboration. And then over time, somehow it morphed from that into, well, here's some CI tools for you. And you heard Gene this morning talk about how he changed the name of the conference because one of his friends said she couldn't make it. I think the quote was, "Why would I go to a conference that's about a bunch of CI tools?" And that had to hurt a little bit to hear that. But I think over time we've seen how DevOps has morphed from a philosophy or philosophical approach to a set of tools, platforms, and very engineering-centric. And also, I would point out, like agile, it did not address the motivations and incentives that made bureaucracies what they are and why they exist.

So the question is: why is this problem so difficult? And I think it gets back to the old adage that managing people is hard, changing process is hard, behavioral change is very difficult. But essentially you have to get to the core question of what drives people, what drives the processes that exist today, what drives the hierarchy that exists in most companies, and how do you make changes in that hierarchical approach to produce something closer to what we want to see, or something that we would enjoy working on more, or that has better results.

Innersource: borrowing from open source

So I had a couple of ideas. One of those ideas is around: we can take examples from what happens in open source software. So let me go back to innersource. One of the things I found out was kind of surprising to me — the term innersource was coined before the Agile Manifesto was written. It's actually the year before, by Tim O'Reilly, who had noticed all this success happening in the open source space. Tim started saying that, well, why don't we take all that good stuff that happens in these collaborative communities that produce all this great software, and let's transpose that into an internal enterprise environment.

And it was an interesting idea. I remember at the time, I thought it was kind of silly because I thought, well, we're all going to be doing open source anyway, so what do we need this for? Little did I know that actually most people don't know how this works. But it's very simple: take open source methodologies, apply them behind the firewall, and see if it can work in terms of internal team collaboration.

It took a while to get off the ground. After Tim mentioned innersource, it was kind of bandied around as a term for a while. But then over the years, no one really talked about it for a while until about a little over 10 years ago, when a couple of folks at PayPal started investing in what's now become the InnerSource Commons. If you go to the InnerSource Commons site, you'll see a lot of great material there — a lot of good training material about how to implement innersource at your company, different organizational patterns you can copy from. It's a great resource for everyone, if you want to go to innersourcecommons.org.

And so when you think about innersource, and you think about this idea of borrowing from successful open source collaboration — what does that really mean? When you look at the success of open source as a collaborative model, the lessons that we can learn and apply are: transparency is good. That's always good. There's the idea that you have openness in terms of who's participating, openness in the code. But it's not just about code, and it's not just about transparency. One of the core drivers of successful collaboration that I've noticed in open source communities comes down to: everyone who's participating is participating more or less on an equal footing. No matter what company they come from, whether they're individuals, whether it's somebody from another organization, another community — when you participate in an open source community, you're participating as an equal partner, with the assumption that you have established credibility to be working at a high enough level to be a leader in that community.

In addition to transparency and the frictionlessness of designing and writing code, it's about the fact that everyone who participates can win a spot if they prove themselves. And I think that's one of the key pieces that I want to harp on as we go through our innersource journey and how we're implementing it at Fannie Mae. By the way, I highly recommend you go to opensource.org/osd and look at the Open Source Definition for yourselves. There are some key things there — like, I can fork a project, anyone can take it and use it for their own purposes — things that generally speaking can be difficult to do in an internal context, but I think are important nonetheless.

Center of gravity and the marketplace of ideas

So let's look at the model of developing an innersource system. When you look at the modern SDLC — and this is a very simplified picture — in general, what makes a successful developer ecosystem comes down to: did you get a center of gravity? Did you get a place that people are naturally pulled to to develop the thing? Did you develop a natural center of gravity that people were pulled into, as opposed to pushing everything in 500 different spots? If you think about a well-developed open source or innersource ecosystem, these things happened naturally because in the quote-unquote marketplace of ideas, the best ideas won out and things gravitate towards the winners. And this is the thing that you want to replicate if you have a successful developer ecosystem internally as well as externally, regardless of context.

The scary side of innersource for traditional IT

One of the things I think is key when you look at the incentive structure — or lack thereof — is to look at why innersource is maybe not accepted as much as it could be. A lot of the things that we mentioned as benefits from open source collaboration can actually be kind of scary in an internal context.

If you think about it: okay, we're not going to have any limitations on downstream usage. Anyone can use this code for any purpose. We don't care what you do with it. You can contribute to us, or you can take it and fork it — whatever you want to do. Well, companies often don't like that behavior. Companies want a prescripted way of doing things in an IT context. If we're going to make a tool, here's the tool, you use our tool. We don't want to do this forking business, and we certainly don't want people taking it off and running with it in different directions. The idea, of course, is that if you do it right, eventually it'll come around because you've created the type of transparency and center of gravity that things will coalesce around anyway. But you can understand why someone from a traditional IT background would not like this behavior. Companies want consolidation more than anything else.

The other thing that can be a little scary is if you take to a logical conclusion the idea that all participants are equal and all parties have an equal say, then you could also say that someone can participate regardless of title, regardless of organization, regardless of what VP they report to. And that can be kind of scary, because most companies are very top-down, very hierarchical. You have a set of leaders, you have teams that report to those leaders. They might work together, but everyone wants to make sure that their leadership gets credit for their work. In an innersource model — one that is flowing freely — you don't know where all the teams are in the organization, you don't know who your prime contributor is going to be. You might have someone from a lesser-known part of the org come in and take a leadership position on your project, and you have to think: are they going to steal my thunder? And you get to the whole incentive model that is broken at companies. So that's another scary thing.

If you look at transparency and frictionlessness and take that to its logical extent — the whole concept of the open marketplace of ideas, the total transparency — again, it can work against your traditional mode of decision-making. If I have the backing of leadership and I create one project or product, and it's in competition with something from another part of the org — am I going to be embarrassed, or is my leader going to be embarrassed, if this other project wins out? Again, that works against the usual incentive model.

And finally, most successful open source communities have very specific rules of engagement. Things like succession: how do you attain leadership positions? How do you attain the ability to review and approve code contributions, for example? Those don't just come on the first date. You have to show that you can do the work before you're afforded that privilege. So there are governing rules, and you win that on the merits of your work, and that's how you get leadership. But again, that's kind of orthogonal to how a lot of companies work, where decisions are made according to your title and where you come from in the organization. It's not necessarily working against each other, but it's definitely not the same. It's a very different decision-making practice.

The flip side: why innersource is rewarding

On the flip side of this — if you do it right and you do it well, it's very rewarding. The idea that downstream usage is unlimited and anyone can fork your code — the logical extension is that you're actually providing the basis for great API-driven, accelerated innovation taking place around platforms of substance with a high degree of gravity, because you developed that center of gravity. This assumes that you've developed that center of gravity and have this system of transparency working well.

By the same token, if you say that all participants are equal and everyone has equal say, well, all contributors are valued. One of the things we've been struggling with in internal systems is: how do you incentivize or inspire people to be better, to work better? Well, here's a model where you can ensure that if someone has good work and they're contributing it, they're rewarded, they feel valued, and they're incentivized to continue down that path. They're not automatically pushed aside because they didn't have the right title, they didn't come from the right part of the organization.

When you think about a marketplace of ideas — yes, it can be a little bit chaotic. But on the flip side, healthy competitive ecosystems can be good for innovation. And if you have a model where you have psychological safety and the freedom to fail, you're going to have a much easier time in that kind of environment with a team saying, "You know what? This didn't work out so well. We'll join efforts with this other team, or we'll make sure that we'll push our work into this other platform." If you've created the right kind of environment and incentivized the right behavior, that's what you'll end up with eventually.

And finally, when you have the right kind of governing rules and they're transparent — everyone understands what they are, they're clearly published, and people follow them — you end up with a culture of accountability where someone achieves something because they work with others. We have clear rules for how people engage with each other. And that accountability serves a purpose of making sure that everyone understands who's doing what. These are, in my mind, very beneficial things to strive for. And I'm wondering why more of us don't really work towards that standard.

Closing the transformation circle

So in the previous version of this talk, I spent a lot of time just kind of raking agile over the coals, but I didn't really want to do that so much today. But I did feel it necessary to point out that when you look at the types of transformation that companies undergo today, and the work that they've done under the term transformation — it looks to me like if you take what's been done with Agile, and the automation that's been implemented under DevOps, and then you marry it with the transparency and the incentivization structure of innersource, it seems to me like it completes the circle. It completes the transformation story.

I'd really like to see more companies take this approach and look at how innersource principles can add to what they already have with Agile and DevOps. In fact, I want to show some examples to work through some use cases from Fannie Mae where we've done exactly that. In general terms, you can take what you already have, accelerate it, get the innovation you're looking for, get the incentive structure you're looking for, and have people more valued and willing to participate in these systems.

A contrast: scrum-style product development vs. innersource

Let's look at a difference in example of what I mean when I say that innersource works differently from something like agile transformation. If you look at your typical product development scenario — the classic one in agile scrum — and before I go down this too far, let me just say that there's a time and place for everything. So it's not necessarily better on the right; it really depends on the time and place. But it serves to show the contrast.

In a typical agile scrum model, you have your product and engineering in one place, you have your customers in another, and then you have the idea of coming together in the middle to talk about it, to iterate, to improve. Continuous improvement, yada yada — it's all very nice and good. The contrast here is on the innersource side: there is no internal scrum and iteration. Everyone is at the table to begin with. Your customers are providing real-time feedback on the ideas with the engineers as they're doing the work, because everyone is contributing to the project as an equal partner. You end up with a scenario where the traditional technology vendor model — the vendor makes it, they give it to the customer, the customer uses it, the customer provides feedback, and lather, rinse, repeat — is replaced. Whereas in an open source or innersource model, the customer is on the product side already. They're part of the process, they're contributors, they're valued partners. It's a very different relationship than a producer-customer relationship.

Y'all with me so far? Alright. So it's an interesting product creation model that I think we can use in the right times, in the right scenarios. You only use this model when the customers and the people participating have the capability to be equal partners and can actually feel comfortable adding value at that level. That's not always the case. So I don't want to say one is always used over the other; it really depends on the makeup of the people that you're working with.

Fannie Mae's progress

One thing I want to talk about with the success we've had at Fannie Mae is — this is my favorite graphic. I love it. I added the fire to this. But we're not done. This is a work in progress. It's a marathon, not a sprint. But at the same time, we have some things we've done that we feel pretty proud of. We have over 20 innersource projects. We've developed communities of practice and special interest groups — and that's kind of the marriage of Agile and innersource I was talking about. I'll get into that in a bit more detail. But all in all, we have momentum. We have people doing work, we have real examples of collaboration that resulted in a better outcome.

Example 1: The Python special interest group

I mentioned the marriage of Agile and innersource. This is one example of that. One thing we realized is that our existing processes for change management really weren't conducive to the type of innovation that we wanted to incentivize — and certainly not conducive to an innersource style of working. So we started thinking about what's a way to use what we have and expand it from there to get that innersource spectrum going.

We started with a known problem — an area of Python where we were getting a lot of tickets. We started thinking in terms of special interest groups and finding avenues or venues to bring people together to at least talk more. And then from that, we can spur actions that would turn into projects or real actual work.

So the first example was the Python special interest group. This stemmed from a single problem that we had — developers were entering tickets related to Python usage because they couldn't figure out how to get it to work, because we don't make it easy. They were inundating our support teams with tickets, basically just informational in nature: how do I do this? How do I do that? Our support teams came to us, the Open Source Program, and said, "We don't know what to tell these people. What should we do about this?" And we said, well, we actually create the opportunity for these peer-led groups who can come together and help each other. That way these peer-led communities can become the support of record, and they can record what they're doing in different forums online and elsewhere in Teams.

So we started with a special interest group. We were able to cut the tickets in half that were reported from developers. We were able to grow a community. And then from that, we actually had three innersource projects that were started that then consolidated, because I realized that there was no point in them all trying to solve the same problem differently. So it's an example of a workflow where you go from problem, to here are ways to bring people together, here's how innovation is spawned as a result. And this is how you get that incentive model that works — that results in an innersource success story.

Example 2: The Clean Dependency project

Another one I talked about is a Clean Dependency project. Stop me if you've heard this story. You've got applications that use open source components and libraries as your dependencies, and you get these scanning tools, and the lights go blink red, because you've got thousands of vulnerabilities. Scary.

In some cases there is no path to remediation, because the libraries are essentially — there is a disagreement with the people that produce the libraries and they say it's not a vulnerability, and the scanning tools say differently. So what do you do in that situation? Well, what we did was we recognized that some of our developers were stuck because they couldn't remediate. We thought, well, okay, why don't we actually use an innersource model to modify some of these libraries — reduce the attack surface area of these libraries and make them available so that we can get past the scanner and solve a real problem.

And then we realized that, by bringing people together and talking about it, it occurred to us that we can take this and push it into an actual open source project external to Fannie Mae, because other people must be having the same problem. So this is an example where you go from people coming together to solve problems, creating innersource projects, and then linking it with the outside world in open source communities. You're getting not only agile and innersource, but you're getting the real honest-to-goodness open source effect when you're going external. We've got two libraries; we're expanding from there. We're looking at incubating in foundations. So we look forward to having more to announce there.

Takeaways

The takeaway slide here is that innersource can accelerate your existing transformation. It can accelerate your innovation internally if you get the incentives right.

One point that I would like to emphasize: when possible, when you're looking at key stakeholders to make these work, it's important to get doers — people that actually write code or write documentation. Those are always at a premium.

And then finally, all systems require belief and embrace that this is the right way to do things. Getting people to that point can be difficult, but I would argue it's essential to making this whole thing work. At some point it comes down to: do people believe that this is the right thing to do and the right way to go? And do we all feel good about doing things this way? If you can achieve that, that is the difference between good and great.

And then finally, as I mentioned before — entrenched processes and behaviors. Very difficult to dislodge, but you've got to try. So with that, I'll leave it there. Thank you.