Enterprise-Scale DevOps Between Companies and Service Providers
This is a joint talk between DXC Technology, a large technology Service Provider, and FCA (Fiat Chrysler Automobiles), describing how they combined to overcome many common challenges of DevOps transformation when working in outsourcing arrangements.
As a Principal for DevOps Transformation, John works with external clients and internal DXC organizations to help consult, advise, and drive DevOps transformations from the executive to the team levels.
John has been an IT leader for over 25 years with a variety of R&D Functional Manager and IT Director roles from leading large development organizations in multiple domain areas to running enterprise architecture, data architecture, operational excellence, testing, high availability, and numerous others. As a recognized thought leader and change driver, John is applying his leadership skills, engineering experience and passion to help drive enterprise-wide improvements and continuous transformations.
Christopher has 20 years in automotive IT across a variety of technical and business disciplines. Currently responsible for the Global quality and testing function at FCA. Co-leading DevOps transformation.
John Ediger, Principal, DevOps Transformation, DXC Technology
Christopher Gallivan, Co-leader of DevOps Transformation, Fiat Chrysler Automobiles
Chapters
Full transcript
The complete talk, organized by section.
John Ediger
So I'm John, this is Chris, and this is about DevOps at the enterprise scale between a service provider and an actual client or company.
Wanted to start out with a little background. So this is going to be the format. It's basically Gene Kim's order outline for experience reports. And so we'll talk a little bit about the goal, the journey, and then what we learned. And hopefully you'll get something out of this from specifically what we did in the journey, but also about working between clients and service providers.
How many of you are working with a service provider today, IT service provider, some kind of outsourcing? Just about everybody in here. Mostly anyway. Okay, good.
So let's go through a little bit of background first, and then we'll jump in. So talk a little about FCA.
Christopher Gallivan
Okay. For those of you that don't know FCA, Fiat Chrysler Automobiles, it used to be one of the big three when we were Chrysler. Now we're number seven in the world. We operate in over 40 countries. We have a lot of employees.
Some of our recognizable products that you might be aware of: we make Jeep. That's one of the most recognizable products in the automotive world. Also, the crown jewel of the company is Maserati, so that's another interesting vehicle that we make.
We're really a merger of two companies, as the name of the company suggests. It's not our first merger. We were DaimlerChrysler before. Now we're Fiat Chrysler. The interesting thing is that Fiat is the premier automotive company in Italy, pretty much the only one. It's been around since 1899. It has a very proud, long tradition. And Chrysler was founded in 1925. It was an American company. So we have two very old, longstanding companies coming together into a single company.
John Ediger
And just a few words about DXC. So we're the world's leading independent end-to-end IT service provider. So about $21 billion revenue, 6,000 clients, and that number is probably small, but 10,000-plus Agile and DevOps practitioners and experts. So we're increasing that number, which we'll talk about more soon.
A little bit about the offerings. So there's nine main offering families, 96-plus overall offerings, and this goes anywhere from consulting to app services to cloud services, so the whole gamut end to end.
All right. So let's talk about us just briefly. So, Chris.
Christopher Gallivan
So I'm Chris Gallivan. I've been working at Chrysler, and now Fiat Chrysler, for about 20 years. So I've done 20 years in the automotive industry. Prior to having a DevOps role, I was managing a global testing practice that we had set up across our company. And in July, I assumed the DevOps enablement responsibilities working with DXC.
On a personal note, I have seven kids. So I have a lot of kids. So I know what it's like to deal with a lot of different personalities and challenges.
I'm also a former musician, and at the conference, I know music came up a number of times. I think there's a lot of musicians in IT, and I think there's a lot of similarities between music and DevOps. I think Olivier is also a musician. And I get to travel to Italy a lot. So that's one of the best things about my job, because there's no bad trip to Italy.
John Ediger
Just a few words about me. So I'm a principal of DevOps transformation, so I'm working internally within DXC to help our DevOps transformations, as well as with clients to help them on their journeys.
Let's first talk about, as we go into this journey between our two companies, let's talk about the original goal that we talked through with the executives.
Christopher Gallivan
So I'm not going to read the words of the goal, but it's the same sort of goal that most companies would have that want to move to DevOps. They want to deliver faster, higher quality. They want to lower their technical debt. But behind that goal, there's a lot of other things. I think there's some context that's important to understand.
In terms of the way our organization is staffed and is structured, we're in year 11 of what was supposed to be a five-year transformative outsourcing agreement with DXC. And at this point in time, we have approximately 1,500 developers in North America. If you want to know how many developers we have globally, you could pretty much triple that number, so somewhere around 4,500 globally.
In North America, 80% of our developers are offshore, so we went very strongly offshore. And I was a part of the team that actually did this contract 10 years ago, so I remember, first of all, the contract itself was called The Transformation. Kind of an ominous title, but there's been many transformations at our company over the past 20 years. That was one of them.
Some of the slogans, like any transformation, you come up with slogans. Some of the slogans that we used to talk to our people about what was changing: the first one I remember was, "We're out of the people business." We were going around telling our people that we're not in the people business anymore, and we're going to outsource this work.
"Software development is a black box." So that's what our service providers do best. Let's let them do it, and let's just get out of the way.
And I think the third one was, "We want our service providers to own our applications." These were things that we stated to our people. And obviously, a lot has changed in 10 years. Now that software is actually, ironically, eating the world, I think that the key thing from a client's perspective that we've learned, or that I've learned at least, is that we need to understand software. As a client, we need to understand how you build software in the modern world.
So, Fiat Chrysler, two companies, outsourcing. One of the things about this company is that it couldn't stand on its own, if you really look at the history. And that may sound like a bad thing, but I'm actually happy that it couldn't stand on its own, because as a result of some of these things, like outsourcing and merging with other companies, I think in some ways we're actually stronger.
So today, part of that is we have a partner like DXC, and we're going to talk about how we've partnered together to try to address some of these challenges, moving to a new delivery model.
John Ediger
So the journey, as we engage together in this, we'll walk through, briefly, some of these six areas that we went through in this journey. So to start, we started with executive alignment. And we did this in a unique way, particular to the way DXC treats DevOps. And we started this with what's called a Green Belt DevOps Dojo.
So some of you may have been in the session that Joan and Olivier were a part of yesterday, where they talked a little bit about the DevOps Dojo process. And we adopted this from Target's original dojos when we were here at Enterprise Summit back in 2015. We modified that and made that our own and added different flavors of the dojos.
So I won't go through all these in detail. There is a presentation that you can find online that Olivier and Slavik did from last year in London's DevOps Enterprise Summit. So I encourage you to go find that on YouTube and listen to that pitch. You can hear more about our dojo process.
But the Green Belt Dojo is a unique one. This is one where we get together with the executives, the leadership, the CIOs, the business partner, the executive staff, and we align on what DevOps really is.
You often hear the adage that you ask 10 people in a room what DevOps is, and you'll get 11 different answers. So we centered on what DevOps really is for this company, for FCA. And of course, a lot of people start at the tools at the top, and that's kind of their mental model of what DevOps is. But of course, we all know it's mostly all the stuff beneath it: the principles, the practices, the patterns, et cetera, and mostly the mindset and the culture.
So we level set on that, had some really good discussions about that, and discussions about what leadership of DevOps really means. This isn't a one-and-done thing. Chris and I and account team and others are going to be working continually with the executive team to have constant conversations about this because it's not something that just you absorb and then you get, especially with everything that the CIO does. So this was one thing we talked through in that dojo.
The other thing we talked about is aligning on a set of common principles around transformation.
Christopher Gallivan
So these were the four principles that we agreed with the leadership on. The first one was, we had done a lot of big-bang things in the past, and very few of them yielded the kind of business benefits we were looking for. So very clearly, we were going to take an incremental approach to this.
We have service providers. We spend a certain amount of money. Nobody was foreseeing a huge amount of money coming in to redo the way we do everything here. So there was a clear message from our CIO that this was a transformation in place, not just with DXC, but with all our providers.
Definitely it was a culture-and-practices-led type of approach because the people were one of the more important parts of this, actually the most important part of this. And we wanted to show results as soon as possible, so we had a real bias towards action.
John Ediger
And the last thing that we really emphasized in this Green Belt Dojo, this engagement discussion with the leadership team before this journey started, was an approach, a sort of a methodology to doing transformation.
So I won't go through all the details of this here, but there were four key elements. One is the value stream optimization, so actually picking a set of value streams, of application areas to begin with and iterating on those using value stream mapping, DevOps Kaizen, the dojos that we'll talk about.
Focusing in parallel on the cultural shift and what that means from a leadership standpoint, and we did several activities around that, and we continue to do that. And then having an enablement team, having a core group that doesn't do DevOps for others, but makes sure they enable. So this is everything from a common pipeline to helping to coach, to kind of overseeing as a central group. And then lastly, a couple of levels of governance, and I'll reference that a little bit later.
Christopher Gallivan
So from there, we wanted to prioritize what the core starting point was. Where do we want to begin? There's 1,600 apps within FCA, approximately. Where do we want to start?
So some of the factors that we looked at were the importance of the application to the business, specifically applications where we know we're going to be making investments. Success likelihood, so we didn't want to pick something that was a really hard start where it was really difficult to prove value. So we went with people that, or with applications and areas that we thought we could have a likelihood of success.
Improvement needs. Some of these applications, I remember there was one application where their release cycle was over a year, because we have a lot of plants, and by the time they roll it out to every plant, if you add up the time to market, it could be a year. So those are some apps in need of improvement.
And lastly, of course, if an application's only changing once a year, it doesn't make sense to be a part of this. It should be something that's changing frequently. So with that, we came up with a set of applications that were spread across numerous executives.
John Ediger
Yeah. And then we took those application areas and we focused on this cross-functional learning. So we brought together the QA folks, the operations folks, support, devs, as well as the folks from the business, into a week-long Yellow Belt DevOps Dojo.
So if you heard Olivier and Joan's pitch yesterday about the Yellow Belt Dojo, we've turned that into a virtual set of modules for learning. This was our version of a week-long session where we have games, we have simulations, we have deep discussions, and we actually dive into all kinds of areas of DevOps. So this is just a smattering of some of those areas and practices and patterns that we focused on.
So that was a core kind of a launchpad for all these cross-functional teams to work on. A lot of good feedback from this, and also there's a lot of value in having people be immersed. So that was part of this dojo concept of immersion of what this really means to their job. So a lot of discussions about how this applies to them.
Then following that, we took these initial apps and we did value stream mapping. I'm curious, how many of you do value stream mapping in your organization? Okay, maybe about 25%.
So, I often say that value stream mapping is the most underrated and underutilized Lean and DevOps practice. It's getting a little bit more popular year after year. People are kind of understanding how valuable it is. We believe it's crucial, and it's a part of our main practice at DXC. And so together, we did value stream mapping for all the initial applications that we worked on.
And this is a whiteboarding exercise where you get the whole team together and you're mapping out end to end all the different steps of software delivery from beginning to end, and finding out where those bottlenecks are, where those pain points, friction points, constraints, et cetera. So that was critical.
They actually took, after the whiteboarding, and they decided to create a template and do this in a spreadsheet mode. So they had this captured for all of them. It's not a must, but it worked really well for FCA.
And then with those applications, we went into the Black Belt Dojo. The other dojos are more of a set of trainings and training sessions and discussions. The Black Belt Dojo is about less talking about DevOps and actually doing it. So we get those application teams together with the support and the cross-functional groups, and we focus on actually building out, hands on keyboards the entire week, building out actual implementations based on what was found in those value stream maps, a set of improvement themes that get the teams toward measurable goals.
So that was a key step. Won't go through the detailed process, but you can look at these slides after and also look at the video that I referenced before.
Christopher Gallivan
So what did we find? The biggest outcomes that we found in the initial pilot had to do with what I'll call low-hanging fruit. So one of the first things that we saw was that we spent a lot of time creating packages and deploying these packages.
Actually, we were using a very antiquated way of doing software development where we had kind of a chief developer that would take the input from various developers and then pull it onto his desktop and somehow merge them together. And this process took a very long time. And honestly, as a client, we weren't aware of this time. So this was really part of that black box that I talked about at the beginning. A black box is not such a good thing in this case.
So we saw some significant days to minutes. It was pretty easy to achieve that with DevOps. We also adopted GitHub. So that sounds pretty obvious to you guys, but at a conservative company like us, to pick a tool like GitHub is a pretty big deal. And basically what that allows us to do is do parallel development, which we haven't been doing.
John Ediger
Following on the Black Belt Dojo, and Chris mentioned some of the low-hanging fruit, and that continued and extended, but part of the next step is DevOps Kaizen.
This is a concept that many of you might have heard Damon Edwards talk about several years ago here at this conference and other conferences. To me, it's the secret sauce of DevOps transformation. It's the process of having teams own a specific business outcome or result: cycle time, MTTR, whatever that specific need is. And then do the experimentation, the implementations, and iterations based on what those biggest bottleneck areas are.
So that's the follow-on step that we're taking now for all these application areas.
And then, maybe you can talk a little bit about the... Let me just preface this by when we do these value stream optimizations within each team, often there are items that bubble up and say, "Hey, this is an org-wide constraint. This is an org-wide issue that we need to address across the board." Things that we don't want each team necessarily to solve by themselves.
Even though we want to empower each team to drive their specific numbers, their goals based on their value stream mapping, we want to also solve some things horizontally.
Christopher Gallivan
Yep. So some examples of some of these enterprise-wide things.
We established what we're calling a minimum viable pipeline. We had a lot of silos that were doing this themselves. And while we don't want to tie anybody's hands, we saw a lot of duplicate work that could've been done. I call it the rich and the poor. There were some people that had DevOps engineers that were the rich, but then there were these other important areas that didn't have those engineers; therefore, they didn't have this. So try to put something in place that anyone can take part. Anyone in the company.
Common measures. Honestly, we tried to, what I'll say, Chrysler-ize the measures. But John was very strong and adamant about: don't get too many measures, and these are the industry-standard measures.
I mentioned the GitHub, and we also use ServiceNow, and we have a lot of controls around managing changes that go into our environments. So we've already, honestly, in a very short amount of time, we've been able to stand up something that was SOX compliant with a lot of processes and a lot of manual stuff around it using GitHub. In a period of about a month, we replaced that.
John Ediger
So looking at the follow-on to this first set of steps in this journey, some of the next items, why don't you talk through a few of some of the things that we're doing next?
Christopher Gallivan
I think one of the things that, in my position in our company, we're not really interested in having pockets of this. We really are interested in doing it across the board. And I don't just mean in one region, I mean globally, across the board. So we're definitely trying to figure out a way to scale this across all the rest of the areas.
I think one of the key elements to doing that is the coaching and the DevOps enablement that DXC has done. I don't think it would work in an enterprise like ours without that strong level of guidance.
I think the enablement teams, it's one thing to know that there's a concept of an enablement team, but it's another thing to actually physically staff the team and build those structures into your organization. Those are some of the things that we're currently working our way through.
And one other thing to mention: in a very conservative company, it's very difficult to explain the value to your CFO. It's one thing to explain it to your CIO, but ultimately that means explaining it to your CFO, at least in my company. So we're still trying to figure out what's the magic there. We've heard a lot of people with a lot of examples, but that's really the honest... That's the thing that's going to be our tipping point.
John Ediger
Yeah. That'll be a constant discussion we have with that executive team because IT, unfortunately, still is thought of in many areas as a cost center and not as a value-add center. It varies by group, by team. But we have to continually have that discussion, and you heard some of those pitches this morning, for instance, on how to best do that. So we're madly taking notes on that.
So let me talk a little bit about scaling this now. So we started with our initial set between the various teams, and that's been growing. But to scale this across a larger and larger group of these 1,600 apps, we have an approach that DXC uses and that we've customized for FCA, which is based on phased roll-outs, not big bang, incorporating learning and improvements at these different levels.
So if you look at each one of these lines, that's a set of product groups. That's a product group of many different value streams, many different applications. And so as those sprints proceed, there's parallel Agile sprint work with features, with the DevOps Kaizen continuous improvements based on the goals and the value stream.
There's feedback loops between each of those, of course, with retrospectives. There's feedback loops that go from product group as the next product group takes this on. And then there's feedback loops between each domain or each portfolio, each major organization, so that each learning that happens at each level then cascades to the next.
There's also a set of governance, a couple levels of governance that we're establishing. And we hope to come back together in a year and talk about where we are on this overall path. But there's a lot of learnings, though, that we've gotten out of this so far that we want to share.
So the first learning has been the alignment. And alignment between the client and the service provider is critical. Not only even without an outsource environment, even within one company, that alignment up and down the chain is critical. But it becomes even more important between the two companies on what we're doing, and we're all aimed at the same goal. We started that with the Green Belt Dojo, but we need to continue that ongoing. So that was one key learning, that alignment is critical.
A second key learning, maybe you can talk to this one, Chris.
Christopher Gallivan
So there's a lot to DevOps, even for somebody who's technically savvy, and I think you need to hear it many times before you start to really understand a big chunk of it. I think it's especially true of executives.
One thing we observed is that we had these sessions, many of them were long sessions explaining this, and then we would get a question that sort of indicated that they didn't understand what we were saying. So I'm pretty sure that everybody has seen this before.
I think the key thing is don't be afraid to repeat yourself. Don't be afraid to repeat the basics because you should probably repeat the basics enough times to where you start to get the kind of responses from them that indicate to you that they did understand it. Don't underestimate what they don't understand.
John Ediger
And shifting from a tools mentality to an overall culture mentality, we all talk about it at these conferences. We all know it. But actually doing it within the leadership team and all the way up and down of what DevOps really is, is another story. It's something that we're going to keep and continue to emphasize and continue to focus on.
Most people are starting to get that now, and we'll have to continue this as we scale this across. So that's a core learning that, similar to what Chris was saying, don't take that for granted.
I think that opinion is a hard one to break because we've kind of been in that mode in IT for decades.
Trust between the service provider and the client is critical. So trust means a lot of things here, but the part that I think is most important is the trust that the two organizations are after the same goal. The two organizations are after not just renewing the contract or cutting the cost or driving this. The trust is that we're both aligned to making FCA the best software development organization possible and continuously improving that and meeting some specific goals like we talked about upfront.
Trust that we're after the same goal. And so when you have an outsourcing engagement or arrangement, I think this is one of the things that, in especially a DevOps transformation, needs to be looked at first and needs to really continue to be nurtured.
Leadership may underestimate the transformation, the bottom-up aspects of it. I don't know, Chris, this is something that you mentioned to me and I thought that was fascinating.
Christopher Gallivan
So I mentioned there's a lot of things that have happened in the last 20 years in our company with the word transformation in the title, probably at least 10, 15 major things.
Normally, what a transformation means is somebody's going to come and ask for a bunch of money. They pull together a bunch of people from around the org, the best people. They come together on this special project, and they do this thing to the org.
This is not like that. This is something that honestly would happen anyhow, from my perspective, and it was. It's happening from the bottom up. The danger is that they think that it doesn't require investment because it's happening from the bottom up. So there's a danger that they don't understand that the value of this could be greater than all of those other put together, honestly. So it is a transformation.
John Ediger
And another learning is, I don't know how many times I've heard the concept of: when are we going to be done with that DevOps change? When are we going to be done? Where's the Gantt chart that shows us? When are we going to adopt DevOps for that department? What's the end date?
And we all know, of course, here that DevOps is continuous improvement, so by definition, you can't complete a DevOps transformation. There are milestones for sure, and there are levels. But on the flip side, or the corollary to that, is that starting DevOps, when should we actually start with this "transformation"? So the answer, of course, is start now, start small, start moving forward, and iterate.
So I think these are things that many of us know, but doing this in a large organization, especially an organization with a lot of history, a lot of success, but not an organization that's been used to this type of "transformation" of a continuous nature. This was something that was a key learning.
And then results ownership versus compliance. So this was a fundamental shift that some of the people, even in DXC and in FCA, first thought of this as being "DevOps compliant." What's this set of items that we need to do to go along with this?
And this is going to take constant discussions, but it's really about the team owning the result. This team owns their cycle time, their MTTR, their specific goals, and they leverage whatever means needed. Yes, we have some common pipelines, some common capabilities and so forth to leverage, but having the team own the result is, I think, the most critical part of a DevOps transformation.
And that's a cultural, that's a mindset, that's a leadership. There's lots to that that can't be found in a checklist or a game plan around DevOps. That's the part that you need to focus on. So we learned that, especially here.
For sure to resist the temptation of big bang. That's a natural tendency, especially of executive management, is to big bang the thing and say, "Okay, let's put a plan together and let's go DevOps across the board right away." And then to not skip DevOps Kaizen. I showed that loop around DevOps Kaizen, again, the secret sauce of DevOps transformation. So this was another key learning.
And then lastly, to embrace... This is more about FCA embracing themselves as being a high-performance software delivery organization, right? And that's something that, as Chris alluded to before, that it's not something you outsource, that ability to do software. If you're working with an outsource partner, still know the whole process, own the process, own the pipelines, the value streams.
And then the obligatory challenges or remaining things we need help from in the community. There's three of these that we listed.
The first one is about mutually beneficial goals between a company and a service provider. And there's a long history of decades of having contracts and agreements that don't necessarily align to win-win between the two organizations. There's starting to be a movement in that direction, and we're starting to engage with our clients within DXC to help try to shift that. But a lot of contracts have been existing for some time. So any help on success stories on where that's been done, we'd love to hear more about that.
Agile DevOps shift with decades of waterfall structures and mindset. So many companies run into this where you switch to Agile and DevOps approaches down in the trenches, but the overall planning, POR, budgeting cycle kind of are thought of differently. So I think we've all run across that, but any additional success stories in making that shift, I think sharing out that would be good.
And then lastly, ingraining the continuous improvement mindset across the organization. We do that in pockets, but how do you really do that across the board? We've had some success with that with other clients, but I think this is one that we could all collaborate on as a community.
And with that, we'll see if there's any questions. We'll be here a little bit afterwards to answer questions. So again, I'm John, this is Chris, and thank you very much. Hope you enjoyed this talk.
Thank you.