A Trail of Breadcrumbs: Virgin Atlantic Airways' Journey Towards Agility
This isn't a talk about...
-how we've got it nailed at Virgin Atlantic
-Puppet or Jenkins or Azure DevOps or Teams vs Slack or how we only use perfectly-chiseled T-shaped SREs or The Golden Framework etc. etc.
-the right way to "do it"
This is a talk about...
-what to do when you've been told to fix the un-fixable
-what to do when you don't know where to start
-what to do when what you're doing doesn't seem to be working
Chapters
Full transcript
The complete talk, organized by section.
Nic Whittaker
Welcome to A Trail of Breadcrumbs, Virgin Atlantic Airways' Journey towards Agility. My name is Nic Whittaker. And I'm Michelle Moss. Hello gorgeous.
So who are we? Well, no matter to us if you're flying on a budget or in upper class, we will make you feel the most loved. We'll do it with Virgin flair, whoever you are and wherever you're going. So we fly passenger planes and cargo planes to a little over 30 destinations, and we also do package holidays. Prior to the COVID situation, we were reporting a revenue of just short of three billion, which is an improvement on previous years, directionally, and a year-on-year increase in passenger numbers. Post-COVID, well, you'll have seen the news. We're not flying passenger planes at the moment. Although the good news, we're planning to restart next month to a small number of destinations. However, on the upside, this has been our busiest cargo period on record.
Michelle Moss
So we have around now 9,000 employees, though this is now being reduced by a third, with around 90% of us furloughed. And within technology, there were 220 with a heavy supply chain, and this is now being reduced to 165. Our structure within technology is we have our CIO, who has a VP of transformation.
Michelle Moss
Nic reports into her, and then along with testing, dev, and DevOps, I report into Nic.
Nic Whittaker
So what's this talk about? Well, I've been to many of these events. I've sat where you all sat, not at home, obviously, in an auditorium. I've seen companies like Adidas, Barclays, Jaguar on the stage explaining how DevOps they were, how they had these cool pipelines that were deploying billions of times a second into production, generating huge value for customers and internal value for the organization. And I'd sit there where you are, and I'd think, "Well, that's all very well and good, but how do I get from where I am to where you are? I've got no funding. I've got no buy-in," and it sounds a bit like what you've been challenged with is a standard three-tier web stack. I've actually got an eight-year-old EBCDIC flat file mainframe at the back end. I've got Oracle Solaris servers. I've got BizTalk. I've got MQ. I've got Azure. I've got AWS. I've got no idea where to start. So this is our attempt to describe how you can navigate that journey. What it isn't is a talk about how we've got it nailed at Virgin Atlantic. We haven't. We're on the journey, but there's a lot of work to do. It's absolutely not a talk about what tools or frameworks you can buy that will make you agile or make you DevOps. And it's absolutely not a talk about the right way to do things, although hopefully we'll be able to impart some lessons as to how we did it.
Michelle Moss
It is a talk, however, about what to do when you're asked to fix something that is fundamentally broken, and you've got no idea where to start from, and what you are doing doesn't seem to be working. And given current circumstances, we are also going to touch on what happens when big things go wrong.
Nic Whittaker
So how did the journey start? Well, it started with a reorg, and my boss at the time saying to me, "Nick, we need some DevOps." And he gave me this pithy job description, lots of words. After I'd assimilated it, I turned around to my boss and I said, "All right. How are you expecting me to do this?" His response was pretty much, "Go and figure it out." So once I'd calmed down, we asked the question of the first breadcrumb. Why is it we were really having the conversation? What is it that my boss was really getting at?
Michelle Moss
The problems we were having were that Virgin Atlantic was heavily siloed. We ran within divisions. We had technology within commercial. We had a digital team with their own tech people and operations with their own technology being delivered. It was leading to lots of conversations that sounded like, "We want a website." "Didn't you know we've got one already?"
Nic Whittaker
We had lots of technology. We had all the technology. As I said, at the back end, we got big eight-year-old TPF mainframes. We had a huge range of middleware. We had some nascent cloud technology. We had a number of websites. And we had a very complex technical architecture.
Michelle Moss
And whilst you wouldn't expect Virgin planes to have a near miss, we did have different projects in different silos delivering overlapping things tangentially. We own all of the CRMs, and we use 10% of each of them.
Nic Whittaker
Business often didn't trust technology, so it would go off with a credit card and deliver its own solutions. First thing we would hear would be a call to the helpdesk saying, "Quick, our mission-critical Smartsheet application at the airport has gone down. Can you fix it?" To which we'd reply, "But we don't use Smartsheet. We're licensed for Power BI. What do you mean we can't land planes?"
Michelle Moss
To make it worse, leadership would disappear off-site once a year with a bag of bones and runes, and after all of their mystical dancing, would come back with a pronouncement of what we'd be delivering this year to make it better. Do you think we ever over-delivered and unspent on all of this stuff?
Nic Whittaker
Finally, I had a growing awareness that there might be a better way to do this stuff.
Michelle Moss
So our second breadcrumb, what did we need to do? Our silos wanted more from us than we could deliver. We had people using the word agile a lot. We were giving people external training, receiving their Scrum certificates, and coming back to run waterfall projects. We were spending money on large projects which were running for ages and delivering something. We didn't have a day one starter pack, but we did have nice textbook pictures about our process. We also had a lot of high-cost contractors with info in their heads, but nothing written down or reusable.
Nic Whittaker
We didn't have a single source of governance truth. What we did have was a huge landscape of suppliers, creating, testing our code, and supporting it. In short, it felt like we were reinventing the wheel for any new project, and little of it was under central control.
So what might fix the problem? Well, people were bandying around the word DevOps. Give us some DevOps in technology and that will sort it out. That will make us more agile. We had absolutely no understanding in the organization of hidden work or the cost of handing off between complex processes. People were using the word speed. I want it faster. But did that mean they wanted the same thing for the same money only sooner, or did it mean they wanted to just see some incremental value arriving? It was becoming apparent that DevOps wasn't just about a technical conveyor belt, but was something to do with the kind of collaborative behaviors that we were not seeing at this point across the organization. Meanwhile, people were asking the questions, but so what? Why did they want this stuff faster with more agility?
So the third breadcrumb, what did we have to work with? Well, we had some great partners. Air France-KLM, Delta Airlines, were both on their own journey. Air France-KLM, for example, were into their new way of working, and they were rolling out SAFe. So we had some conversations there. Our partners at Delta had just started their own agile dojo, and they were mandating that any projects that delivered into their new cloud stack had to go through training through the dojo. So there were the lessons we could crib there. Meanwhile, I was having conversations with external organizations. For example, Tom Clarke, friend of this conference, taught me everything I needed to know about culture and people. Meanwhile, we had a lot of partners in terms of suppliers, but were any of them willing to step up and be delivery partners? As it happened at this point in time, one of our suppliers was undergoing a contract renegotiation. That gave us the opportunity for a bit of quid pro quo. Could we give them something in return for an alignment on agility and some free DevOps resources? Turned out we could. We also looked at moving some of those project costs around, and also we focused on that training budget. Rather than sending people out, at Virgin Atlantic, we've got classrooms where we train crew. So why didn't we use that environment to help align people on agile best practices?
The fourth breadcrumb, we just had to start. To remind you, we had no money, a sketchy mandate, and no top-down buy-in, but start we did. Turned out we had one project that actually could lay claim to looking a bit agile. It was called Crew Apps. It was delivering tablets to our flight crew. It had to iterate a lot. The crew were exhibiting really mature agile behaviors. However, they were being funded as a project, and they knew that they would soon run out of runway when that project funding stopped, and they had to hand over this living set of products into a legacy operational support mode. Meanwhile, we had a couple of other projects that were starting up. We had a data insights project. We had an external API project. We had that holy grail of projects, the single customer view. All of these things required data governance, API governance, technical governance. They're all deploying to cloud, and they're all going to iterate. So we decided to get our arms around those initiatives. Meanwhile, we brought some external agencies in to give us a benchmark of our own maturity. We also ran over a dozen workshops across the different silos in the organization to figure out what we did well and what we didn't do well. And we also began doing value stream mapping across some of those processes to begin to understand the bottlenecks and the hidden work and all that kind of stuff.
Michelle Moss
So fifth breadcrumb. We're taking off, but how are we going to generate thrust? So we were looking at partners, and they were using assessments and certification within their teams. And this had led to increased autonomy to improve their team's capability, and we wanted some of that too. We then developed this for ourselves to give our teams ownership of their improved capability and to then increase their autonomy. Part of the output here was looking at ecosystems and the ops processes into them, where we were building CI/CD that then needed several days of change management process to reach production. We were then tackling these bottlenecks. Teams were generating their own values, which over time then converged across the teams. This is about people first, tools second, getting ourselves closer to a high trust, low blame culture. So we were now fostering engagement to bring in our naysayers. We set out to do a series of educational one-on-ones, getting a sense of fun into the work and bringing in several areas and partners. This resulted in a raft of resources that we could then draw on as we built up the teams. We had been flagged by our partners that security was a key area to have on board early. In doing this, they supported champions in our teams and ended up being our tightest partners from ops. This then started encouraging other communities to start to form.
Nic Whittaker
So the sixth breadcrumb. At this point in any good narrative, the middle act, you begin to hit the jeopardy. In our case, the problems. What were those problems? Well, there were many, and they started mounting up. We had sponsors who decided that it was taking too long to get the DevOps-y quick delivery that they wanted, and it was costing them money. Now, at this point, we were beginning to generate hard data from our capability assessments. And so we were able to use that to demonstrate that the focus on our teams was the right focus, and that actually it was yielding a sprint upon sprint improvement in things like quality, and addressing technical debt, and improving turnaround times, and realizing some kind of value from the stuff that was being delivered. And we were able to bring those people back on board. We were also trying to have conversations with legacy areas in ops, like infrastructure, who would say, "Yeah, we'd love to help. It's a great idea. But look at our workload. You're going to have to give us more people just to get through this workload before we can even learn to be agile." There's always hidden costs. What do you mean we need more MSDN licenses? Every time we spin up a new Agile team, it would have to go into a new room on the other side of the building. We were just building even more silos. It was taking weeks just to onboard staff, just to get them a lanyard into the building. And meanwhile, those first resources who I'd brought in from our partners were poached by external consultancies selling DevOps. So the one thing that we were learning at this point in time was the toughest lesson really, around resilience.
Michelle Moss
But we'd made real progress. When we stopped to look back over our shoulders, we saw that we'd moved much further than we had thought. We were working with Team Delhi, Team Crawley, and create our offshore capability, and I had this explicitly as part of my job description, and we began seeing our offshore teams setting up their own Kanbans and addressing their bottlenecks. We were working up core platform teams to help delivery teams be more autonomous. We were creating our dojo to optimize, collaborate, and create the best outcomes for those teams. We were moving from Jira and Confluence, QC, ALM, and bringing them and all of our new projects into Azure DevOps. So we were starting to increase our process and testing automation as part of this. And with the capability being owned by the teams, champions from teams began meeting as communities, especially our productionuct owners, deciding how they were going to generate the value that they wanted.
Nic Whittaker
So why were we bothering with this again? Well, we now had a groundswell of people who were getting it, and they were collectively beginning to understand the need to move away from siloed projects to products on platforms built and run by core capability teams. And we were collectively beginning to understand the need to think about this idea of value streams and outcome-based thinking. Which takes us to the seventh breadcrumb, a change at the C-suite. I've been at Virgin Atlantic for around six years. I started in 2014. At that point in time, we had a new CFO and changes in the tech leadership. 2015 brought a new interim CIO and an accompanying reorg in technology. 2016, yet another CIO and another reorg in technology. 2017, a new CCO, a new CFO, a new VP for business systems, VP for ops, and another reorg. In 2018, another new VP in technology. And 2019, we got a new CEO, a new C-suite, including a new CIO and a new VP for transformation. And guess what came with that? Another reorg. However, the new CEO brought with him a new CIO, and this individual had an Agile background and also, crucially, a growth mindset. He and his team understood the need to view all change in a common way from a single place, to reduce the technical landscape, to begin to target core capability and remove what didn't actively support the vision, and to increase the responsibility and repeatability and quality of output from teams, which was really a lot of the work that we'd been doing up until that point in time.
Michelle Moss
So the CEO now had a very clear why, to be the most loved travel company. And in technology, we had the same vision, to be the most loved technology against which our work was now aligned. So we then brought this into our teams, withering the things that were outside of this vision. We started delivering the right things to bring the value early. This was supported by our amazing delivery partners to help us achieve this vision.
Nic Whittaker
So we now had eyes on us from the top down. We'd learnt how to do repeatability at a basic level. We knew we were onto capabilities, and we knew we needed to reduce divergence and waste. We knew we needed a single route in for all demand, and we were beginning to talk about value streams and portfolios. In short, we were approaching a critical mindset change at organizational level which takes us to the eighth breadcrumb. What you can do when you've got that top-down buy-in.
Well, now we were approaching a big cultural shift at enterprise level. We set about changing the way that we thought about change. We established a bunch of core value streams, and each of those was owned by a C-suite executive sponsor. Everything that was going into those value streams or was being proposed for absorption into those streams was being scored against the why using our corporate measures. We insisted on visibility and discussion between those executives across those streams, talking about dependencies and risk and the cost of doing it, and the risk of not doing it. We ensured we had representation from everybody involved who might be accountable in the delivery chain. So change, run, finance, procurement, HR, these people were all co-opted. We synchronized everybody to a quarterly planning cycle. The enterprise PMO was brought from its original home in finance nearer to the change delivery, and that allowed them to have good governance at CEO level and good reporting across the whole piece. And that allowed us to get to a single source of the truth. So what were some of the lessons we learnt during this critical period? Well, we found that if we referred to the historical fact that we couldn't net off demand versus budget versus resource versus outcome, that gave us the impetus to move to a new model. We needed to show that there was a model that was externally relied upon, something that had been proven. Okay, so we started with SAFe. We used baseline scoring against our well-established corporate measures, also factoring in risk not to do so, ROI and all those kind of things, to allow us to deal with conflict in terms of prioritization. We mandated that anything coming in, any investment proposal, had to use that single funnel. We also had folk all across the data, and we dashboarded everything and made it visible all the way up to the CEO on a minute-by-minute basis using our common forward-looking strategic tools. In our case, we're a Microsoft shop, and that's what we were focused on. And during the events accompanying this, we filmed them all doing it, and we put the leadership team in the middle of the room to resolve conflict in front of the entire audience. Repeat.
So in summary, we now had a new way of working. We had a single plan for all the demand that was going into the organization, broken down into value streams that had visibility and dependency at the core of their thinking. That could be broken down into programs and into delivery teams who were already thinking of things in terms of iterations, Kanban, scrum boards, and so on. Now, it's worth noting that at this point in time, the teams that we'd spun up with agility and DevOps were probably only 20% of what was being delivered. Everything else was still Waterfall. However, even those Waterfall teams were still leveraging the best practice, things like test automation, things like stating out their value and reporting those metrics back up on a biweekly basis into the dashboard. So we were beginning to see progress.
Did it work? Well, see for yourself.
Embedded PI Planning Video
At the start of this new decade, we're having to think differently about how we prioritize and change. And what we're seeing is organizations having much more interlinking capability in what they're doing, in their priorities, and their strategy. So today, the PI is about, for the first time ever, it's a first for us, everyone in this organization sharing their assumptions and their planning assumptions to one outcome around velocity. One of the things I really believe in is a growth mindset. So Shai talks about growth all the time, and we will be profitable this year. But for us to remain profitable and to continue growing our organization, we have to have this growth mindset to investment, to planning, and to execution. And this is what the PI is really about. It's about having a growth mindset to adapt, to adopt, and to absorb new ways of working. The first purpose is really to decide what's important for the organization, what will deliver us the most value, the most benefits, the most risk avoidance. But the second purpose is also to get to a particular number. We have a cost envelope that we have to hit, which is 35 million of CapEx this year, which has been decided. It's not for negotiation. And so we need to make sure whatever we think is important fits within that 35 million. I'm most concerned that we'll come out at the end of the day and the number won't be 51 million. It's going to go up again. Everybody started adding things back in. We went from 35 to 40 to 48 to 51 overnight, which surprised me this morning. The thing that makes me most excited is that we do bring it back down, and people start to be the real leaders of the organization. If we get this plan right and we deliver the things that deliver the real value, we all deliver our velocity plan, so we all win. It's intense and it's noisy, and it's quite hard. So yesterday obviously was the first day, so none of us really knew what to expect. So it's quite hard to work out exactly. But today, I would say we're a lot more, I think we're getting there. We are a day and a bit in. Within three hours yesterday afternoon, we closed the gap that it takes most organizations at least a year to 18 months to achieve. So from my perspective, how's it going? It's going phenomenally well. You've got a whole cross-section of the company to come in and really understand what their projects are, what they mean, the value of the projects, and the cost of those projects, and how we prioritize those projects. We've got a huge amount of demand and a small pot of money. You've got all the business partners here, you've got the SMEs here, you've got the budget owners here, and it's making some really challenging and some really interesting conversations. We have a lot of finance representation in this room, which is fantastic, giving our team actually the opportunity to showcase their skills and how they challenge, influence, and make sound financial decisions on behalf of our business. There's been a lot of collaborations, have conversations and really understand what we're trying to do across the whole business, which I've never seen before. So it's really good to understand all the dependencies on each other and different projects that maybe we didn't know about and how they might interact with our projects. There's lots of exciting stuff to come down the line. The difficulty really now is because there's so much great, exciting stuff, is really trying to prioritize what order we need to work through as to what's really going to deliver the most value this year and within year where we can. We're a very ambitious company. There's always more that we want to do. Our eyes are a bit bigger than our belly, I think, a lot of the time. But I think the most valuable thing that's come out of this is the conversations and the quality of the conversations we've had in the room. What I've learned, don't underestimate the culture of Virgin Atlantic. There are very few companies in the world that can go from an old way of prioritization and enterprise to the most modern ways of thinking in agility. We can do it, and not only have we done it, we've done it brilliantly and come out of it still as friends.
Nic Whittaker
So I think at this point it's worth noting that you heard some numbers in that video. Those numbers were all pre-COVID. The world has now changed. Those numbers don't apply. Nonetheless, the outcomes do apply, and we're going to talk about those now.
So we went into that event with, if you like, an MVP. Number one, we needed everybody in the organization to understand that this new methodology was how we were going to do it from today onwards. We needed everybody to understand their part in that chain. We had a challenge. The demand at that point in time on the first minute of the first morning was over 51 million and rising. We had to bring it into a 35 million envelope, and we had to do that within a headcount constraint. And finally, we needed to get commitment that we would do this together today and going forward on the premise that the siloed, discrete way of working previously had not worked.
How did we get on? Everybody understood their roles. You saw in the video, everybody got on like a house on fire. Yes, it was difficult, but we got through it. We hit the numbers. We went from 51 upwards to nearly 60. We pressured everybody. We held everybody to account. We made the leadership speak in the center of the room. And by the end of the event, we came in at 35 million and on headcount. And we got that commitment. And there was nothing stronger than one of the C-suite who came up to me and said, "Nick, I never thought it would work. You changed my mind." So that was a win.
Michelle Moss
So out of breadcrumbs. Just when you think things are going well, you may have noticed that lots of things have gone sideways recently. We've been hit with many challenges all at once. You may have seen in the press that globally passengers have stopped flying. Cash preservation has become our number one focus. We're also managing rescuing customers, refunds, postponing bookings, and starting up cargo-only flights to ship PPE for the NHS. Huge impact was felt by all of our people, but with a culture that has really shone through.
Nic Whittaker
There is an upside. We have had to learn rapid adaptation. We've had to move legacy teams away from stating things in terms of complexity towards a view of iterating on the complex. A good example of this is the infrastructure team that I alluded to earlier. They had a two-year plan to roll out Microsoft Teams across our organization. It was very stepwise, and there was a lot of change that had to be managed during that period. However, we were suddenly in a period where we were furloughing huge amounts of staff, and those staff who were remaining at work were going to have to work remotely. That two-year Teams rollout happened in two weeks from a legacy thinking team who've now changed their mindset. We've also had our teams having to disperse and work offshore and really learn how to coordinate and work highly collaboratively.
Michelle Moss
And there is stuff that we can do. Our skeleton team are focusing on maintenance and governance that we've often neglected, and we're getting our standardized tools up and running for when we do restart our passenger flying at the end of the summer.
Nic Whittaker
So there are still obstacles ahead, and probably help needed. Staff reductions and the huge reduction in our portfolio have obviously hit us. I mentioned we're starting flying again in the next few weeks. Buy tickets and holidays, please. More seriously, we are focusing now on those more legacy teams, the infrastructure teams and so on, and beginning to help them in the way that we've helped our change delivery teams. I think the next big step for us, and the area that we need support and help, is around revising and improving that first pass that we took at value streams. It was a good start. It was a mechanism to allow us to get the C-suite to work together. However, they're not yet true value streams. A number of those value streams are measuring and reporting on the same values. So that's going to be a work in progress. And I haven't dropped my mouse. Let me just pick it up. I'll just leave you with the thought that during this challenge, we still have to constantly adapt and revise and change our way of thinking. And these are all lessons, skills, values that we as an organization have picked up over the last of this journey. Thank you very much. Thank you.