Raj Fowler and Rochelle Norton

Raj Fowler: So we're here to talk about DevOps and SAP. Excellent. We're going to hopefully have a good time. We're going to have a bit of crowd interaction as well. Rochelle, introduce yourself.

Rochelle Norton: Hi, I'm Rochelle Norton. I'm the head of ERP for BAE Systems Applied Intelligence.

Raj Fowler: And I'm Raj Fowler, formerly head of product delivery at BAE Systems, and currently principal consultant at DevOpsGroup. And I am fashioning the unicorn socks.

Rochelle Norton: Just need some glass slippers and then everybody can see them better. Can we get a whoop for Raj for the unicorn socks? Come on. Thank you.

Raj Fowler: Excellent. Rochelle, tell us about what we're talking about today.

Rochelle Norton: Really, we're going to talk about how we applied DevOps principles in BAE Systems Applied Intelligence as we redesigned how we do IT and SAP. And also, on the back of it, how we enabled business transformation because of those changes.

Raj Fowler: Excellent. We're going to get some crowd participation here, so I'd like everybody to stand up. This is the stay-standing game. If whatever is true, you stay standing. If it's not true, you sit down. The first one is: stay standing if you've ever been involved in a project where it didn't deliver what it was expected to deliver, or delivered stuff that people no longer wanted. Okay. Mad.

Raj Fowler: Next question is: stay standing if you've ever suffered the pain of moving work from one functional area to another, one team to another, from dev to ops, to architects, to analysts. All right, everybody. We're getting a bit of empathy here with your story already. Good. Stay standing if you've ever been in an environment where people do not speak up. They're not prepared to say that the emperor has no clothes. Everybody's still standing.

Raj Fowler: Wow. Okay. Stay standing if you've ever been involved in a really big project where everything is hanging on the line. Wow, nobody sat down. Oh, one person sat down. Really? I don't believe you, Steve. And finally, stay standing if you've ever been involved in a project where it's all about the technology, it's all about the implementation of a technology. Right, okay. I think we've got a lot of empathy. Your story's going to resonate. You can all sit down now. Thank you very much. Or you can stay standing. I don't know.

Raj Fowler: For us, some of the themes that we want to talk about in DevOps are all about having a purpose, having a goal. We've heard that; there are a lot of people already with some really great talks who've talked about shared goals and things like that. With having the right people in the right teams doing all of the work, and creating an environment where you're really empowering the people, you're really getting the best out of the people, with a view that they're able to deliver a flow of work, a consistent flow of work, where technology is then used to accelerate the work. We're going to take these five areas here, and we're going to talk about the experience that we had at BAE Systems.

Rochelle Norton: Does everyone remember Bill? You remember Bill? Great. Fab. I like Bill. The fact that I like him so much that on really tough days in the office, I will always say, "What would Bill do? What would Bill do in this situation?" So obviously... Fab. Anyway.

Raj Fowler: Let me start with this then. You're talking about bad days. Do you remember January 2017?

Rochelle Norton: Unfortunately, I do. It actually started as a very peaceful day, just came back from Christmas holidays. It was really peaceful in the office. Had my morning coffee, sat at my desk. Really, everything going swimmingly well. And then suddenly I get a call from Raj. Ring, ring. Do, do, do, do. I pick up.

Raj Fowler: That wasn't in the script. Ring again.

Rochelle Norton: Ring, ring. "Hi, Raj. Good morning. How can I help?"

Raj Fowler: "Hi, Rochelle. I have a probletunity for you."

Rochelle Norton: A probletunity?

Raj Fowler: Yeah.

Rochelle Norton: All right. Okay. What's it about?

Raj Fowler: The cyber division, Applied Intelligence, they've got an SAP environment with about 40 people looking after it. They're in a bit of a mess, and they want to take some of the learnings that you've had in applying DevOps principles and practices, and they want to apply it there. What do you think?

Rochelle Norton: How big is the problem?

Raj Fowler: Little bit.

Rochelle Norton: Little bit. Okay. All right. I think, yeah, sure. When do I start?

Raj Fowler: This week.

Rochelle Norton: All right. Fine. Thanks, Raj, for the really long notice. Thank you. Bye. So obviously, Raj sounded really excited about this probletunity. I wasn't quite as excited. Anyway, as a really good employee, the next few days I drove into Guildford, thinking, "It can't be that bad, right?"

Rochelle Norton: Got into the office. No one knew me. No one knows my name, no one knew who I was. And then apparently they've got three P1s, two projects which are blocked, service requests backing up, and a lot of incidents being raised. In summary, SAP was down and the whole business could not transact. No one could log orders, no one could book time, we couldn't bill. Really fundamental stuff.

Rochelle Norton: So I suppose I sit down at my temporary desk and think, "Right, this can't get any worse than it is." So I go back to my car and what do I see? A parking ticket. So it was one of probably the worst days of my life, and that was about two years ago.

Rochelle Norton: I suppose we'll be sharing with you how we got from there to here. Essentially, within that SAP system, we've halved the number of incidents that we have. We haven't had any P1s for the past 18 months. And also, at peak, we deploy about 100 changes a month. So quite a big jump from where we were.

Raj Fowler: Excellent. We're going to take you through the story, and the first thing we're going to talk about is purpose and outcomes and goals. One of the things we did is we went from business requirements, and those of you who've read "A Seat at the Table," Mark Schwartz's book, he talks about the requirements from the business: thou shall do all of these things, and the dutiful IT will then just go and deliver them without asking any questions. Well, we had to switch that. We had to start thinking about working backwards from the customers. What are the outcomes that they're looking for? What is it they're trying to see? What insights can we get by the way they behave? What is it they really want? So we had to focus on outcomes over requirements, and we needed to look at what are the measures and metrics that help us define whether we're doing well or not. Things like customer satisfaction and business performance measures. Rochelle, tell us about how you worked backwards from the customer.

Rochelle Norton: I suppose the most important point here is that we had to contextualize the change that we were about to do. Essentially, we were about to make a massive change in IT, and we had to make sure that we got enough buy-in from our senior stakeholders. Whether it's the FD, the HR, the operations director or whatnot. We started changing the way we talk in IT. We stopped talking about requirements. We started talking about outcomes.

Rochelle Norton: We also used a lot of different words like partnerships. We stopped taking orders. We were more like partners to the business. And also, I think the most enduring change that we've done is that we've built customer communities that straddle across the business, and that covers, for example, record to report, hire to retire, order to cash, blah, blah, blah. That enabled us to actually see what the whole business needed, and then it allowed us to prioritize and also align what we do in IT to achieve all those goals and essentially make sure that we're doing the right things for Applied Intelligence.

Raj Fowler: Excellent. But your customers weren't your only problem, right?

Rochelle Norton: Nope. The biggest problem actually that we found is the structure of the teams. Essentially, we had a lot of teams in silos, and they had conflicting and competing goals. Obviously, we had to resolve that. If we are to deliver a lot of changes and also expedite the transformation, then obviously we had to change that.

Raj Fowler: One of the things we had to do was get the right people in the right seats. We had to change the teams model, and you've all heard about the project-to-product conversations that have been going on this week. We had to move people from their traditional silos, business, PMO, dev, ops, et cetera, where you had distributed accountability, to more product-centered teams. The whole Amazon two-pizza team idea. Those teams having lifecycle accountability, whether that's for the product or for the platform. We had to really focus on the product over projects, and these guys all had to be aligned to the business performance measures. Rochelle, what do two-pizza teams look like in SAP?

Rochelle Norton: I suppose I'll start a bit before that. We did something really bold in terms of our restructuring. We truly merged dev and ops. At the point, there were dev people and then there were ops people, but we said, "Well, actually, we don't want that." We merged the two, and essentially there was only a DevOps team. I think that helped us truly resolve a lot of constraints around engineering and also passing the buck to another team.

Rochelle Norton: When we've done that, we then structured scrum teams around products and value streams. What does that really mean? I was talking earlier on value streams like record to report or hire to retire. What we did was look at the skill sets of the teams and said, "Well, you're great at HR. You're going to be in this scrum team. You're great at development, and we also need you in that team." Essentially, that allowed us to streamline how we engineer and how we develop.

Rochelle Norton: In terms of future-proofing this model, we also made a conscious decision that we will only hire people who are really keen to learn and also really proactive, so that we could get to a point where we have E-shaped people within our team, so that we truly become a cross-functional team. And to your point, two-pizza teams, what we said: we should only really have a maximum of seven or, worst case, 10 in a scrum team, so that we could see all the work and then it's effectively managed by the scrum masters.

Raj Fowler: Excellent. But we needed to give some power, some autonomy, an environment for them to be able to speak up. And you had a challenge as well because you had teams in the UK, teams in Kuala Lumpur, big distributed team. How did you address this?

Rochelle Norton: This is an interesting one. I suppose when you try to shuffle where people sit and what they do, you somehow get a lot of concerns from people. So we started with empathy. We had to make sure that we were looking at it through the lens of the engineer and also the lens of the scrum masters. Three key things we did here. First was coaching, personal coaching, specifically with managers. Because I suppose there's a cultural difference between UK and other geographies. In KL, they're more like, "I'm your boss, so you do what I tell you." So we had to change the way that they managed their people, so that they're more like servant leaders and they actually take the whole team onto the journey. Again, did that across the teams.

Rochelle Norton: The second part was around training, and this is really important because when you talk about DevOps or Agile or Lean, everyone has different meanings to that, and we had to make that meaning consistent across the teams and essentially contextualize that in terms of what we're trying to do. We went to KL and did a lot of workshops around the teams and got their ideas out. Then we shared with them what we think it meant, and essentially also rolled out ways of working to support that consistently.

Rochelle Norton: I think the third one is around open communication. For example, we had a really successful project go live. It's like retrospectives, but perhaps we go a bit more person to person. I will speak to an engineer and say, "What did you think? Why was that successful?" Or if we had a really bad P1, I would say, "Why do you think we badly failed in that?" Essentially that allowed them to be more open, and we built the trust across teams.

Raj Fowler: Excellent. A real key for us was to unlock the power of the people. Those of you who are familiar with Westrum's typology of organizational culture, there's a number of dimensions that's talked about. We had to go from a very bureaucratic organization, very process-driven, kind of failures weren't really tolerated, to a more generative culture. It's interesting, those of you who are familiar with this, you see that novelty in a bureaucratic culture causes problems. Well, funnily enough, in SAP, novelty still causes problems. It's not the right environment to really innovate. Really, we started to focus on being more generative over bureaucratic, and measuring things like employee satisfaction. These were the things that really helped, and I guess all that training and coaching and mentoring provided that environment. But what we really needed was that safe place, right? Talk about that a little bit.

Rochelle Norton: I think with any transformation like this, it takes a toll on people because obviously you're asking everyone to work faster, work differently, also engage differently with the customer. You're really asking everyone to step outside the box. I think especially for the managers, they were sometimes clueless or just tired. It was important to let them know that you truly support them.

Rochelle Norton: I think I was quite lucky when we were initially doing the change, because I could have an open communication with you. I could rant about so many things. And then you're pretending to be Erik at that point. But anyway. Also, we had other product owners and we were able to share our problems and then just bounce ideas and just a support group really.

Raj Fowler: Yeah, and that was not just important for the teams and the product owners and the engineers and stuff. It was important for me as well. I was leading the transformation. It was the first time we'd kicked anything like this off in the company. Having a sponsor on the exec board who was really 100% bought into this was really powerful. And also, James, who's in the audience here, we had a call every Thursday night, and many times I'd be ringing him saying, "I can't do this. It's too hard." Nationwide just talked about culture change is hard, and I was finding that really, really hard. So I had to not only provide that safe place for my teams, but I needed that safe place as well to be able to just say, "I'm really struggling," and somebody just holding me up and keeping us going.

Raj Fowler: So we had to really start thinking about accelerating value realization. We kind of went from the whole traditional big batches, long cycles, high investment, high risk kind of ways of working to reducing everything into smaller batches. Having less risk, lower financial impact, with much faster feedback cycles. We took this iterative over big bang, all-in kind of way of working with things like lead time, deployment frequency, MTTR, et cetera, as things that we started to measure. What did this look like for SAP and some of the changes you did in SAP?

Rochelle Norton: Historically, SAP implementation has been something that actually kills some of the companies out there. It makes them bankrupt. You're talking about 24-month deployments or even, sometimes, five-year deployments. We had to really truly think. We said, "Well, we're not doing that." Essentially, the first step was really to see, well, actually, what's our constraint? Really, the constraint was sometimes SAP itself, because it's tightly coupled as a system.

Rochelle Norton: We again looked at it differently, and we said, "Well, actually, is it because we do not know what we do as a team?" The first step was to visualize work. All that lovely agile type practices. We had Kanban boards, we had burn-down charts, blah, blah, blah. That helped us actually know, well, why are things moving so slow in the software delivery lifecycle? On the back of that, what we did was look at our engineering practices. How are we designing things? Are we slow because we're so risk-averse? Are we slow because, again, we're building too many things into one release? Et cetera.

Rochelle Norton: For complex changes, we would've done TDD-type development, so that we know that either people can't do it or can they do it. That shortened our build cycles. The other thing was around on and off switches and also dynamic programming so that when we release into production, if it's really bad, we just turn it off. That allowed us to, again, be a bit more open to risks.

Rochelle Norton: One key thing that I wanted to share as well was that, depending on the type of work that we do, we found that actually some of the changes that we implement into production are repeatable. We did an exercise and said, "Well, what are the things that's repeatable and then are predictable?" Then essentially made those standard catalog changes. That meant that some of the things that used to take 20 days, we did in one or two days because we knew it will work. Again, that helped us improve satisfaction from the business.

Rochelle Norton: Also, a very key thing, and I think this is a good thing for our senior managers, make sure you know when to negotiate requirements out, because perfection is always the enemy of quick deployments. You can't really get everything perfect.

Raj Fowler: Good. And I guess, in trying to speed things up and everything, the tools and technologies didn't help to start off with.

Rochelle Norton: It wasn't our focus. We didn't really focus on the technology. We wanted to first master what we do, and then also bring everyone into the journey. But I think it certainly slowed us down at the back end of the transformation because we knew, well, we're not so fast to deploy. Can we do this faster? Or can we quicken our adoption rates? Or can we improve our delivery times in terms of functionality? So, yeah, it's the usual problems that you'll see other companies have as well.

Raj Fowler: This is really interesting because then we had to look at how do we accelerate the momentum. The type of things we had: we had your physical infrastructure, your big releases. Most of the stuff was manual. You had unhappy engineers. We had to move to a much more... We didn't go all-in cloud, but we started to use some cloud capabilities, started to look at automation capabilities. We had happier engineers, and we were doing the hard stuff more and more often. As part of that, we started looking at more adaptive technologies, architectures over rigid ones. These were then still accelerating the lead time, deployment frequency, MTTR, and reducing change failure rates. What did that look like for SAP then?

Rochelle Norton: First thing on deployment: we implemented a tool that helped us manage our code versioning, and also it helped us back out code if something really went wrong. The name of the tool was Transport Espresso. It really helped us quicken our deployment times.

Rochelle Norton: The second thing was around architecture. Because we knew, and we had to accept, that SAP is tightly coupled, we had to say, "Well, actually, for some of the non-differentiating processes and functionality, can we actually move that to the cloud?" Then we looked at the likes of, for example, SuccessFactors, Planview, Concur, blah, blah, blah. And then really truly pushed that into the cloud. That meant it's quicker to make those changes because it's just configuration.

Rochelle Norton: The last one was around adoption. I think this is a big issue for us because imagine if you change one process in SAP, you have to implement it to all your employees, and that's difficult. It takes time for people to know that. What we did was we used Enable Now to help with guided end-user journey so that you might still want to do an initial training, but because of this tool, then they know what to click and when to click it, which helped us with adoption times.

Raj Fowler: Excellent. We're going to fast-forward a little bit. We know what it was like in January 2017. What was the run-up to January '18 looking like? I believe there was quite a big transformation going on.

Rochelle Norton: Yeah, there was. It's interesting how 2017 was obviously very difficult, and then you think, oh, it's going to get easier. But anyway, obviously, we had to respond to a business change, and there was a drive to change the whole business operating model. What that meant was who reported to who, how we report our statutory accounts, blah, blah, blah, had to all change.

Rochelle Norton: Usually these types of things are planned 12 months and then executed in 12 months. We didn't have the time. We only knew about it about maybe August. So we said, "Well, actually, we're going to go out to market, ask all our suppliers, and see can they help us with it." And they all said, "No, we ain't helping you. You're on your own. It's impossible."

Rochelle Norton: So we didn't have a choice. The internal teams that we took on this DevOps journey had to deliver the change in three months. I think really truly applying agile methodology, having the team within one team as a DevOps team, actually helped us drive that. We did that in three months, and we opened SAP one day earlier with very minimal incidents. We got nominated for a Chairman's Award for that because it was a big feat.

Raj Fowler: Right. You didn't tell me that earlier.

Rochelle Norton: I didn't.

Raj Fowler: Right. To wrap this up, because that was really hard, a big part that came out was that interface with the customer, with the business. Them having to have the buy-in to say, "Right. We're going to be making changes. You've got to be ready to do user acceptance testing. You've got to be making sure you're really clear on your requirements. We've got to be there when we're doing all the data cleansing." They had to put just as much skin in the game as the IT teams did.

Raj Fowler: To summarize this up, how does this all fit together? We went from focusing on the requirements and all that kind of stuff, to outcomes. We went from distributed teams, distributed accountability, to lifecycle accountability with the two-pizza teams. We went from a bureaucratic culture to a more generative culture. The flow of work, we really went from these big bang releases, big risks, high financial kind of projects, to small incremental changes, more iterative, more adaptive.

Raj Fowler: And then we went from physical servers, manual work, et cetera, to the use of cloud, breaking up the architecture, and more kind of push-button deployments and change with happier engineers. That really made a difference to the measures and metrics, and Rochelle mentioned some of the things: halved the incidents, no P1s for 18 months, 100 changes a month on an SAP environment. That really gave the business the ability to be super adaptive. This is our adaptive IT framework that we applied to the transformation. Rochelle, if you were going to summarize all of this, what would you say?

Rochelle Norton: I think, as with some of the other talks, it talks a lot about culture and then people. For me, the most important thing here is to leverage the people that you have in your organization and look at their strengths. To truly summarize it, I think it's really around that transformation starts within teams. If you could start transforming one team, then that's the first step, and then you move on to the next team, and then you move on to the next team. Then that allows you to just propagate the transformation.

Raj Fowler: Excellent. That's the end of our talk. If you wanted to have a version of that adaptive IT framework, you can scan the code, and then you'll get access. There's a link there to download that. Thank you very much, everyone, for listening.

Rochelle Norton: Thank you.