Marc Price

Hi, everyone. Welcome to our session today, where we'll be discussing a measure for learning at Nationwide Building Society.

If you cast your mind back to DOES Europe, hopefully you would have seen a presentation from our colleagues, Leanne Bridges and Mark Rendall, where they discussed intelligent control, enabling safety at speed. From that session, you will have noticed that they provided pin boards, and they were one of our early adopters of the product that we plan to show you today.

A bit about myself. I'm Marc Price, and I'm product owner for our measurement and insight product here at Nationwide Building Society within our data and analytics community. The main vision for our product is to enable colleagues with access to relevant data and metrics to support a culture of data-enabled continual improvement within our ways of working. And yeah, Zsolt, over to you.

Zsolt Berend

Thank you, Marc. I'm Zsolt Berend. I'm really happy to be here. I'm a business agility coach at Nationwide Building Society, specializing in data and insights, working with Marc and helping the organization on the journey of becoming a learning organization. I'm co-author of "Sooner, Safer, Happier." Back to you, Marc.

Marc Price

Thank you, Zsolt.

So yeah, what do we plan on covering today? We're going to start with a little bit about our journey, how we've transitioned from a startup kind of ways of working to a more fully fledged product as to the way that we work today. Then I'm going to cover a couple of observations that we've incurred over this time around how we're transitioning from anti-patterns to patterns. Hopefully, we're going to give you a good demo of our product and show you some of the metrics that we've provided to our end users. And then we're going to end with a voice of our customers. So you're not just going to hear from us today, you're actually going to hear from the users of our product, and how they are using it within their context across Nationwide Building Society.

With that, let's go on to give you a bit of an overview of our product. If you can cast your mind back now, back to pre-COVID times when we were all in the office around whiteboards. What we observed at Nationwide was we had a lot of manual reports off those whiteboards in the local context of each of their teams. What we saw was they were measuring outputs: how many tickets could we get done in a sprint, for example?

As we came into the start of 2020, Q1, what we quickly found is that we needed a product. At the time, we decided to partner with the Sooner, Safer, Happier team. What was key to this was getting that leadership buy-in, so we worked together with our leaders within the ways of working team and within our data analytics community. We were given the opportunity to experiment, find new data sets within the society, and try some things out. What we were key on saying was nothing was a failure at this point, because we learned from every experiment that we did, even if we didn't have the desired outcome.

We started to bring together a small team within the society. We decided to build internally rather than using an off-the-shelf product. We wanted to use the capability that we already had.

Coming into Q2 of 2020, what we started to do was get our product out there and seen within the society. We followed the Sooner, Safer, Happier pattern of invite over inflict. As a central team, we started to run show-and-tells with users that we identified across the society who wanted to improve their ways of working, and we really focused on those early adopters. Like I mentioned before, with DOES Europe, the team in intelligent control were really a key early adopter of our product. We started to invite them to show-and-tells. Instead of the central team providing the insight, it was actually those early adopters that started doing that for us.

As we go into Q3 of 2020, what we started to see was that we were really able to unlock the measurability of the flow of end-to-end work across the society. One of the key decisions we made up front was bringing in our delivery information from Jira, which we had adopted at the time, together with our run organization that was using ServiceNow. We linked those two systems together to give a full end-to-end picture of flow of work across the society. You can see an example of one of the charts that we produced early on, and the trend there is really getting better over time. We were able to reduce that end-to-end flow of work lead time.

Coming towards the end of 2020, we really started to see the adoption of our product take off. Where we'd attracted the early adopters, we started to cross the chasm and really start to bring in the early majority and the late majority into using our tooling. That was capped with us becoming a finalist at the UK IT Awards, which was really good recognition, boosted morale, and we could see that the tool was becoming used and fundamental within the society.

Coming up to the start of 2021, in Q1, what we really decided to do now was target the C-suite with extra insight. We created our CIO scorecard. Instead of just putting metrics on a matrix, we decided to use case studies at this point. We would partner with the early adopters and users across the society that had adopted our product and come up with those great messages of how the data and the insight had improved their ways of working. That really got the C-suite excited, and they wanted to adopt this themselves and started to access the data directly in self-serve.

Now, coming up to the present day, Q2 of 2021, it's been a great journey. You can see that the adoption of our tool really has taken off. We now have over 900 users at all levels of the society accessing our data on a day-to-day basis. What's great to see is that we're actually seeing our end users now create content themselves. They're creating their own insight boards on the flow of work, quality, and the alignment to the golden thread, which we'll cover later on in this presentation and hopefully give you some examples of that. As you can see, a lot can change in a year to a year and a half. It's been an incredible journey, and one we're not at the end of. We have big plans for our product, and we continue to add new features, remove technical debt, and continue with our product.

Let's go on to some patterns and anti-patterns that we've observed during our time on this project so far. Like I said, we're not at the end of our journey. We can see we're moving from anti-patterns to patterns. What we wanted to reduce was the measuring of outputs, so how many stories are being completed in a particular sprint or how many features are being delivered, to measuring the outcomes of that: what's the value that's been delivered as part of those deliveries? Adopting the better value, sooner, safer, happier concept as well.

The next one is all about reducing that kind of centralized and often manual reports, like I said at the start of our journey, where you're siloed within your team. We wanted to reduce that and offer that self-service discoverability of data, so that users can start to interrogate the metrics themselves, create their own insights, and share that within their own context. As you saw on the last slide, we're definitely starting to see that take off within the society.

The next one: weaponized metrics. No one wants to weaponize or feel that metrics are being used to weaponize. One of the key concepts that we've always pushed at the start of the product was this isn't about comparing one team with another. This is all about local context and enabling that measure for learning, focusing on maximizing the outcomes and that key learning that then you go and share with that other team. Instead of comparing yourself to them, you're actually learning from them, which is great to see.

The final one: what we wanted to do was remove disconnected datasets. We wanted to create a single truth of data that all can use. We're starting to see that unlock measurability of flow, quality, and value. It's been an incredible journey, and these are some examples of some of the patterns that we're moving to on our journey.

With that, I'm going to pass over to Zsolt, who's going to take you through some of the metrics that we've been delivering.

Zsolt Berend

Thank you so much, Marc. The first dimension we provide service on is around quality, around better service and better product quality. These are mainly production quality metrics around incidents in production, mean time to restore, change failure rate, and others. It's all about the quality of our product.

The value, which is in the heart of why we are in business, is bespoke to the particular products within the society, and it is measured through OKRs, objectives and key results, which we're going to hear a little bit more from our value realization office team in the voice of customer section. What we provide for them as a measurement and metrics is around alignment: how the work that the team is working on is aligned or not aligned to the strategy. This is about whether it's high alignment or low alignment, the percentage of work, the percentage of stories aligned to the top of the strategy, the top of the portfolio outcomes.

Then we have sooner, which is all about flow, all about end-to-end flow, visibility of flow, understanding how long it takes when work starts, when work is committed, how long it takes to get to the hands of the customer, to production. Also, flow efficiency, which is the time work is in working divided by the end-to-end elapsed time, which is usually at large organizations quite low, less than 10%. This is good information, and we have been on the journey to unlocking measurability of flow, end-to-end flow, so we can see impediments, queues, and flow efficiency end to end. The chart on the right is about a really positive trend: lead time is halved for a particular value stream.

Then safer. This is about not only speed but also control: speed and control. This is about continuous compliance. As Marc said, one of our very early adopters, the intelligent control team, started using our dashboard to show risk across the society, control risk, how it's been mitigated, and compliance.

And then happier. This is about colleague happiness, which is the heart of what we do as ways of working. If colleagues are happier, then the customer is going to be happier. There is a direct positive correlation between colleague happiness and customer happiness. These are not measures in isolation. None of these should be measured at the expense of the others. As in the State of DevOps reports, we see that if teams have mediocre performance, only looking at sooner, only looking at getting new features out at a much faster pace, then it is going to be at the expense of quality. The quality is going to go down and also colleague happiness is going to go down. This should be considered as a balanced measure because it's all intertwined and should be measured, all of them, and not just one in isolation.

Thank you. And now we're going to show you our product. We're going to be doing a little bit of demo of our different services we provide.

Marc Price

Thanks for that, Zsolt. What we're going to do now is give you a brief demo of some of the metrics that we've been creating as part of the product.

As Zsolt mentioned in the better value, safer, happier slide around the different metrics that we've been providing, this is all around value. What we've enabled the team to do, using the chart on the left-hand side here, is see how many stories that are being created by that team link into that golden thread.

The golden thread for us is linking stories that are being created in Jira right up to the top of the portfolio around the outcomes that are being delivered and the strategy that the society is looking to implement. What you can see here is the team adopted Jira around January 2020. At that time, they were able to link in around 25% of their stories into the golden thread. As time goes along, they've started to adopt new ways of working, and they've been able to achieve around 80% to 70% of their stories linking into that golden thread. You can see in September, as we're halfway through, around 45% of their stories are actually linking into that golden thread.

Why is this important? We want the team that are actually doing the work on a day-to-day basis to see the impact of the work that they're doing for the society: that work linking right up to the strategy and those key outcomes that we're looking to achieve.

On the right-hand side, we have a measure as a combination of work that's being delivered. This team has been able to link in 75% of their work right through to the golden thread since they started adopting new ways of working. Why is that important? Again, we've partnered with our value realization office to really govern this metric in the society. We would like teams to be able to link in at least 80% of their stories and the work that's being done within that team into the golden thread, to help with the value realization that we're trying to achieve within the society.

The last chart in this case is the top down. What this is showing is the percentage of objectives that the society is looking to achieve, or this team is looking to achieve, and what percentage of those have at least one story linked into them. We don't want any orphan objectives that might have planned due dates and no work being done on them.

We just wanted to give you a quick overview of that, and now I'm going to pass back to Zsolt, who's going to give you a brief overview of some of the other insights that we've created as a product.

Zsolt Berend

Thank you so much, Marc. This is another board, another service, and this is concerned with flow. We talked about this as a relevant dimension. Flow, end-to-end flow: how long it takes when work is committed to work delivered into production, in the hands of the customer.

The top of the board is a reflection of the end-to-end flow of work mapping, or in other words, the value stream mapping for the particular teams. We differentiate, as you see, blue and amber bars. We differentiate wait, so tickets in waiting, versus tickets being worked on. What you see on the bars is the actual average time spent for tickets in these particular states, so the cycle time per state.

This is popular because teams are using it to drill down and identify impediments to flow and how to overcome flow. They are looking at whether, for example, ready for development is a big queue, ready for deployment is a big queue, or tickets are spending too much time in blocked/on hold. Then it drives discussion: what are the impediments and how can we collaborate more? How can we shift work left? How can we reduce lead time? Equally, if the time in analysis is big, that will drive conversation around whether we are doing big up-front analysis and not working together with the developers on what we have, and getting more stuff to done before we start something new.

The one at the bottom is around distribution of flow: the different types of work and how much focus we have on technical debt. For example, whether we only do feature development as a digital feature factory, or whether we have enough focus on technical excellence and the distribution over time, whether we have a balanced backlog or not a balanced backlog. Again, teams are using it to drive conversations around whether they have a healthy backlog.

The next board I want to show you is around key metrics in DevOps, which were defined by the State of DevOps reports and published in the study in the latest book, "Accelerate." It is around main key metrics around release lead time. We talked about lead time, from when work starts until production, and we see here a good trend. These particular teams halved their lead time from work started to production, so their end-to-end lead time.

Change failure rate is the success rate or failure rate in terms of production releases. Mean time to recover is around how long it takes to recover from an incident based on different priorities, the number of hours it takes to recover. Deployment frequency, which is really important for us, is the percentage of application releases into production. Particularly, we are looking at the percentage of applications having at least one release into production once a month, and whether it's a positive trend so that more applications are releasing more frequently.

This is not a comprehensive overview of the different charts, but just wanted to give you a taster around the different dimensions and the different metrics that we are providing to colleagues at each level at the society. With that, I would like to invite our customers to tell us what they think about our product.

Voice of Customers

I work in the Lean Portfolio Management team, and a fundamental for us is to be clear on how every piece of work contributes towards the society strategy, making sure that teams are really aware of the true value of their work, helping them prioritize and focus on the right outcomes.

ThoughtSpot has really unlocked this ability for us. It gives us an overview of where teams have or haven't aligned their work to the strategy in Jira. It lets them drill down to identify which pieces of work haven't been aligned, and they can quickly correct these. One of the really key things that makes it work for us is the BVSSH team itself, because they're always ready to make fast improvements or give support where needed.

Hi, I'm Mel. I'm part of the team helping to build and enable intelligent control as a way of working. Intelligent control is a collaborative approach that ensures for the work that we do, we understand our risk as early as possible, and we embed the right control capabilities at the right time. My focus has been on providing improved visibility and traceability of our controls position, increasing the confidence for our end users across the society that risks are being mitigated appropriately. This is all about measuring what matters.

We've been using ThoughtSpot to enable this. With data from Jira and ServiceNow feeding into ThoughtSpot, we've been able to create interactive pin boards, creating data and insight from multiple sets of data visualized in one place. By having this controls data in ThoughtSpot, we can more easily and quickly identify impediments to flow whilst the controls are being implemented. Where there are releases, our customers can filter the data for a release and check the status of controls prior to that release being approved, which will speed up any go/no-go decisions. Within just a click of a button, we have the ability to more easily drill down on the data and link back to the source data, which allows us to explore the data further for better informed decisions. Of course, it means that we have that audit trail all the way back to the source data too.

Hello, my name is Craig. I work in payments, and I help our teams improve how they work. Where our teams are achieving regular production deployments, we're encouraging them to measure how work flows within the team. Specifically: how long does it take for work to go from in progress to in production? How are they decreasing the amount of time, on average, that it takes to go from in progress to production? They've been achieving this by making ongoing improvements and reducing batch sizes.

Marc Price

Well, that was great hearing from our customers on how they're implementing our product within the society. You're going to hear some final thoughts from our leaders across the society. Myself and Zsolt are going to remain in the chat, but we'd really like you to stick around and hear those final thoughts. We wish you all the best and enjoy the rest of DOES US. Thank you.

Richard James

Hey, it's Richard James, Ways of Working Enablement Leader here at Nationwide, and I just wanted to take a minute to give a perspective from myself on the importance and the impact of measurement and insight here.

What I've observed is that actually the team itself are acting as an exemplar, an exemplar for a lean startup product development approach. They have found excellent product-market fit. They have well and truly crossed the chasm, and they are now using viral engine growth to really maximize that sense of impact here for our colleagues as customers.

Importantly, from my perspective, the proposition they're selling is also supporting, itself, our business agility agenda. If I look at the Golden Thread initiative, really seeking to support that high alignment, deriving high autonomy for teams, and the dashboards that also support a better appreciation of flow, a better appreciation of the causal relationship across better value, sooner, safer, happier. I couldn't be more proud of both the work the team have done to create the dashboards, as well as the quite profound impact these dashboards are having on providing measures and insight for teams to improve in their context across the society. Chapeau to the team. You are amazing.

Nationwide Leader

In a large organization, it can often be difficult for people to really understand how the work they do every day relates to our bigger, more strategic objectives. We're on a journey at Nationwide to help colleagues understand how they can better prioritize their work, focus on value, and improve flow right across the society, all for the benefit of our members. We call this our Golden Thread initiative.

We've already seen that using the relatively simple metaphor of a golden thread has brought to life for our people the importance of making sure that their work is visibly aligned to our strategy, to ensure that what they do every day makes a difference, not just to the organization, but to our 15 million members.

But you don't need to take my word for it. You've heard in this short presentation that we have the data to prove it. It's great to hear our people talk about how they've found having greater access to measurement and insight so powerful, and the benefits it's brought in terms of making work visible and measuring progress.

As a mutual, making sure that we're focusing on the right things for our members is at the heart of our purpose, and this has been a real game changer for us here at Nationwide. Being able to clearly see information on release frequency, end-to-end lead times, wait times, and production quality is helping strengthen relationships right across the business, dev, and ops teams. It's already helped our understanding of how we can better connect work to value and continuously improve the quality and pace.

On top of that, for teams to be able to clearly measure and track their work against their objectives has been quite empowering.