Rich Jordan

Thanks for coming along. My name's Rich Jordan, as you can see on the sign. I'm not Steve Faloney, and I don't work for Broadcom. You might have read that on the sign out the front. I'm actually a customer of Broadcom, and Broadcom have invited me along to essentially take on this subject. Your shift-left initiatives aren't working. Here's how to fix them.

Okay, so I'm a test engineering manager within a test CoE within Nationwide Building

Society, which is a UK, as you can probably tell from my accent, UK-based business.

Okay?

So, the agenda I'm going to walk through today.

A bit about what Nationwide Building

Society is. Appreciate you guys are from the US. You probably are thinking that we're the insurance people, but we're not.

Okay? I'll explain what we are and what a building society is.

A bit about Nationwide, where we've come from, what our IT estate looks like, a bit of commonality around the challenges that we face.

The changing world of banking and the need for change and transformation, a lot of common themes with the agenda of the event coming out through these things.

And then we're going to tackle the question of everybody shifting left.

What do we mean by that?

What are the common industry buzzwords around shifting left and testing that tend to come out?

And sometimes they work. A lot of the time they don't.

So we'll explore what they are.

And then we're going to talk about what we're actually doing within Nationwide around test engineering and really making some of those shift-left initiatives work for us.

Okay?

And we'll go into a bit of detail about what that really means.

Okay?

So come on clicker.

What's Nationwide Building

Society?

We are a UK-based financial service organization, pretty much.

We're a bit of a bank, but the difference between ourselves being a building society and a bank is that we don't actually have shareholders.

Okay?

We have members.

We're owned by our customers, essentially.

Okay? Our roots trace back to 1846, so we're quite an old company.

Really, the main takeout of this slide is the third bullet point.

We've grown to be the world's largest building society over a series of mergers and acquisitions over the last 30 to 40 years, and especially the last one in 2008 with the banking crisis, where we will have amalgamated with a number of smaller building societies within the UK.

Okay?

And as you can probably tell, if you merge companies together, you need to merge those IT systems together.

Okay?

And that creates challenges.

A bit about our customer base.

So we have a relationship with one in four households in the UK.

If we've got any UK-based people here, you probably know exactly what Nationwide is. You probably bank with us, or you have a mortgage with us or something like that.

We have about 17,000 employees.

We also have a large strategic partner base based out in India.

And our channels include everything that you would expect from a large financial service organization.

So we do current accounts.

We do everything under the sun in terms of channels, mobile, tablet, smartwatches. You name it, we'll do it.

We also have a big presence in terms of brick and mortar in the UK and 700 branches.

Okay, so some stats about what we do in terms of IT, so a bit of an idea about scale.

So in 2018, we made 25,000 production changes.

And year on year, we raised the bar in terms of our resilience in terms of operational delivery.

Okay? Our IT development is broken down into projects and squads.

We're currently going through a transformation exercise where we're moving from a project-based organization to a product-based organization.

We're not unique in that.

Okay?

And there are about 130 change initiatives going on within IT at any point in time.

And majority of those things go through what are 21 squads, essentially. A squad for us, if you break that into business units for what we will do, we do current accounts, and therefore, we've got a squad for current accounts.

We do mortgages.

We've got a squad for that.

We'll do a mobile app.

We've got a squad for that. You get the idea about how we align our squads. Our technologies include a large array of newer technologies.

So we're implementing Kafka and event-driven architecture at the moment. That's the newer technologies and APIs and microservices.

But because of our legacy, we've also got some older technology.

Okay?

And one of the ways we explain it is we used to sponsor the England football team.

Okay?

So what we did is we did a mapping between England football players and when they made their England debut and when we introduced systems into Nationwide.

Okay?

Now, I appreciate context is everything here. Football to me is something very different to you.

Okay?

So if I asked you to name, or sorry, guess when Trevor Brooking made his England debut, you wouldn't have a clue who he was.

Okay?

So I've switched the context, and

I think it will work. NFL drafts.

Okay?

So our oldest system was introduced where the first draft pick was a chap called Ed "Too Tall" Jones. Anybody want to have a stab at when he was picked in the first draft? '74. Who said '74?

There you go. 1974.

So you get the idea in that, yes, we've got some new stuff.

But we've got some challenges around legacy, where we tend to bolt onto a highly coupled state where we still have a legacy dependency on those old technologies.

Okay?

So a bit about me. I traditionally come from a test engineering background and what we will have called historically as a test center of excellence.

Okay?

So anything technical test, that will sit with a team that I will run amongst a few other portfolios.

And that's how we operate.

We essentially deliver capabilities as those teams. As you can see from there, that team is called DevOps.

There's two reasons for that.

The first reason is slightly provocative. It's two fingers up to people in our organization who are saying the right words, they're using the right terminology, they're bringing in a load of the tools, but they're working in exactly the same way as they used to work.

Okay?

The other one is far more PC, and it stands for Data, Automation, Virtualized Environments, Operations, Performance, and Security. It's what we do. Yeah.

They are testing capabilities that... Historically, we've worked as a center of excellence, and we've had a pull effect in terms of people coming to us and using those capabilities.

We figured out that they didn't work, yeah.

We created our own little ivory tower, and we evangelized from the top how everybody should be doing it, and they went off and did their own thing, and it didn't really work.

Okay?

So what we started to do is turn ourselves into a center of enablement, where we're essentially indoctrinating, educating, evangelizing.

We're creating, incubating capabilities that we're then federating out into those areas.

So much more about enabling rather than trying to hold an expertise internally.

Okay?

So that's a bit about me. That's a bit about Nationwide.

Why do Nationwide need to change?

Okay, so anybody involved in financial services, you're probably very familiar with these bullet points.

Okay?

So the Fintechs are coming. Yeah.

The Monzos of the world, the Starlings of the world, they are, to a certain extent, greenfields.

They're born in the cloud.

They have an ability to be agile with a little A that is very scary to us. Yeah.

They can offer new products far quicker than we ever can.

And they're only going to grow in popularity, and we're seeing that over the last couple of years. Coupled with that is something that happens in the UK called open banking. Open banking is a regulatory demand on the banking sector to essentially open up competition.

Okay?

So it's an API based regulation where we essentially need to make it as easy as possible for our customers or our members to switch banks or understand the best offering that they can get from any of our competitors.

And we need to make it as easy as we can for them. Yeah? Which then leads on to the next point, which is customer experience.

If we're not giving the best customer experience to these people, they're just going to use open banking and go off somewhere else, yeah, because it's easy to do that. Cost of IT change, we're always under pressure to reduce costs. That's not a new one. Everybody's there. Resilience. Yeah.

We're a regulated organization.

And not only do we want to be resilient, we need to demonstrate to our regulators that we are resilient. You'll be familiar with a couple of high profile banking outages in the UK quite recently.

Because of that, the regulator is very keen to understand that peers of those organizations can demonstrate that they are resilient in the way that they operate.

So that's a very important thing to us.

And we've also got things like industry trends. Yeah.

We're very aware that our stakeholders are sold certain dreams in terms of the way these Fintechs are working. A new way of working that can generate far greater outcomes.

There's a slight bit of cynicism in the way that I've articulated that in terms of buzzwords, but you get the idea in terms of the way we call ourselves DevOps and the fact that we disrupt the people who are using the right terminology, using the right tools, but working in the old ways.

Okay?

And within Nationwide, we are delivering a much wider, broader transformation agenda called

Something Through, #WeAreChange.

Okay?

Now, this is an organizational change delivered from the top downwards, and you can see the OKRs, if you like, around what that change capability is delivering.

Really, the top two sit firmly within the strategy of what we're doing in terms of test engineering. Yeah. Improve pace and predictability. Predictability is all about test. Yeah.

And we want to do it fast.

So that's a bit about Nationwide, bit about what I do, bit about what

Nationwide, the broader agenda, is doing. Let's try and tackle the question. Yeah. Shift left.

Okay?

So what is shift left? Bit of research for this. I went on Google very quickly, and it's quite interesting that the first 10 hits all come up with shift left is shift left testing. Yeah. It's all about testing.

Why is it all about testing?

And it's an interesting one, and I went to images very quickly, and that was the picture it came up with.

Okay?

And it is quite odd.

If you look at the scale in terms of attention to detail. Attention to quality, sorry.

And if you look at the old way of working, does that suggest that no one paid attention to quality until test? Yes, it does. Which is kind of worrying, isn't it?

And inaccurate.

And inaccurate. Yeah.

And so what the model is suggesting is that the attention to quality is essentially moved to the left-hand side.

Okay? That does beg the question, does that suggest that quality then drops off after development and build? Again, I'll leave that open to interpretation.

Now, what we've then got is a number of transformation initiatives that become very unique to test.

Okay?

And for anybody that works with consultancies, these things will be very familiar with the things that are pitched to your organization in terms of what will transform your test capability, your test velocity, enable you to save cost in terms of your testing costs.

Okay?

And they're not wrong.

There are some slight misinterpretations about certain things.

The obvious ones being things like BDD and TDD as a test-only activity. That's quite a common thing within a lot of the engagements that we see.

But they are outcomes of something that is missing foundation.

Okay?

And really, this is where our journey starts in terms of our test engineering journey within what we're doing in Nationwide.

Okay.

So, what are we doing, and how are we responding to the transformational strategy that Nationwide is asking for?

So what this is is a tester manifesto, and you can see from the tagline at the bottom that this is an adaption of a chap called John Ferguson Smart's adaptation of BDD.

Okay?

This is driven development.

Now, we're not actually trying to do BDD within Nationwide.

We're actually trying to do model-driven development.

Okay? Slight adaptation, but the picture and the statements fit well for us.

Okay?

So, if you cast your mind back to the challenges that we face, we carry a complex, highly coupled IT estate.

We're going to carry a bit of technical debt. Yeah?

And we start to build models to understand where we carry that technical debt, and we acknowledge the fact that we actually carry that debt, and we don't know certain things about our IT estate.

And the only way to address that is to first acknowledge it, and then to start conversations about chipping away about how this thing should be working.

Okay?

And they're the first two points that are on that slide, and this is the crux of what John Ferguson Smart would talk about, for example, in BDD, is that BDD is not about using Gherkin or Cucumber. It's about the collaboration with the three amigos. Yeah? It's not lost on me, by the way, there's four people at the top, but the designer and the business are the same person in that model.

Okay?

If you've got that, you have got a stable base for predictability.

Okay?

We know what we're doing, or we know what we should be doing. Yeah?

We have an articulation of what a common understanding of quality is. Yeah? Quality is subjective. It's in the eye of the beholder or in the eye of the collective, and therefore, for the first time, we are understanding what a quality benchmark means, and we can go for it. Yeah?

We have predictable outcomes.

We know what we're delivering in terms of development, design, and test.

And we can go for it.

Okay?

If we know those things, we have a very good grounding around automation. Yeah? Unless it's predictable, you can't automate it. Yeah? You can't.

So when we've got those things, we can automate.

If we're automating, we can be faster, we're generating lots of feedback.

We're on number four, and we can execute things in our CI/CD pipeline. Yeah?

We're gathering lots of metric.

We can start to understand where inefficiency sits, where we are winning at certain aspects, where more technical debt carries, where we've got uncertainty. Yeah?

We are becoming very driven by analytics.

And where these things are happening, we want to evangelize, we want to celebrate the fact that these things are working.

This is the best way to attack these things.

We've got to stop doing certain things.

We want to share that best practice with the community. Yeah?

This is all about iterating, continual improvement. You get the idea.

Okay?

So this is a 65,000-foot view of what we are aiming to do.

The next few slides are really going to deep dive into each one of them to give you a bit more substance around what I'm really talking about here.

Okay? It's easy saying these things, actually then putting them into practice, that's possibly why the shift left initiatives are failing.

Okay. Come on.

So, problem with test. Ice creams and volcanoes. Testing hasn't really changed in the way that it's delivered in the past 20 to 30 years.

Okay?

We've delivered through ice creams. Lots and lots of functional UI-based testing by probably ex-business users executing happy path functional tests.

Okay?

We've made attempts to do automated GUI tests, but automated GUI tests are very flaky. Yeah?

And not always having predictable results mean they fall over a lot, and therefore we spend a lot of time putting a lot of effort into creating these automated packs, and they just don't work, and therefore we revert back to doing manual testing.

Okay? Integration tests, they look the same as the functional tests.

They're big end-to-end functional tests. Yeah?

We're not really doing proper integration testing or integration in the small.

And then unit tests, a bit in the eye of the beholder in terms of the way that the developer built it. That's the way he tested it as well, and therefore the negative test or the structured testing that they've used kind of doesn't exist. Yeah?

And underpinning this is a transfer problem, where although we will have been on certain courses, got certain certifications about the way we do test, we don't apply structured testing techniques to the way that we work.

And that's a problem. Yeah? It's kind of subjective.

So what we need to do is we need to move from the ice cream to the volcano. Yeah?

We love a triangle in test, and there's a few presentations that, although didn't call out test specifically, had triangles in them in terms of the way that we're attacking this thing.

And what we're talking about is deconstructing the topology of what the system and attacking the functionality where the functionality sits. It doesn't all sit in that UI in the context of the business.

Okay?

So we're doing far more server-based tests.

We're doing far more ETL-based tests.

We're doing far more API-based tests.

We are being far more structured in the way that we're attacking these things because they are smaller components of a bigger piece.

We're deconstructing the monolith essentially from a testing perspective.

Okay?

We're pushing the rigor down into unit testing because if we're doing much smaller components of test, why wouldn't we be doing that earlier?

Okay.

And if we're doing these things and we are strangling, if you like, constricting the amount of UI-based testing we need to do, we've got a much smaller capability or footprint that we need to create in terms of UI-based testing.

Okay?

So we've got highly automated API-based tests, highly automated server-based tests, automated ETL-based stuff, automated GUI-based tests, and we're still going to do a bit of exploratory testing at the end. Yeah.

Because we're only doing testing because we're changing something, and therefore there is a certain degree of a thing we don't know about what we're changing. Yeah.

And it's going to be inefficient to automate it.

So we're going to be doing manual testing. Let's be realistic, right?

So Ice Cream Volcano kind of says, "Let's do away with all UI-based testing." It's kind of not getting at that.

What it's getting at is we attack the estate or we attack the functionality at the level the functionality sits.

Okay?

And therefore, what we're saying is that if we actually concentrate our effort in terms of where we're doing the testing, we can pay far more attention, for example, to user experience at the UI-based level. Yeah.

We don't really want to be doing end-to-end functional testing at the UI because that's not what the user experience is about. Yeah.

We don't want to be proving the integrations at the UI-based stage. Let's do it at the integration.

Okay?

So easier said than done.

This is a few years old now, and it's an architecture diagram of one of the changes that came through the Nationwide estate.

Okay. Lot of systems, lot of interactions, highly coupled, few legacy systems in there. It's a challenge. Yeah.

Where do you cut the scope of testing? You know where to start, where do you stop? Yeah.

We risk never delivering anything if we try to do everything and therefore you just can't. It's a risk-based approach. That's a testing question.

Where do you understand the impact of change? Yeah.

Because let's not treat this as a testing problem, let's treat it as a change impact problem.

Okay.

Now what we start to do, and I like the way that Martin Fowler, who talks about technical debt and technical debt quadrants goes about it, is he comes at it from a design perspective.

Okay.

So he cuts technical debt into four quadrants.

And Nationwide historically have very much sat in the orange and the red quadrants for sometimes the absolute right reasons.

I mentioned the mergers and acquisitions. For the right business reasons, we cut a few corners to make sure that we were hitting certain dates for the regulators, but we created a bit of debt for ourselves. Yeah.

And sometimes we've done them for the unknown reasons, the naive reasons, and that's the bit where the layering comes in. Yeah.

We don't even know what we're doing it.

And a good example of that is this picture in terms of where do you cut the impact of change. It's kind of like saying, if I chuck a little pebble into this architecture diagram, where does the ripple stop? Yeah.

And if I keep chucking pebbles into it, sometimes I'm going to miss where the ripples stop. Yeah.

And inadvertently, I've layered. Yeah. My documentation for all of these interconnected things have become out of date. Yeah. Or they don't quite fit together anymore.

Okay.

And that's a problem.

So going a bit old school here.

When we do testing qualifications, one of the first things they will teach us is verification and validation.

Okay. Verification, do we understand what this thing is supposed to do up front? Validation is testing. Yeah.

Now I'm going to suggest as an IT organization, as a testing function, we do far too much validation, we do nowhere near enough verification.

Okay. To go onto the shift left picture. It's kind of accepted that requirements are not very smart. Designs will be slightly out of date. Yeah. That's not good enough.

And why do we not spend more time doing verification?

Because prevention is better than cure. Yeah.

If I'm missing something in the design, I'm only going to build that thing and hopefully I'll find it in test.

If I don't find it in test, I'm going to find it in production, which is an even bigger problem.

Okay.

So Michael Bolton is a quite prominent voice within a test engineering organization, or, sorry, industry, and kind of what you're saying is true, right?

If this was any other industry, any other engineering function, we wouldn't let them get away with it, whoever they are. Yeah.

And really what we're talking about here, and this is where we get onto Richard Bradshaw's FART model, is a realization that IT systems are complex.

They can do many things.

And therefore we can never know everything about what that thing will do.

But we need to keep pushing the bounds of what we understand, and we need to keep pushing the bounds in terms of our knowledge. Yeah.

And importantly, when we're doing that, we need a way of capturing that knowledge, and we need a way of reusing that knowledge because if we are becoming an ever-learning organization, we can push the bar in terms of what we know and what quality looks like to us.

Okay.

So we model. Yeah.

And our models are living articulations, living specifications for us.

And the important thing for us is those specifications, they live and they tell us impact of change.

If we make a model change, and this is the hierarchy of topology that we actually model at, if we make an API function call change, and we change our models, it will tell us the ripple effect into the system layer, into the UI-based layer.

So we're spanning the business function into the technology functionality or implementation of that business function.

Okay?

And that's a really important differentiator in terms of implementing things like automation, things like data, things like SV.

We're creating the foundations around how this thing is supposed to work.

Because if we know how it's supposed to work, we can validate it.

We can automate the validation. It's easy, right?

And these models that we create, they are not just testing models.

They are models for design.

This is where we're doing model-driven design, not model-driven testing.

Okay?

And we recognize that change will happen. It's inevitable.

And this is why we need living specifications because if we're using Word documentation, if we use Visio diagrams, static documentation, they're only going to go out of date. People are not going to update them.

We will miss some stuff.

And those living specifications, they give us insight far and above ever what we would know from manual analysis.

So what we've got here is an output of a collection of models we've created for...

I mentioned we're doing an event-driven architecture implementation.

This is Kafka implementation about streaming customer data.

Okay?

And what we've done is you can see that we've broken up the topology of our infrastructure.

And what you can see there is the functional path through, essentially, the topology of get customer details.

Now, you could probably do it in a traditional non-structured way and if I was to pose to you and say, "Guess what kind of customer detail transactions that you would create," you could probably have a guess and you'd be 80% there, 90% there.

But we go at it from a very structured, collaborative, conversational point of view where we thrash out not only happy paths, negative paths, data quality paths, all those different things where iteration after iteration, we're coming up with new scenarios around how this thing could work, could go wrong, what technical debt do we carry? How do we address that technical debt?

Okay?

And you can see from there, without knowing anything about our estate, where the complexity sits.

So you can see that BDT is where complexity sits for us.

And what we do, and this is where we use modeling tooling, ARD being a Broadcom tool, is we use test optimization techniques, pairwise you may have heard of, orthogonal arrays you may have heard of, to make sure that we refactor the design, the test pack, so that over time our test pack doesn't layer. It's the same problem with designs, if you like, around if I create a regression pack for a typical system, and six months later I iterate that, six months around, I iterate that again, six months later, I iterate around. Chances are that's proportionally grown or multiplied out with duplication in it.

Because I'm not going to do the analysis manually.

But these tools allow me to do that.

They allow me to refactor at runtime those test packs to keep them slim, to keep them lean.

So you can see there that I've got the functional analysis around what this journey is.

You can see what the optimization takes from a lean testing perspective.

So although I could run 6,675 tests, I don't really gain a lot over and above running just the 256.

So although I could automate those things, and we have automated a lot of the all paths, because they're very quick, they're APIs, basically. I'm creating a lot of analysis for myself for not a lot of gain. I can do 256 and get the same output.

Okay?

Now, four weeks later, if we didn't have this capability, we'd probably go into rooms and do a lot of archeology, a lot of analysis around where is change happening. How do we essentially curtail the scope of what change or complexity is? How do we curtail the scope of test? You name it.

We'd probably go and talk to some SMEs who know the subject or think they know the subject, and they'll tell us exactly what the scope is.

But we know that's flawed, right?

And therefore, what we do is we use our living specifications to essentially give us the numbers.

So you can see here where we've made change.

So you can see there CDT jumped from 25 past to 31, and you can see the ripple effect as we go down the estate.

So not only can we have an end-to-end scenario, but we can also treat them in isolation.

So we can see if there is no change down the stack.

We can isolate the testing. Essentially, we have a lot of variables that we can now use in terms of understanding the scope of change, understanding where we want to put our effort in terms of test, and everywhere else as well.

And that's really powerful in terms of understanding impact of change, understanding risk, understanding estimates.

And keeping our velocity going.

So

there's a slice of a CI pipeline.

So for us, in the context of DevOps, we're doing continuous testing and the idea being that we need to test continually.There's one such pipeline and a few of the tools that we use in that pipeline.

But the tools are not really that important. It's the methods, what we need to achieve in terms of continuous testing and how those tools enable those methods to be true that's really important.

And that's especially true when we get onto things like test execution tools, where they're more logistics than anything else.

They kind of do the same thing. It's more the interaction with the technology you're trying to test more so than the tooling that we found.

So, we measure a lot, and we capture a lot of measures as a byproduct of what we do.

And you can see there, it's about a two-week snapshot of execution of a typical pipeline.

So nobody's actually executing any tests. It's all been orchestrated through Jenkins using

SoapUI. It's fast, it's recordable. I guess the interesting point there is over that two-week period, we were in test execution for a whole nine minutes.

And

I think the interesting point for me is around, just because I can execute 24/7 doesn't mean I should. I'm creating lots of analysis for myself.

If I'm achieving the same goal with nine minutes, that's wicked. I can spend the rest of my time exploring what this thing should be doing. I have a lot of debt that I still need to address.

Why do I want to waste my time executing tests for no value?

So we get into continuous improvement, continuous feedback.

And what we've started to do, and if you're familiar with Gartner Magic Quadrants or Forrester Waves, is we've created a bit of a kind of a league table of teams with our organization that are employing model-driven development, automation, ice creams, volcanoes, kind of testing heuristics, which we know to be true, but we need to measure where these teams are, and we need to make examples of the people who are doing it really well, and we need to understand why those who aren't doing it aren't able to do it.

If you read on, in terms of the bullets, you'll see there's a little bug here in terms of Usain Bolt is not a marathon runner.

The initial slide said you need to be more like a sprinter. You don't. You need to be more like Eliud Kipchoge. He's the marathon runner that ran just under two hours for the first time.

And why not Usain Bolt?

Why you need to be this chap?

Because he does it continually.

I think the interesting stat is, although Usain Bolt ran about nine and a half seconds, he did it once.

Whereas this chap did 26 miles of 17 second 100 meters, which kind of is the continuous element that we need to be striving for. It's all about eliminating waste for us. Again, just because I can execute that many, why do I want to?

What value does each test have?

And the important thing there is about the optimization.

Because we use orthogonal arrays, because we use pairwise testing, we eliminate redundant paths within our models.

We are constantly refactoring to understand whether we've got the right coverage.

If you were in Nicole Forsgren's session, one of the big challenges around teams that dropped off is around their ability to articulate coverage.

Well, we can, and we can do it really, really quickly. It lives and breathes for us.

And if we do those things, we can spend far more time doing or freeing up our testers to be doing manual testing, which is the real value add, not just mundane instruction following. Being predictable, then you can be fast again and again and again, is the key thing there.

I'm sure we're all avid readers of the World Quality Report. Year after year, they will call out the biggest blockers for QA are data environments. You will have known from the DevOps, I run a data team and I run a virtualized environments team.

And we really started doing data seven or eight years ago, not in the name of GDPR, but we did it in the Data Protection Act.

And we went at it in the wrong way initially.

We took it as a technology challenge when it never was. It was a data challenge.

And it's a lot of the same problems that I've talked about earlier on in that we really didn't understand the data.

We carried a lot of data debt.

And if we went and talked to our testers about what the test data they needed to satisfy their test cases, they couldn't tell us.

If we went and talked to the architect to say, "Can I have a data model?" He couldn't tell us.

If we went and talked to the business designers to say, "What data triggers that business functionality?"

They couldn't tell us.

But they wanted data, and they would moan if it was too slow.

And that's a real challenge and a real problem.

And we didn't actually acknowledge that until we kind of got past early man.

So we tackled it from a technology perspective.

We went into some really heavy lifting, old systems of record, and we understood how to mask them. Brilliant from a technology perspective, really interesting, but totally useless because we knew how to mask it, nobody knew how to use it. It was still taking a long time to provision test data.

We centralized that to make it a bit cheaper because we realized how to do it, but we really didn't start cracking on until we realized that our problem wasn't necessarily a test data problem.

What we were trying to do is face into the matrix. Everybody seen the film?

So the basic premise of it is humans stuck in a computer world, and they don't acknowledge it, and a few of them that do are trying to get out.

And kind of where we're coming from, and I don't know whether you've seen this thing on Twitter in the top right-hand corner, is quite an interesting one. Classical programming, rules, data, you get answers.

The bottom one is a really interesting one for me.

And data answers, the person who posted it said machine learning is the rules. Yeah.

The big problem for me is how do you train the machine to know what they should be knowing? Yeah, you can't.

And the matrix draw for me is we're stuck in a world of technology when we work in a world of IT. Yeah.

When does the information technology come in more so than the technology?

Because we can't defer to machine learning until we can teach machines what it should be doing.

Okay.

And the bottom one is very much the hierarchy of we take data to build information, to gain knowledge. It's very much on par with what we're trying to do in terms of modeling, chipping away at design knowledge unknowns that sit in our estate. Yeah.

They're not technology unknowns for us.

They manifest themselves as technology or through technology, but they're not.

They're really information. Just a few stats. 2019, we did 3.75 billion masked or synthetic data records, 1.2 billion interface calls through stubs.

So if you take stubs, for example, if you take the picture of the very complex architecture estate, that's breaking up that estate for us. That's breaking the monolith into understanding or isolating change so that we can release quicker.

Okay.

So there's some pretty big gains that we're making, and we're making them quite quickly as well. Yeah.

And we're not doing them as tests.

We're doing as model-driven development. Our three Amigos are in this together. Cells are chipping away, and they're acknowledging the fact that ways of working previously might not have been that efficient.

Okay.

So to consolidate all of the theory that we've just gone through there, what we've got here is a continuous testing strategy on a page.

Okay.

So

I mentioned I'm from the center of competence or center of enablement.

And what you've got on the top there is the enabling factors of those.

So you've got a data team. You've got an automation best practice, guardrails, patterns, all of those things that we used to have as a center of excellence, but we didn't federate the right amount.

But we're doing that now, and this is where we've got the squad model or the cell models where they are actually going to put these things into place.

Okay.

So we've got the right testing strategy. Ice creams, volcanoes.

We are test engineering.

We're modeling the hell out of this thing. Yeah.

We're creating living specifications.

Okay. It's a bit old. It says we're doing TDD and BDD.

We're not.

We're doing model-driven development. Apologies for that one. Once we've got predictability in what we're doing, we've designed our tests really well, then we're into the execution phase. Yeah.

We check.

We automate the hell out of what we're checking, the stuff we know isn't changing. Yeah. That's the stuff that's really fast. That's our CI/CD pipeline.

We're winning on that one.

And we apply structured testing, or exploratory testing I should say, to the stuff we don't know. Yeah.

And that's not just functional testing. That gets into the non-functional world as well. Yeah.

And again, there's a lot of technical debt around non-functionals that can only be unearthed by understanding what the actual requirement was in the first place. It's a conversation, and then we bank that.

We create models for it. Yeah.

We do those things, and we want to create a lot of metric. Yeah.

We want to learn where we're still making mistakes, where we're still inefficient, and we capture a lot of those things through a central repository and a lot of dashboards.

What we don't do yet is exploit AI. Yeah. Partly because we can't find the technology to do it.

We know we've got a lot of structured data that we believe in. Yeah.

Because everything that we've talked about has built the trust in what we're doing.

We can't find a source of information or actually a technology where we can exploit that to give us new insight that we don't already know.

And that really brings us on to testing volcanoes 2.0 for us, which is where we've got our engineering element.

We're doing the right structured testing at the right level of topology.

We're being fast enough.

We're automating. Our levels of automation are through the roof. Stakeholders are happy.

We've got that buzzword going. CI pipeline-wise, testing is not a blocker. Yeah.

What we need to be doing is how do you exploit AI to accelerate the addressing of technical debt? Or making sure that inadvertently we don't build it up somewhere else.

And that's really where we're into the futures in terms of what we're looking at.

So if anybody's got any ideas, brilliant to talk to you.

Okay.

So the last point is, for us anyway, don't just shift left. Learn. Go right, go left, go up, go down, go in the middle.

But one change at a time. Yeah.

If you have an estate like us, no one is going to pay to address everything at once. That's stupid. Chip away at it when change initiative comes along. Eventually, you'll get momentum. Eventually, reuse comes along, and what seemed so hard the first time isn't quite so hard the second time.

But you've got to bank that learning, and you've got to continually raise the bar in terms of your understanding of quality. Yeah.

So that's the end of my talk.

Broadcom Announcement

Christine from Broadcom wanted to make a quick announcement. Yes.

So we have cookies in the back.

We are launching two exciting things at the Broadcom booth, number 902.

So please stop by. Please grab a cupcake.

And thank you for coming, and

I think we have time to answer some questions.

If there's any questions, happy... Yeah.

So thank you, guys.

Q&A

Any questions? Cool. Thank you. Oh, go ahead. How did you guys first start-

So we started in the wrong order. Yeah.

So we were trying to do automation for ages, and it kept going wrong.

We were trying to do far too much UI-based automation using some older tools.

And we had a little island of automation testers.

We would say to the test team, "Chuck over your test pack."

We'd automate that.

We'd give it back to them.

What do you know?

They'd moved on.

So we knew that didn't work.

We started to understand why that didn't work, and it really came down to predictability of...

Well, we were set up wrong in terms of automation, but the fundamental problem for us is really what we were trying to test wasn't actually articulated.

There were no expected results. It was kind of a bit human in terms of I'll run a few instructions, and if it comes out with this report, I'll go and check the answer.

And you can't automate that thing. It's not predictable, and it's not repeatable.

And that's the real big learning point for us, which is why we've really gone for modeling.

In your volcano pyramid- Yep. How do you decide the ratio in coverage between layers? How do you came out with that number?

The ratio? Yes, the coverage.

There is no number necessarily.

We have certain teams that work at certain levels of the stack.

Okay?

And if you work at the full topology, we will look for a ratio that is artificial, but you should have less UI than you have API. Essentially, it's going to be through collaboration, yeah.

And part of it is, again, unearthing your technical debt in terms of we have an interesting conversation, or we had an interesting conversation about exposing SOA to the testers so we could create SOA-based tests.

And the fact that we didn't have designs for it meant that we couldn't do it.

And therefore, we raised the conversation, we raised the debate to say, "Actually, if we want to implement a lot of these practices, we've got to address a bit of this debt before we can do it. Otherwise, we're a bit screwed."

Another.

Has your skip rate increased or decreased after this? Oh, what?

Sorry.

The skip rate defects.

So the interesting point for us in terms of modeling, we found very quickly that our defect prevention in terms of conversations or eliminating defects in design increased by about 30, 40% quite quickly. Yeah, so we were quite infantile in terms of doing modeling, and we were still reducing a lot of defects at the design level by 30, 40%.

Okay?

From an automation perspective,

I think the interesting thing from an automated execution point of view is we are far quicker at finding configuration issues and exposing configuration inadequacies that we've got in certain teams.

Okay. Is that all right? All right. Cool. Thank you very much, everyone.