Fliss Bennée

Hello, I'm Fliss Bennée, and for the last two years I've been advising the Welsh Government on science for COVID, attending SAGE, and generally trying to do six impossible things before breakfast.

Now, when Ben and I did this before, we were in front of a live audience, and it was much easier to say that, as you can see, I looked a lot different pre-COVID. Trust me, I'm larger now than I was then.

Anyway, I've been in and out of data and digital as a civil servant. My career working on open standards on complex data. I've been helping the government put user needs first. All the stuff that's quite foundation level, but it takes time for a big organization to learn. And frankly, it's still going to take thousands of us working together constantly in a lot of different places to make lasting changes to something as large as a government.

But in late 2019, I took a role in the Welsh Government. I was just starting to help really shake up the NHS Digital agenda in Wales when suddenly COVID came over the horizon and boom. But in fact, the reason I met Ben in the first place was that Armakuni were working with the Welsh NHS too.

Benedict Dodd

Hi, everyone. I'm Ben Dodd. I'm a co-founder of Armakuni. I spend my time helping the Flisses of this world, people that want to deliver at pace, build out an engineering and data future. Day-to-day, I work with governments, with global energy and financial services organizations.

And what I'm most proud of is over the 10 years, Armakuni have built out this amazing team of 60 engineers and consultants, and it's really this crack team of engineering first responders that I'm representing in this story. And that is the foundation for this story. Great people doing an amazing thing in an emergency.

For my part, for my time working as a designer and a developer, and a change agent in these really large, complex organizations, I brought my ever-growing list of ways not to create a light bulb, and to be fair, a pretty good list of ways that you can. I also spent a lot of the pandemic developing a worrying photographic equipment habit, and that's why there's an overly dramatic self-portrait of me on the conference website and this photo. And if you look very carefully, you'll notice I'm trying a misguided attempt at a sort of tweed Steve Jobs, wearing the same outfit in every opportunity.

Fliss Bennée

Good man.

But we're going to tell you a little bit about our story, which is how a very small group of people in a small country in the middle of an emergency was able to share our data our way and make a difference.

And I'm not going to bore you with the maths. I'm not going to talk about the R number, which all of you are sick of by now. But I think there are some things that can be applied to your own situations.

And firstly, one of the reasons is I say it's a small country, but we shouldn't forget that it is literally the size of Wales, which I think of as the unofficial scientific unit for area. We often talk about pieces of the Amazon, such as this, area the size of Wales, when we talk about deforestation. My own personal favorite iceberg, Iceberg A68, was described as a quarter of the size of... Everyone should have a favorite iceberg. It's handy as a comparator because it helps people put things into perspective.

This is a quarter of Wales, which technically means Wales is four icebergs large, if we flip it around. But Wales has 5% of the population of the UK, but it is just less than a tenth of the size. We've got 20,770 square kilometers, which, as far as the internet knows, is 14 million rugby pitches. It's my preferred shape of ball. I don't know about you, but it just works.

So we don't have the size, we don't have the economic power of bigger countries. We have to innovate a lot. And it also means we put a great deal more stock in the strength of our relationships.

What does that mean? Well, when COVID came, we had to start giving science advice, and I wanted to make sure we could be open and transparent right from the beginning to explain as much as possible to as many people as possible. So at the beginning, right from the start, we asked ministers for permission to share our independent scientific advice and evidence with the public. At the heart of it all, that seems to have resonated with people.

These pages are still live. You can go to them and find our work. In Wales, we've talked about the good and the bad things that are happening without fear or favor. We're just trying to help people understand what it means for them. And I hope that's part of the reason why trust in Welsh Government decisions and in our science advice seems to have remained relatively high over the last two years.

At the beginning, our data was a total mess, though. It was mostly all there somewhere out in the system. It was going from A to B, but we'd never tested how long it would take to get access to the datasets. We didn't have any agreements in place to cover emergencies, and so we couldn't get what we needed in order to see what was happening.

We knew we needed to see how fast the infection was reproducing. So from the first day, I had my very sole team member literally phoning around every one of the intensive care units in Wales every day just to ask for the count of people in there who might have COVID, so we could work out how fast the exponential growth was.

My minimum viable product was a spreadsheet and a phone, and I knew that this information was lagging three weeks behind the end of the pandemic. There was no mass testing yet. So we set out to identify what information there might be that would get us closer to the leading edge of the infection so we could actually make a difference.

It was pretty chaotic, and the only thing that held back that panic was the offers of help that came in. There was plenty of snake oil on offer, but groups like Google and Facebook opening up their anonymous travel information, King's College with the ZOE symptom tracker, Armakuni, they were all early volunteers. Public Health Wales was looking for outbreaks and sharing their information. We were reiterating our operational and our academic models daily. I spent every day going from meeting to meeting on the phone, then on video, briefing people with slides that I knew were going to be going out of date literally as I was explaining them. And I had no spare people and no space to think.

When I had to spend all of my time interpreting data with ministers and NHS boards, first responders, I couldn't do the analysis in the building on my own.

Benedict Dodd

So from my perspective, and from Armakuni's, being given this challenge was a great call to arms. The pandemic was an extreme version of a challenge that all organizations face eventually, and it's a challenge that my team have been facing again and again over the last 10 years.

And Welsh Government, like everyone, had to respond to a fundamental change in the customer need and the environment. They had to do this at pace, but importantly, they had to build that trust in the information and the digital solutions they're providing.

But this time, it wasn't going to be a two-year transformation. It more or less had to happen overnight.

So any effective response needed all the things that us, as a community, have been talking about for years. We needed a really rapid discovery. We needed a true cross-functional team. We needed to probe into this really complex and chaotic environment, and the ability to deliver this change to customers multiple times a day, safely and securely.

Fliss Bennée

I think it helped that we didn't actually have time to do things in what my organization used to call the usual way. There wasn't any time for a long tender, but I still had to safeguard public money. So I took a contract through G-Cloud using an organization that already had the clearances sorted and the security certificates in place.

And because we were in an emergency, we didn't have time to shoot ourselves in the foot trying to spec up our needs. Instead, we just got to leap right into a real partnership. It was kind of me introducing Armakuni to our growing team of volunteers and scientists and saying, "Talk about what we need. Make sure you're open. Try to do things in this new way and see if it helps."

And that prepared my team for a fairly intense period where agility wasn't just nice to have, it was actually a necessity. So we got to be more of one team than two, I think, in the end, trusting each other's specialisms, giving Armakuni full rein to help us turn our science and data into advice against an ever-changing background of needs.

Benedict Dodd

And that's what we formed. This was really the best cross-functional team that I've ever seen. And it's also what we were seeing across the pandemic. We were seeing people come together to create ventilators in days, weeks, where previously it would've been months and years.

And in terms of the skills that we brought, we brought the data scientists, we brought the data engineers, people who could build that prototype in hours. And Welsh Government brought the healthcare, the population, the epidemiological experts...

Fliss Bennée

Well done.

Benedict Dodd

... all working at that national scale. Thanks for that.

And we all sat side by side. It wasn't us and them, it was us working together and working out how to work creatively together overnight.

We also brought that technical communication and collaboration expertise. There was none of this headphones, heads down, writing code. It was all about understanding what we had to do, sharing that understanding and that expertise.

We also brought the ways of working. The team were fanatical about wanting to know the people we're solving this problem for. They all got names, pets, quirky habits, anything that we could do so we could have them in our minds as we were creating the end solution. We also love extreme programming, so we love to work in pairs. We love to work with other experts, and this was all great fun.

So actually, we had all the things that we needed to be successful, the mandate to move really fast, and a really high-value, nearly impossible outcome to achieve.

Fliss Bennée

I've just been looking a bit more closely at this picture, and I really hope that in the pets and characters, that I was the grizzled policeman with the top hat. Because I don't think there are enough police with top hats in film.

Benedict Dodd

So really the first rule of building a data dashboard is do not build a digital data dashboard.

We started out with this Excel version, which was actually built by the Welsh head of radiology, who was the person that just happened to be walking past Fliss's office door at the time. So we built this really initial, relatively horrible-looking prototype, and it's really took me a long time to learn that it's important that they do look horrible, and this messy first step is a really great signal to people that their time to contribute and to give feedback hasn't yet passed.

This progressed into using things like Jupyter Notebooks running in the cloud, and this allowed us as a technical team to collaborate with the data experts and get feedback from our users in hours in those early stages.

And getting feedback was a real balancing act. When working for the ministers, obviously they are a very key consumer of what we were going to create, but there was also a broad community of other users. And even in the real heat of the battle, we still had time to do focus groups and really focus on what the customer need was.

So this is what the actual dashboard looks like. The intent was it to be at a glance. You could see, just looking at it very quickly, what the state of the data was, but you also had that ability to click on any of the components and really drill down into the data. So that allowed for a broad range of users, some who just wanted to know at a glance what was going on, but some who had a deeper understanding of the knowledge and could consume more of that.

We also were very customer-centric and user-centric when we picked our tools. We need to be safe and secure, but we also needed the ability to onboard people and a range of organizations really quickly. So that's going from the ministers, but also to, at the time, things like the armed forces who are being drafted in to help. So we needed a tool where people could get access within hours and minutes.

And that was a really big bet for us, that we could have used the existing systems that were in place. But people just didn't have licenses, or they didn't have access, or the onboarding was going to take two, three, four weeks. So focusing on that customer need drove us to the tools that would allow us to fulfill what they actually needed.

We also automated everything. So in terms of the data, the delivery, the platform, the security, the graphs, everything was automated. A, so we could deliver it extremely quickly, change to users, but also that people could start to build that trust and confidence in the data and the system itself.

And there was a really broad range of formats that we were having to get our data from, and really, that was our mission. It wasn't a time for elegant APIs from everywhere. It was about getting the data together really quickly, reading from APIs, scraping websites, using AI to parse PDFs. It was not pretty at times.

But that was the task, to get all this data in a renewable way. And the real focus for us was the telemetry to understand if the data had changed underneath us or something like that. If there was a reason to suspect the data was no longer accurate, the investment in that telemetry was more important than a really beautiful, elegant set of data for us.

Fliss Bennée

Yeah. I interrupted you here and said, "Ooh, show them the sperm graph." But just to let everybody know, that graph at the bottom, I particularly love. I've been calling it the sperm graph for a while because it shows you that there's a set of dots that you can see. They're all close together right now. But there's also dashed lines, and what happens is the end of the dashed line is where the data were exactly a week ago. So I can see whether they're all swimming together.

I'm sorry, I've interrupted you again. I apologize. Carry on, please.

Benedict Dodd

That interruption was much better on stage as you were sort of leaping up and down, pointing at a giant 40-foot screen. But, so yeah, the tool was open source. It was using technologies like Python. We deployed it to Google Cloud. And it covered a huge array of data, data from the entire Welsh landscape. So from mobility to Google's travel data, to genomics, to testing data, all the way to sampling data from the Welsh sewage systems.

So it was a really great test of our humanity that even with a room of relatively serious people, the time it would take for us to be giggling about poo was always very short, even though it was a really great leading indicator of localized outbreaks.

And one other thing is that we needed a platform that could keep up to date with the public branding of these interventions. So over the course of the pandemic, how people were talking about the situation, what was going to trip us over into another state of the reaction to the pandemic, all of those sorts of things. We were part of that marketing and understanding that allowed people a good understanding of the current state, and what was going to happen next.

So the layout, the colors, the graphs, all changed over time as we searched for the best way to articulate what was a very chaotic situation. And we tried to flush away all of the puns, but we haven't quite got there yet.

Fliss Bennée

I would say we've washed our hands of it.

Benedict Dodd

Do feel free to try and spot any more poo jokes coming through the rest of the talk.

Fliss Bennée

Obviously, we went from static to real-time data, and we were starting to have time to actually go out to the public and explain things. And that was when I got my first big reminder that I'd moved too far away from what my user needs were and into my own space of happiness.

My comeuppance came at a technical briefing for the media. We did an off-the-record session where we could take people really behind the science curtain to see the wizard. And there's me with my whiteboard showing why the R number is really one at the top of the curve and at the bottom because of the ratio aspect. We had a behavioral science colleague talking about protective behaviors.

Someone said, "What's the point of saying masks could help if there aren't any to buy or if we can't afford them?" And I kind of went, "Aha. I can't sew, and I've wondered exactly the same thing, and I've been looking..." France has managed to get this great standard, and they've gone through all of this many nanometers at this much.

And I was looking at it and going, "Well, you could get a good five-to-10-nanometer filter mask that fits you well if you've got a good cotton sports sock that's close weave and a piece of kitchen paper or something non-woven but chaotic and a pair of scissors." And I showed them. I brought it in to show my colleagues, so I showed the media.

And later that day, the press team said that I'd been asked to do ITV and BBC Evening News, and I was so excited. I kind of went, "Should I bring the whiteboard? Can I get any of the good graphs?" And they said, "No. But can you bring the sock?"

And I just kind of went, "Oh God, it's Blue Peter all over again." So my very first live TV outing as a science advisor, 6:00 PM and 6:30 on national news, was doing the arts and crafts corner.

But what I've realized is it was exactly what my users needed because they needed a science advisor showing that there were things that everyone could do to make themselves safer when people were scared.

And having sorted that out, the next thing our users wanted was a solid and reasonable worst case specific for Wales. So I turned around to Armakuni and said, "Nice work on the dashboard. Now, can you spin up a virtual supercomputer to crunch some new models, please?"

Benedict Dodd

So yeah, having worked night and day just to really try to get a handle on what the present was, all of a sudden we were asked to try and show the future. And personally for me, this was a real education. Seeing graphs that were plotting the end of humanity was an interesting thing to suddenly have to come to terms with.

Fliss Bennée

I'm sorry about that. I'm still sorry about that.

Benedict Dodd

We don't really have time in this talk to go into the depth of how we did that and all of those sorts of things, but a few things are, one, this was a parallel set of probes. So the global community was trying to work out what the best model was for this. So we were essentially running one of these parallel probes into, is this model the right model?

And other universities and other institutions across Wales were looking at a different one. So this is exactly what you want in this complex situation, a set of parallel probes to try and work out which is going to be the most valuable and most impactful.

We also took the same approach as we'd done for the rest of the project, i.e., everything was as code. We built a massive Kubernetes to run it at scale. Something that would scale from three nodes that it was, up to 100 and 150 within 15 minutes, and back down again.

So this meant having this really elastic resource to run that, that we could easily recalibrate the model relatively often for a very small amount of money. And we also did the same investment in the UI, so it wasn't just one single report that would be printed out to a PDF. It was also that interactive element, so people could start to deep dive into the data, build up graphs and views as they tried to understand what the model was showing them.

And also because we maintained all those principles and practices around continuous delivery and everything as code, it meant that, when we'd gone through that initial phase, we could really easily hand it over to Swansea University, who then took it on and run it in the future.

Fliss Bennée

I love containers. They make everything so easy. Just keep connecting the nodes on the virtual machine. It'll all be fine.

Benedict Dodd

Exactly.

Fliss Bennée

Were you disappointed that as a customer, I never asked you to put anything on the blockchain?

Benedict Dodd

I thought it would be the next thing. It felt always like it was going to be inevitable. We always had something ready to go.

Fliss Bennée

Yeah. Let's make Dogecoin out of it. I don't know.

Benedict Dodd

Yeah. So the other thing around this is that we formed an instant community. And this transcended nations and commercial boundaries, so very quickly working with colleagues in other institutions in England and elsewhere, we could start to share, and we were sharing code, we were sharing insight, the configuration of the model, all of those sorts of things were just instantly shared. We had face-to-face meetings that were suddenly available the next day.

So this is exactly what you want as a community, that real practitioners working on the ground could come together extremely quickly, and it was important that it was a direct connection. It wasn't a set of asks that went somewhere, and then they would get them, and they would respond, and it would take hours. It was proxy-free, really quick response, which was fantastic to see.

So for the good bit, what went wrong? Fliss, what would you say we could have done better next time?

Fliss Bennée

So, and you talked about face-to-face. Obviously, we were face-to-face virtually, which made it really hard for me the first time I met you in the flesh to identify the fact you are actually considerably taller than me. I always sort of loom quite close, so I look very large, and you always stand back, which makes you look rather smaller, and it was very disappointing to discover that I could barely reach the top of your head on my tiptoes.

But look, we integrated. We didn't integrate fast enough. In fairness, I was busy and thinking about other things when we met, but really I would've integrated us much faster. I would've asked you guys to come and sit on as observers on some of our primary considerations so that you could really do more user research and understand what we were trying to get at. Because I think it probably would've got us even more benefit than we did, onboarding you so slowly as it were.

What about you?

Benedict Dodd

Well, yeah. We've only got a couple of minutes left, so I'll talk about these relatively briefly.

So one was really to take the product that we developed on the road to give more guided tours to more people in the Welsh landscape so we could get, A, their feedback, but also that we could remove any duplication. That we had a product that people could use, and it could stop them having to create their own, freeing up their time to do some more stuff.

I think the other one that really springs to mind is that we could have used, what I would say is Wizard of Oz-ing more. That we did that initial sort of ugly versions of the dashboard. But then once we started to automate and created that digital version, when we came to thinking about new features, we didn't go back to that more basic way of getting really quick feedback, and I think we probably spent more time delivering features that we could've realized their value if we'd have done more of that sort of prototyping later on.

Fliss Bennée

Once you get it easy, you keep going to it and that thing.

Anyway, look, it really is what made it work was our willingness to co-produce, on both sides, and the trust we had in each other from the start. They were exceptional times, but I really don't think we were ever supplier and customer. We felt like partners all the way.

And I did admit, in front of your finance officer, that I've been a terrible customer. I know I am. It took me longer to navigate the intricate finances of the public sector than it did to navigate the intricacies of COVID, which is saying something.

But you guys have been so amazing. You folk have been so amazing at keeping the service going while I sorted out the back end because it was important, and because telling a story behind these data was literally helping to save lives. And I feel that was part of why it worked so well.

This is one of my favorite graphs, this sort of rather strange cubey-looking one. Actually has within it some quite significant story. So you can see that the color changes from light blue, goes through white, and then into orange and red.

Actually, if you look around September 2021, and into October, sort of middle of the graph, when you look along the second row from the bottom, the 10-to-19-year-olds, you can see that that was infection coming in through schools. But actually, it really just wasn't getting up into those older age groups. And that really was our first indication that we had a good barrier from our first two doses of vaccine, which was amazing.

And if you look in January, you see the spike, how fast the Omicron virus went straight up through everyone. And then it sort of hits that tail of people for whom there isn't sort of the strength. Perhaps these are people who haven't yet had their booster dose, although by then I suspect most people have. So this is sort of the trickle-through of this huge quantity of infections in the younger age group, but it's really amazing.

It's such a paradigm shift the way we work with data visualizations in science now. We've seen real-world evidence of how much information into action comes when you get a lot of different specialists together just sharing and testing things.

And look, we've worked with analysts, we've worked with visualizers, with the data we had, and we keep telling stories about the data and iterating. And the funny thing is, actually visualizing and telling stories and iterating isn't a new idea. We know that fast, multidiscipline groups sharing things is a really good way to get things done.

But the difference is, even in the public sector now, in the cabinet, our senior decision-makers have seen how good the stuff we can do is when we're allowed free rein to really communicate and collaborate across those silos.

So what we've really learned, I guess, is that you can be evidence-led and data-enabled in an emergency at speed. So there's no excuse when there isn't an emergency or when you've got time. We've saved lives with maths in the driving seat.

We've blended an agile, knowledge-based economy of science with the more process-based economy of public service, getting results fast, moving our specialists up the value chain so they can concentrate on higher order problems. And we really like working like this.

So my question to all of you watching this is which one of you is going to be next? If we can put our trust in external partners in the public sector, not blindly, not trying to outsource our accountability, but in open partnership, in the spirit of curiosity, then what other complex enterprises, what other large organizations can make that change?

We've learned that uncertainty is not something to be feared. It's something to be explored. Exploring uncertainty together brings us to understand ourselves and our organizations better, and being open about what we do and how we do it with all of our users and our partners, our consumers, helps us to travel the same road to certainty. It builds trust.

Trust lasts longer than contracts, and it lets us do amazing things.

Thank you.

Benedict Dodd

Thank you, everyone. Thanks, Fliss.