Kevin Foley

This is Admiral's DevSecOps journey, and I will go through the journey of what we've done for Admiral in the last couple of years, and where our DevSecOps journey started.

I'll talk a little bit about myself first to give you an idea of what I've done. In 2003, I joined Admiral. I was already a developer before then; I'd been to college. I started with Admiral as an operator, an AS/400 operator.

After a year or so, I was already trying to move to be a developer, because that's what I did when I was in college. I then moved to be a developer, an RPG developer, and worked on the development of all of our systems.

In 2006, I retrained myself to do Java so I could do more front-end development on our websites.

I then moved to be a team manager in 2010. It was a progression: I'd already become a senior developer, and the next progressive move for me was to move into team manager, looking after a couple of development teams.

In 2017, I then became a delivery ops manager, and now I look after numerous tribes and deliver lots of bits of software across the board.

Admiral is a car insurance company, mainly. It started off as a car insurance company. David and Henry, who are in the picture, started it in January 1993. We are based in Cardiff. That's where the main head office is.

When we first started, there was only telephone, TV, and the Yellow Pages. I'm not sure how many of you are old enough out there to know what the Yellow Pages are, but they were about before. We started off, and our growth was pretty small as we started to move into the market.

In 2000, we started looking at the internet, and we're quite proud to say that we were one of the first quote engines for car insurance. The Elephant brand, which you can see in the picture, was based on an internet-only brand.

We've grown since then. When we moved to the internet, we started looking at price comparisons, and 90% of our sales now come from price comparison sites.

We've moved from car into multi-car. In 2005, we were the first to introduce multi-car, and it was true multi-car: the more cars you put on, the cheaper the policy you got, and we'd cover it all under one policy. I was lucky enough to be one of the developers to help build that, which was really exciting.

In 2012, we diversified a bit into household, so we've now got a household product. I was a team manager at that point, and my team were lucky enough to basically build household. I'm pretty lucky that the two strategic parts of Admiral, of how we've grown, I've been a part of, which is quite exciting.

We're a little bit bigger than just Cardiff, and we've got lots of other companies that we're a part of. I won't go through them all, but we've basically more or less taken over the world.

In 2013, we were looking at time for change. The reason was that we were on an old AS/400 system with green screen, and we wanted to move to a more friendly user experience with the software, especially for our call center departments to work on. The age of digital was coming along.

We embarked on a journey which took numerous years to do, but we moved from our legacy system to a software called Guidewire.

One of the things that we're looking at doing right now is moving to digital: self-service and more focus on mobile. That's very key in our strategic plans.

Analytical data is super important, the same as any other company. The more you understand about your customers and what you're doing, the better we can help price and make sure that we do things right.

One of the other things that's happening is we've moved to the online account. If you're lucky enough to be with Admiral or any of our brands, whether Diamond, Elephant, or Bell, you can go onto our website and manage your own account online. This has become far more important in the last three or four months since COVID-19 has come in and everybody's working from home, which makes it a lot harder. We've now pushed a lot more to our online account to allow customers to self-service.

Once we'd done that, about two and a bit years ago, maybe two and a half years ago, we started looking at the security side of what we do. We'd moved from one aspect to another, and it becomes far more complex as you go through.

We've got our own security team. We had a small one in IT originally, but it's moved out and now we've got a huge security team. We work with them about delivering a secure software development lifecycle.

One of the things that we did to help us understand what it is to be more secure in our development is training. Ryan Ackroyd came in as a consultant and helped train us. Everybody in IT basically went through the course. We did the basic secure development introduction so everybody knew and understood what we were trying to do, including our BAs, team managers, and product owners.

Everybody who actually cut any code did secure development for developers, so they understood what it was to make sure our code is secure. Some of our seniors then went on to do the API advanced secure development. We've got different levels of security through the thing.

One thing I would say about Ryan: if anybody ever wants to have a conversation with somebody who can tell you how easy it is, or not really easy but depending which way you look at it, for somebody to hack your system. We've got big firewalls in front of us, and he said he would never try to attack one of those. That's interesting because we pay a lot of money and you've got these big burly bouncers sitting on the door stopping bad people coming in.

Yet he said he would go off and find who our printer company was, maybe get into them, sit in their software, and then we would bring him in through the door like a Trojan horse. That really opened my eyes to how a hacker thinks, and what we think is secure and what they think is secure are definitely two different things.

One of the things that started me on a slightly different journey around development code was the DevOps Enterprise Forum two years ago, the first time I'd ever been to one of these conferences, this big anyway. I loved it. The amount of people that were there, the enthusiasm, and the amount of information you could get was phenomenal.

I ended up going to a presentation by Nick Wade around open source. We use open source, as most companies do, and I went along to have a look. He's talking about his Raspberry Pi and how he's downloading it, and I'm thinking, "Yeah, okay, that's fine." Then he starts talking about open source and what it's doing, and I'm thinking, "Hmm, that's interesting. We do some of that."

The more he was talking about it, the more I was getting worried. I thought, "We definitely do none of that." I'm pretty sure we had no idea what we'd downloaded. We had no idea where the code had come from, whether somebody had put anything in it, whether it was malicious or not. We just openly believed that it was okay. I came out of there feeling a little bit queasy, to say the least.

I came back and started having a chat to our developers about what we'd done and what we were doing: how our pipeline works and what we've got in place.

We were downloading open source. It was strongly locked down by a security team, so whenever we wanted to do anything, they basically said whether we were allowed to or not, which is great. We thought we were relatively covered. They were doing their job to make sure it was a secure site where we were getting it from, and doing our due diligence.

Eighty percent of our downloads were done outside Sonatype. I don't know if that was a good thing or a bad thing at the time, because from a security point of view we thought we were okay.

That wasn't really true. We got in touch with Sonatype, who came in and did a scan for us. The scan was just to say, "What have you downloaded in the last three months?" They gave us a list of everything we'd downloaded over the last three months.

I was sitting there waiting, thinking this was going to be horrendous reading, I was going to be really scared, and I was going to have to do a ton of work to fix it. It probably was more or less true. It wasn't as bad as I thought it was going to be. We had a couple that we needed to fix more or less straight away, but that was about it. The rest were fixable, and we had mitigations in place. Still, we were downloading some dodgy code, to say the least.

Then we ended up implementing it. The interesting bit is that we've got gateways: how we build our application, our secure gates. We've got SonarQube to look at our code analysis. Static analysis tells you whether you've got security vulnerabilities in it, but it doesn't touch what we bring down from open source.

Anybody that downloads open source knows that you download one piece, it goes off to another piece, it goes off to another piece, and you have no idea what you downloaded. If you do, you're going to spend hours and hours, or days and weeks, trying to find out.

We bought the Nexus IQ scan, which we put in there and have now implemented. It's taken the best part of a year to implement it. It's been really difficult.

Why? It should be really simple to implement these things. The security team needs to get involved. But the same as anything else, you've got teams that are already functioning, then all of a sudden you're trying to implement something that can stop them building quicker. So it took a while.

We also had issues around building it. When we originally plumbed it in, this is what we had: 320 threats. We thought, "Oh my God, that surely can't be true," because we'd already done the scan up front. We already knew this was just making sure that it goes through the pipeline safely, in case it goes off and does anything weird.

We looked at our scan and said, "Actually, our scan says this." It was on our scan, but because we weren't actually transferring this information through our pipeline and moving it anywhere, it still picked it up because it assumed it was there. We had to do a bit of a workaround. We had to look at some community practices and build in some Nexus tooling to see if we could plumb it in.

What we get now is much nicer looking. The problem we have is that you would look at this as 320 we've still downloaded. We've still got work to do around not allowing that code to come down, how do we do it, what threat it has to us as a company, and what threat it has before we move it through our environment.

Right now, we've changed our pipeline a little bit. A lot of things are automated. Most of our software now, around 90%, comes down through the Nexus repositories. We know it's all being scanned and we know what we bring in. There are a couple of outliers we still need to bring in, but nothing that gives us a major threat. Right now, we're in a good place.

We've got SonarQube that helps. We've got Terraform that builds all of our hardware. We've got Ansible that builds all of our scripts. We've got what we call Yeti, which is our internally built application that allows us to monitor and make sure that all the system is working and is all together.

One thing I haven't talked about is security testing. We've got a dedicated security team that lives in security. They are called a red team. They do all of our scans to make sure everything is okay. But they do that scan at the point when we are going live or the week before, which is too late.

At that point, if they find a massive vulnerability that we've missed and haven't picked up through anything, whether it's coded ourselves or not, we'll have to go back. Then it delays things, and you've got loads of other problems from the product owner and everything else.

We are working with the guys to figure out how we move a bit more left, so they don't become so much of a pain at the end, a nice pain, not a bad pain. Whatever they find, if we say we can't fix it now because whatever we've got in our release is important and needs to go out, then we create technical debt, and then we have to go back and do it, which costs more time and effort.

One of the things we found out is that we've got three testers who are teaching themselves to be security testers, which is great. They were sitting in the corner, and we found out about it about a year and a half ago. We're now trying to embed them into the pipeline to get security testing done at the point when we build it. We know then if there's a vulnerability in it, can they do it? Do we need to get kicked out and go, "No, actually, you can't do that"?

We're working with them, and we're working with the red team, because they need to make sure we sign this off. That's work in progress. As anything else, it's really difficult. The longer you have something that sits over here that belongs over here, it becomes a problem. You have the handoff, who owns it, what does it mean, they found it, I didn't: the usual that we normally get into.

One of the other things we've started to add to our pipeline is how we build our environments. I've talked about software and the delivery of software; now it's the hardware side. We scripted everything to make it easy, continuous, and quick. We've spent years on it and we are at a really good place.

But it's not hardened, so we need to harden it now. We're working again with the red team and the security team, who are very much on board with making sure that we can have our hardware already hardened before it goes anywhere.

If we want to build a new server or a new environment, it's already hardened. We're looking to build that into a pipeline to make sure that when we build a new environment and give it to somebody, it's already hardened and we don't have to worry about it not having the right patch or having a vulnerability because we haven't done anything with that patch. Those things are quite important for us. My DevOps engineer teams are doing that and working with the red team.

One of the things I think is important is that's just my journey, or our journey as Admiral Insurance. Would I change things? I think I would. We're a big company. Most big companies are probably doing exactly the same thing, and if you're a smaller company, you could probably do things slightly differently.

The separation of teams is important. As a DevOps, we talk about DevOps and DevSecOps. If you separate the team, you become a handoff. It becomes somebody else's problem, or they don't believe in what you've done, and then you have that fight. If I could go back and do something, I would make sure it's built in there first. Make sure it's already built in and you're already thinking about it going forward.

Creating a secure development lifecycle is the right thing to do. Everybody should have one. The problem is that it was driven from our security team, not driven from us. That's not a bad thing, but it takes a while for the team to take ownership of it, understand what it means, be accountable for it, own it, and deliver with it. That's still ongoing. We definitely follow it and use it, but you need to get more buy-in from the tribe tree.

From a security testing point of view, I think we've got a security testing team that looks after it and will work with us in our teams. Ideally, they should have full sign-off and say, "This is vulnerable" or not.

Even right now, if I'm honest, we have that team sitting in there, they hand a Jira off, and that goes off to the testing team, development team, or tribe. Do they take ownership of it? No, not really. It just sits in the backlog. Then they hand off to the red team, and the red team do something similar. Therefore we've got these Jiras sitting about saying, "You've got this vulnerability; somebody needs to fix it."

That's not a good thing because it becomes a management problem, and it shouldn't be a management problem. It should be the tribe and squad's problem to make sure that they are secure and delivering it. Moving that into the team and making ownership of it is super important.

The responsibility to me is: if a developer, tester, tribe, or squad creates some code, it's their problem. They've developed it, built it, and owned it. They should take 100% full responsibility for it. As soon as you allow that to go somewhere else, it becomes the problem.

If you could change things, it's harder for us. We're already settled, massive, and doing things slightly differently, so it's harder to reinvent the wheel and move things in. It becomes challenging. People get stuck in their ways. It's not that they don't want to move, because they all do and they all understand it, but actually getting it working is really tough.

That's all I've got. You're more than welcome to ask any questions. We're on the Slack channel. There's my LinkedIn if anybody would like to contact me. I hope that was beneficial to anybody, and I'll see you soon. Thank you.