Lee Sexton

First of all, I'll give you the reason why the title is "Primeval Environment."

Typical challenges of a business, what many of you guys will have out there: I'm in a meeting, 15, 20 people, all talking about the next great thing we're going to deliver and how it'll be amazing if we could stand up a machine in a day. "Wow, this project we could get delivered in three months."

And someone turned around to me and said, "Lee, how long does it take you to stand up a new machine?"

I went, "Five seconds."

They threw the project plan on the table and said, "How long would it take to do the project?"

I said, "Three weeks."

They didn't believe me. I proved them wrong.

And that's kind of where we are with businesses. It's not about what we do in tech. I'm not sure where you're all coming from and what you're looking for today. I know I spoke to Dominica earlier, a second ago. Have I got the name right there? A lot of people come to these conferences and they tell you about their journeys. And they're great journeys and they're really well worth listening to, but they're kind of similar. And what I wanted to do today was open this up a little bit to more about the challenges of doing this.

So my favorite topic is, you walk into a conference or a meetup and the first thing you see is...

Oops, I've missed out the company, haven't I? Shh. Let me tell you about Thomas Cook first.

It's 175 years old, so when I say primeval, it really is. They've got 20 million customers. They do nearly £8 billion worth of revenue, 22,000 employees, and they're spread across 15 source markets, so that's all over the Nordics and Europe. It's well spread out across Europe as a whole. Basically, in its market areas, it's number one or number two for revenue.

They want me to tell you it's listed on the stock exchange. I'm not sure of the price now. It's 175 years old, as we say. We're celebrating that this year. We've even got a two-hour party next week for it. So it's all exciting. It's great.

Now back to the interesting stuff.

So we all go to meetups, we all go to conferences, and everyone goes, "Wahey! Guess what Facebook are doing? Guess what Amazon are doing, Netflix, Google."

These are all great companies to listen to, and there's nothing wrong with what they're doing. It's really, really interesting. But for me, and I imagine a lot of you people, your business looks more like this. It's on the high street, and it's getting into the digital age. It's moving forward.

It's not a tech company. Google's a tech company, brilliant. Amazon's a tech company. Their whole management are tech. They love it. Talk to them about just releasing into production: "Yeah, go for it."

There's even an old-branded Thomas Cook shop on this one. So this is the high street. This is where most people come from.

And the reason why their businesses are doing digital is, many years ago, the internet got cool, and people started having a digital presence on the internet. So, "Hey, we'll have a digital presence on the internet. Let's do that. Let's have a website." And they make a shiny website. They hire some really trendy people with All Stars and bright shirts, and they go out there and they build them this really cool website. You can deliver it really quickly. Brilliant.

But your back-end systems are still the ones that are used in the shops and the warehouses and everywhere else. And this is what I mean by primeval environment: as quick and shiny as your website can be to deploy, your back-end systems aren't moving anywhere near as fast as the rest of it.

And this is the place we're at now in Thomas Cook, which is we're now looking at all these back-end systems and trying to go, "How do we make these move as fast as we need them to, to keep up with the digital side?" Because at the end of the day, that's where the money's starting to be made.

I don't know how many people out here, their businesses have probably gone from having a digital presence of 10% of turnover. They're probably now hitting the 50, 60, 70% of where the business is making its money. But everything that's doing the back end, the inventory, the crunching the numbers, everything is still in a legacy system.

We've got systems, and I'll be honest, the only way to find out how it was written is to get a clairvoyant to speak to the developer who did it. It's pretty true and it's pretty scary, to be honest.

But then again, the business are also here. This is where the business people are. They're in the high street. They have someone who sits in the shop. They speak to the customer for 45 minutes. They sell them a holiday. They sell them some other things. Brilliant. That's where the business is, and the business quite like not a lot of change because they want the shop to be open or they want the site to be up.

So the last thing they want is someone to come along and say, "I know everything. I know what you need to do. You need to release every five minutes. You need to have a release once a week."

It's great and we all believe in it. I'm pretty sure if I asked for hands up who thinks we should build a release every day. Come on. Great.

Hands up who releases every day into production.

Yeah, exactly. And that doesn't make you money, that application, does it?

So this is where we are. We can't go to the business and say, "Hey, guys, you've got to release every day. You've got to do this. You've got to move it here. Break this application into 40 pieces, then it's easier. Change is quicker, change is cheaper."

They look at you and go, "Why? It works."

And in fairness, they think it works because they're not the ones fixing it on the weekend.

But that's what it is. The business do not want to be told what they're going to be doing to change. And this is what I've noticed in every business I've been in. You in tech and we in tech, I'm assuming we're all in tech, and if we're not, you should be in tech because it's really cool. This is what we really want to do. We want you guys to release software. We want to be moving forwards. We want to be making money, getting to the market quicker.

We need to then make our business come along on that journey. And the one thing you can't do is, like anything, I bet you're all the same: if someone walks up and says, "You should do this," your first reaction is probably, "Yeah, whatever." You have to bring them on that journey.

And this is the difficult bit. As much as we can build systems, as much as we can build delivery pipelines, we can't deliver anything unless our business is willing to buy into that. They have to be willing to say, "You know what? We trust you to release code into our production system. We don't need to spend two days checking it."

And that's the journey where I think most people are on, and that's where we are at the moment in Thomas Cook. We're now building systems automation, automated tests, because we're trying to get the confidence of the business to allow us to actually do these move-fast, move-quick things.

It's very alien. It's even alien to the people I work with in technology in my company, because I come from the side which was always a little bit bubbled. We were doing digital, we did DevOps, we did continuous delivery. Great. The rest of the business hasn't been doing that. They've been wanting to do it, but without a reason to do so.

But now the digital side, as I mentioned before, is now bigger. They have to now come on that journey because we can't wait six months for the next release of the back-end system so it can take a Direct Debit payment, for example. We want it next week.

So that's where I am at the minute, trying to move these things along. And that's the bit where I want to open to questions and answers, because I think this is the tough bit of us.

This bit, and I'll give the next slide: this is what the business want to do. Great. They want to sell holidays. We have our own airline. Hotels. Yeah, you get it. I'm not going to go through the slide and do execution of PowerPoint. These are all the things that I bet, if you were to move around what your business does, everybody wants to do in a business. They all want these abilities and these agilities in their business.

And we all want to build really cool systems, and we do build cool systems. And I love to think we've built the best system, the best pipelines, and we've got an awesome continuous delivery system, but someone's probably got a better one.

But the truth is, and that's the fun bit, this is the easy bit. The fun part, the easy bit, playing with tools. I put that on there. I wasn't going to do that, but I know everyone likes to know what everybody else is using so they can say, "Oh, he should have used this instead," or, "Cool, I use that, too." And I didn't put everything on there. This is just a real soundbite of what we're doing.

This is all great. This is brilliant. We can deliver a change of software in the same day. It just has to go through the environment. So there is a time period, there is a promotion period. We could skip that if it was good quality. It's not. But we could do all these things really quickly. That's brilliant.

That's the fun part. That's what everyone comes to the conference for, saying, "How do we do that? What automation tools are you using? How are you writing your tests?"

The truth is, there's no real right tool out there to use, in my mind. There's a lot of wrong tools, but there's no right tool. So whatever you guys choose as a tool set, however you guys choose to do this, that'll be fine.

I will upset... Might have to shut the door because I think we've got sponsors next door. I'm not a believer that you need to buy something off the shelf. You will have to spend money. You will have to buy tooling. But give it to your development teams. Give it to your engineers and say, "Look, guys, what is it you want to do?"

I didn't walk in and say, "Guys, we're using Chef." We said, "All right, guys, we've got a choice of which tool to use. Which one do you want to use?" They did a POC, came back and said, "We want Chef. We want Jenkins." And that's, in my mind, how you build your tech up from the bottom.

You will never win the business if your tech teams aren't working together and aren't bought into what you're doing. And that's really the one key thing to say: in this journey of getting the business, which is difficult, you need your tech teams to be bought in. They have to believe in the tools they're using. They have to believe that if they're using Scrum, Lean, Kanban, they have to be in that 100%, behind you, supporting that.

You can't enforce this. You can't bring it on people. As I said before with the business, you can't just say, "I know best. We'll do this." You need to bring the teams on a journey, which is where the painful bit is, because it will never be as fast as you want it to be.

But if you can get your technical environment sorted out, if you can get your teams working together with a bit of harmony, a bit of argument, but a bit of jest, and you've got the tools behind there which they've all bought into, they haven't just been told, "We've gone and bought something, you have to use it." They've all bought into it. They've all had a say in it, and they still have a say in it because they can come up to you and say, "You know what? We'd like to use this linting tool rather than that one." And you make it possible for them.

That's where you really get the harmony. And until you've got that, the business one we discussed earlier on, where we get the hearts and minds of the business, that just disappears. There is no hearts and minds of the business until you've got the bottom layer fixed. Do that, great. Now you've got the business.

And that's the main point of my topic here. The business are great people to speak to. And I've got no more slides because I didn't actually have time.

Yep. It's one of the things, and this is why I wanted questions and answers, really. It's more how I do things. The business really want to trust, and they just need to trust what we're going to do. They need to trust that if we're going to release something into production, they're going to get something out of that.

Because remember, the great thing is we don't own the profit and loss ledger. We don't. They do. So if the site goes down for a day, it's them who loses out, not us in tech. So we have to be really, really keen that we make sure we win that business on that journey.

I'm fortunate in some parts of my business, they're on that journey, and they love it and it's great, and you can work with them and they push, and they push you more than you push them. They go, "I want this and I want that, and how do we do this?"

But for most people, there'll be those legacy back-end systems, and that's where the real challenge is for everybody. It's not your front-end, shiny website. It's not your front-end, shiny UI that you've given the users in the contact centers. It's your back-end systems, and that's where I think the real challenge is in the next... I'm not going to put years on it because I'll probably get it wrong, but in the next number of years, the real challenge is for most people: taking all these legacy systems, putting them into some sort of refactor, remediation work, and then making those systems actually ready for this century, where we are now. Making them ready for us to actually push software out daily.

Which is why I come back to the Google and the Facebook slide. These guys built their systems to be like that. They started their journey like that, which is why they can really shout about it. For the rest of us, we've got a world of now taking legacy systems, pulling them to pieces, looking at the real requirements of them, and therefore making them work in the DevOps continuous delivery way.

Now, I'll throw something out there, which is very simple. These legacy systems, somebody loves it. They're there for 15, 20 years because somebody loves it. It does just what they want, and they don't want to see a change. They like the 30 screens that they have to go through on that old VT100 terminal. That's what they want.

Okay? So you need to win their hearts and minds, and I'm not going to say I've got the answer, but you take them on that journey and be prepared that your shiny digital world is easy. Your legacy world is the difficult bit. And that's what we're working on at Thomas Cook at the minute, is the legacy bit.

And I'm going to throw it out to a quick video because I just fancy playing this video because it makes me giggle. Hopefully, I don't get into trouble.

The key route here is we all want to automate everything, and we should automate everything. I want to automate my job so they think I'm working when I'm on the golf course. That's what I want to do. Okay? So that's what we all do in life. But if you're going to automate something, be really sure to test it, be really sure to check your requirements before you put it into production, as our good friend will tell us here.

Is that on screen? No.

I don't use Windows normally.

Just let me quit that.

It all worked in rehearsal.

Audience: This is the test part before putting it into production?

Yep. You should test before you release.

Audience: And how are you going to measure success?

By how much everybody laughs.

So, as I said, be careful before you release your automation.

Video: You slipped and fell into a robot hand?

Video: Yes.

Video: Penis first?

Video: Yes. Now help me.

Video: I'd suggest a lubricant, but I have a feeling you fell on some of that as well.

Video: Not funny, Leonard.

Video: Really? A robot hand's got a death grip on your junk, dude. That's funny. I'll schedule it.

Video: Please, before my mother walks in, just get this off me.

Video: Okay, let's see.

Video: No, no, don't touch. The program is paused.

Video: Oh, well, then what's unpaused?

Video: No, no, I loaded the wrong program. The hand thinks it's holding a screwdriver in outer space. If you continue the program, it's going to start twisting.

I'm going to leave it there. It's a great clip. Get it on YouTube. But that's kind of the point. He should have tested what it was doing.

I'm going to throw it out to questions and answers. And maybe they're not questions I can answer, but let's try it.

Q&A

Q: How did you start the journey? How did you get people to buy in to making that first step change, particularly around the back end? So the front end can be the cool guys. It's that back end. How do you get the journey started?

A: It sounds really wrong, but the way to get that journey started is when they've got a real bad P1 incident. Because they're normally running around panicking. They don't know how to get the right monitoring out on it. And literally, that's a great place to step in and go, "Oh, I've got an idea." And they go, "Wow, you did that in five minutes."

Yes. And it is true with most companies, they're not willing to make a change until they've got a problem. And you just need to make sure you're in the right place at the right time for that problem. And that's the honest truth.

And you will find buy-in. There's always somebody who wants to buy in, but generally speaking, we want to roll the tool out across an old legacy system. The truth is, "Oh, we're scared what will happen. We're not too sure. We'll think about it." Four weeks later, they're in a big P1. We just rolled it out. P1 fixed. Now they've got the tool. And that's the first start of them moving across to believing in what we're doing.

I don't know if that helps, but maybe you can create an incident. It's not a bad idea.

Any more?

Q: I'm wondering if you should have another version of this film, which is Howard, the robotic arm, and the back-end problem.

A: Well, there is that one as well, I think, if you Google a bit more. There is the back-end Howard one, but that's more pleasurable by the looks of things. I mean...

Q: Can I ask a proper question?

A: Yeah.

Q: Do you have outsourced service providers at Thomas Cook?

A: Yes. As Thomas Cook, they do.

Q: Okay, another question. Do you have anything to do with them?

A: No. Well, very little. Very little, to be honest.

Q: It's all just in-house infrastructure, it's in-house tools? As far as you're concerned.

A: Yeah. So we're using cloud, like we should be. Public cloud, and everything is open source or... We have bought some things. As you can see, you don't get licenses for Chef for free, et cetera. But they're tool sets that we've purchased in bits and bobs and built the system together, the pipeline.

If people are interested, we've got Chef for automation and configuration, and on top of that we've got a Jenkins DSL Groovy pipeline with some little tweaks in it for ourselves. So that's kind of the embodiment of what we're doing.

I know everyone loves the techy bit, so if we want to stand up a new pipeline, it takes about, I think, 15 minutes, and that's just for the actual job to run, not to actually do the work. And then obviously, depending on the complication of the application, a Chef recipe, the cookbook recipe is two weeks, three weeks at most. Sometimes it's less than that if you've already got one that will work.

Q: But you've got to communicate with that legacy back end. That's a big part of what we're doing.

A: Absolutely, yeah.

Q: And not finding that the service provider contracts are getting in the way of that with counting tickets or changes or that kind of stuff. It's really easy to move stuff.

A: I think generally speaking, we do have some issues with that. We've got four or five main suppliers on a lot of the legacy systems. They trip over each other. That's the truth. Suppliers, when you get multiple vendors in, they trip over each other. And not on purpose, because I'm pretty sure they all want to help. It's just they don't know where they should help and where they should stop helping.

The legacy systems, generally speaking, are very easy because, as I said before, we wait for incidents to happen. As I mentioned, to us around a table at the beginning, loads of people, project, how long would it take to run, et cetera. And these are all the quotes you get from the vendors and everyone else. We just went out and built it. So the vendors have gone off now to work out how they can keep up with us, basically.

But it's not easy, and I'm not saying I've got the answer for it right yet. I'm a bit naughty. I just tend to ask for forgiveness, not permission. And that's the kind of mantra I live with. It's easier, by the way.

Q: In dealing with legacy, you can have the multiple strategies, right? You could decide to replace it, refactor it, that's one. You could also decide to make yourself independent of it somehow. How do you choose that strategy? Was it you or either?

A: I'd love to say I got to choose the strategy all the time. So obviously, it depends on the business owners, et cetera, on that. Generally speaking, the strategy is driven in two ways. You will have multiple stakeholders in different ways. So there might be somebody who wants to keep the tool and wants to remediate the tool and just tidy up and try and fix it. There are other people that think, "This tool's really awful. We want to get rid of it."

The hardest part is, and this is where I talk about the legacy system and the primeval thing, and we've probably all got this, is there is no actual right answer where you walk in and say, "We're going to do this," because you have to win the business over. And sometimes the best way of doing that is to allow them to do their project they want to do, which is normally remediate, but at the same time take some functionality out of there, get a small group of people together, run a POC, and show them how that one works.

And that's generally what we do, is we take the monolithic system, we cheekily pick something out, run it somewhere else. They go, "What? We update that every week." Yeah, and compared to every six months.

Q: But you treat that as a strategy. So you're making it smaller by taking stuff out.

A: You take stuff out. You take stuff out, yeah.

Most legacy systems, and I can't speak for everybody, any legacy system I've come across is monolithic. It's just had everything added to it. Some of our systems have got the most strangest things added to it, honestly.

And that's what you find, is you have to work out what is the real core value in this product? What bits change a lot? You could have 20 functions, just as an example, in a monolithic application. There's probably only two of those that ever get changed in a release cycle. So take them out from the other 18, take them out from each other, and then manage it that way. And that's where the value really is.

And that's the journey we're on with a lot of our legacy systems now at Thomas Cook, is breaking them down like that.

Q: That's actually what I wanted.

A: Yeah. Everyone likes to call them microservices as well, but they're just really APIs.

Q: So were you able to make these architecture changes at the same time as delivering new features?

A: Generally speaking, yes.

When we did the dot-com, as we call it, the one web... Just to give you an idea, we've got a platform now with, you might see thomascook.com. If you went to Neckermann and EL and some of the other sites we've got across Europe, they're all actually on the one platform. So there was a strategy originally to build that. So there was a little bit of six-month downtime, go off and build a new platform. That's where the original architecture came from, the digital side.

Now that architecture's up and running, and they can see it's a hell of a lot cheaper than the way it was done before. They can see it's a lot more agile, if you want to use the word agile, and dynamic than the way it was before. They now want a bit of that, basically. Which is the only way you do it, is by POC-ing it or having an area like digital prove the way for everybody else.

Can I get beer now? No, not yet. Not yet. I've got another one if nobody else has it.

Q: Remember Margot Cream, in her one for Zurich Insurance yesterday, she spoke about all the different stakeholders and what she called the monoliths: security, supply management, audit, compliance. And your industry's got lots of compliance issues and governance issues. How do you deal with them? Is it a, "Hey, screw it. I'll ask for forgiveness after I've wrecked it?"

A: Obviously, the great thing about when you're kind of semi-starting afresh and building a platform out or changing things is you have a lot more control over the previous legacy systems. So you're able to ensure that the PCI and compliance is there because you're building it with a structure of processes that mean you're going to meet that.

I think the security teams are always the interesting ones, because I do have interesting conversations with security. You get a tool and they go, "Oh, great. That's an amazing tool. Make sure nobody can see it." Yeah. No. That's kind of the security question.

And I think it comes back to, we talk about security for a second, it's proving the point that this is secure and it's valuable to the business, so we'll give everyone access to it. It's proving to security that actually, you know what? Yes, you could log onto this machine and get to that machine, but actually, to log onto this machine, you've got absolutely no chance of getting through there to there from the front end. You have to demonstrate it.

We're fortunate, I guess, in the sense that from a big business with a lot of applications, with all the pen testing and everything else that goes on, we haven't failed. So the area that we're growing up and growing out of is seen as being the place to go to if you want to be compliant.

That, I think, is easy because we have it built in with... We don't have flat files with passwords in it, we have data bags encrypted. So all the newfangled tools we're using mean we can actually be compliant really easily.

Q: Is that because you just dreamt up because you're really, really clever? Or did you talk to those guys and say, "Have you got any advice on how to make our passwords secure?"

A: No, we Googled it, to be honest. It's true.

Some are security guys. Are there any security in the room, before I offend you? Oh, sorry.

So there's a monolithic approach sometimes to security. Some people are really quite funky. They want to get some monitoring and some apps out there, see what's going on. Some of them want to always think about the worst-case scenario. And the worst-case scenario is not what is their real worst-case scenario. It is almost like, if aliens landed on the planet. It's the question that comes out.

So it's just proving. And it's not easy.

Your monolithic stakeholder's procurement is a great example. We've got a process going on at the minute and around the big table again, I was talking about, "Well, we're going to build a system where we can have 100 new VMs a month."

"How's that then, Lee?"

"I can do 150 a day, just builds and dry-run builds, so I can't use it."

So it is a hearts and minds thing. It is about getting involved with them. If you can somehow get somebody to fund you not to be part of it, that's the best way of doing it.

Otherwise, it's a lot of meetings.

I don't know if that helps or not, but get some budget, build it somewhere else.

Q: So you mentioned bringing the legacy systems in and getting them up to current expectations, if you will.

A: They're quite low, by the way.

Q: What's that?

A: They're quite low expectations.

Q: Fair. You've got to point them to something, right?

A: Yeah.

Q: In that effort, what skill changes for those people had to be applied? What new skills, what new things do they need to...

A: Yeah. Well, to be fair, we're going through that journey now, and the skills are difficult. I think a lot of people need to have more development skills than in the past.

So a lot of ops people, and this is no offense to the ops people in the room, have renamed themselves DevOps, but you need to have that development skill set. You need to be able to code. You need to be able to write tests. And I think there's a fear across some areas that that means if you come along and you're doing continuous delivery and you're doing configuration management, somehow, if they're not a developer, there's going to be a kind of, "Well, what am I going to do next?"

The skill sets are quite simple. You find the people who have the skills or want to have the skills, and you train them to have those skills, or you improve those skills. And the ones who don't want to do it and don't want to change, they're fine because there's plenty of work for them to still do. You're not bullying a P45 down the road. You literally are going to make people innovative.

Your sysadmins, your operation guys, whoever in the room who does that or has done that, the last thing you want to do as an ops guy is install a package overnight that has been given to you from a development team because it's mind-numbingly boring, and you know it's going to go wrong and keep you up the next night.

And I think that's the hearts and minds of the training. You've got to bring people on that journey. And the ops guys who don't want to go on that journey, they still exist. They're still doing everything they were doing, what they enjoy doing, I think. I don't know if everyone agrees or not, but I think they're still getting to do what they enjoy doing.

But you need to bring them along.

Moderator: We have time for one more question.

Q: When you bring up the changes of being able to release every week or every day, whatever the frequency, how do you keep up the changes through the business? How do you train them to understand the new features that you bring in or things like that, compared to the legacy system that was changing once or twice a year?

A: Yeah. I think this is where it gets a little bit easier as well, actually. I think if you change your system every six months and you make big changes and they're fundamentally different, the users are confused. They've got lots of new options on screen, et cetera. It's not particularly good. If it's an old legacy-style system, again, it'll be quite confusing.

There's two phrases I like to hear on this one. One is, if you make changes regular enough, it's not a massive change, so it's easy to learn. And what you need to do with your systems is, and I love this phrase, make them user seductive.

So if you're making changes, make sure... It's like if you go onto Amazon today, and I'm sorry to use Amazon as an example, there's loads of features in there. You don't have a training course to use them. They're very seductive. You know where to go to. They're very simple to follow. And that's the one thing that, within architecture, you need to do that. Keep them seductive.

Thank you, guys. I've been told it's time up, so...