Sanjeev Sharma

Excellent. Welcome to this nice and warm, toasty room.

My name is Sanjeev Sharma. I'm going to be hosting this panel today, and it's going to be kind of an interactive session. If you look at the screen behind you, there's a drawing on it. We don't have any pre-prepared slides. The only person who violated my rules was Mark. He came with a picture instead of drawing it out, so we will refrain from commenting on his artistic abilities.

The panel today is going to be about: how do you assess your organization's readiness to embark on a DevOps transformation?

My role: I'm the CTO for DevOps Technical Sales at IBM, and I have been in this role for around four years. I've been working with clients around the world, and they primarily ask us two questions. In the beginning, it used to be, "What is DevOps?" But we are way long past that. The two questions they ask us today are, "How do I start with DevOps? Where do I start?" Or, "I've started with DevOps. It works in small, co-located, two-pizza-sized teams. How do I scale?"

Those are the two questions we are going to attempt to answer today by having a conversation around it. Where do I start if I want to start? Or if I've already started, how do I scale? What steps have organizations, other teams, and companies taken in order to embark on that journey?

I have three very esteemed practitioners of DevOps today. This is not theory, as has been true for all the sessions at this event. But instead of me messing up their introductions, which I have a great track record of doing, messing up my panelists' backgrounds, I not only want them to talk about what their roles are, but also take a few minutes talking about their experience at their organization with adopting DevOps.

Then we'll go into a Q&A session, and if time permits, we have over an hour here, maybe even open it up for some questions from the audience. Gareth, you have the microphone. We'll start with you.

Gareth Evans

Is that coming through?

Yes? Yeah? Okay, great.

Hi, my name is Gareth Evans. I work for a company called Monitise. Monitise are based in London. We are a fintech company that historically have been based in the world of mobile money. What that really means is we have provided mobile banking, mobile money solutions to a number of banks worldwide. I think it's around 300, 350 different banks, mainly in America, some Europe, some UK.

My role within that company is I run the cloud platform operations team out of Cardiff in the FINkit side of the business. FINkit is a cloud-native platform that we're trying to push, which allows for rapid development of fintech applications where compliance and security are critical, where they are the number-one area that we need to focus on.

We have been using DevOps processes for around the past two years. When we started this journey, trying to take us down this FINkit route, it was a complete re-platforming of everything that we'd built previously. It allowed us to actually set up almost a company within a company to build out this new technology, test it out, put in the processes that we felt were missing elsewhere in the business, and then gradually try to infect the rest of the business. I'm using the word "infect" there. Try to infect the rest of the business to try and see what we're doing, so that we can grow and expand in doing that.

So I'm going to say it was very much a bottom-up approach to introducing DevOps into an organization.

Sanjeev Sharma

Excellent. Pass it over. Mark Howell from Lloyds Bank.

Mark Howell

Hello, everybody. Can you hear me at the back? Thank you.

My name is Mark, Mark Howell. I work at Lloyds Banking Group. I'm sure that most of the people in the room will have heard of Lloyds Banking Group. I'm going to cover a few things about Lloyds shortly, but my role is I work in our digital banking part of the business.

What that is, is the part of the business that's responsible for all of our mobile and internet banking offerings: desktop internet banking, mobile applications, and the internal services we also provide to our colleagues in branch and telephony as well.

A little bit about Lloyds to set the context, because I'm sure everybody in the room has heard of Lloyds Banking Group. We're made up of Lloyds Bank, 250 years in the business last year. You might have seen the media campaigns. We have Bank of Scotland and Halifax as our primary brands. We have the largest mobile banking platform in Europe. We currently serve in the region of 12 million online customers. Seven million of those are on our mobile application.

When we describe that, what I'm trying to allude to you is we have a responsibility within the bank to make sure that we make change safely and effectively. In other sorts of business lines throughout the world, if you can't stream your favorite movie, internet series, whatever it is, on a Saturday night, you just get a little bit annoyed. Whereas we have to put change into production very carefully and make sure that we don't affect our customer experience. That customer experience now, the bar's being raised by your Googles, your Amazons, your Apples, et cetera.

I joined the organization about three and a half years ago. DevOps was, to Sanjeev's point, "What is DevOps?" Basically my remit at the time was, "Make it better."

We are a very large organization. We have large operations departments. We have large development arms. We have large enterprise, I'll use the current terminology, architecture, methodology, and innovation towers within the group. Getting change into a production context is a series of handoffs. It is a series of emails, ticketing systems, et cetera.

With the remit of "make it better," where do you start? I'm sure we're going to come into that in a bit more detail later on. My job was to look at the way in which change started with the business, started at a developer being asked to do something, and mapping that way, and how we get those changes safely into a production environment, making sure that we do all the Reg man stuff as well as all of the testing that needs to be associated with all of the volume of change.

We do quite an amount of change over the year for various discretionary projects, legal Reg man projects. Through the talk this afternoon, we'll get around to talking to you about how we addressed sponsorship, how our executives supported us, what we did to show them what good looks like. Sometimes that is a seek-forgiveness-rather-than-ask-permission model, especially in very, very large corporates.

I'm sure we'll come back to talk about that later on. That's all from me for the moment, so I'm going to hand across.

Sanjeev Sharma

Rosalind, who just did a great job. I don't know how many of you were in her mainframe session, but it was a phenomenal session. So thank you for continuing to talk without almost no break.

Rosalind Radcliffe

No problem.

I'm Rosalind Radcliffe. I'm a distinguished engineer in IBM. I am responsible for building our tooling as it relates to DevOps for enterprise systems. So making sure our products and tools allow you to do DevOps transformations, as well as responsible for our internal DevOps transformation for our mainframe development teams, as well as working with clients to help them on their mainframe DevOps transformation.

Though in the client's case, when I say mainframe, I really mean mainframe and everything else. When I work with clients, it is almost always a shop that has Z, and I'm helping them bring Z into the rest of the world of DevOps and the transformation there. It's not that we only do mainframe. It's that we need the mainframe side not left out.

Inside IBM, we have lots of groups responsible for lots of different parts of the transformation. In my case, I just happen to work in the Z organization, and so I get to help them transform. I do a lot of work with clients in helping them understand where to start, how to move forward, what are the right choices, and understanding where they are now to allow that transformation to happen.

Sanjeev Sharma

Thank you.

Before we proceed, let's set some context. I said I'm going to be drawing live as much as I can while the folks here talk. Let's set the context about what the view of DevOps is. I love the definition given this morning, I believe it was from the gentleman from Barclays, that it is BizTech, architecture, analysts, everything that is needed all the way out to ops, working together. That's kind of, at IBM, our definition of DevOps.

If you see the drawing over here, I have the line of business feeding requirements into the IT organization, and the end goal is to deliver business value to the customer. Whatever it is: whether it is a service like Monitise does, whether it is a banking capability like Lloyds Bank does, whether it is a tool or software like IBM does, or anything else it might be, or a movie like a company like Netflix might do.

The goal is: how do I get it over to the client, convert it from requirements all the way out to code running in production in the most lean and efficient manner? It involves taking care of people, process, and tools.

With that context, let me start with you, Mark, as you have the mic in your hand this time.

Mark Howell

All the time.

Sanjeev Sharma

When you started at Lloyds Banking Group, as you said, three years ago they were asking you, "What is DevOps?" How did you go about defining DevOps? Do you think there were multiple versions of DevOps or definitions of DevOps floating around? And did you actually take an effort to arrive upon, "This is what we, Lloyds Banking Group, call DevOps"?

Mark Howell

Yes, we did. At that time, it was: DevOps is a bit like Agile. There's loads to it. You just pick the bits that are applicable to you. It's one of the buzzwords, and everyone's doing DevOps or everyone's heard of DevOps.

But my approach was to look at our, what we would call a build pipeline now, and look at the component parts of that. We've all got build pipelines, because there's a point at which a piece of code gets to production. Back in the day, we might not have had the tools and the processes and the automation to do that, but there is that there. You've got to look at: how do I want to improve and construct that pipeline together so that we can bring automation and repeatability to that process?

Looking at the old established tools, doing it in the bank in the day, looking at the way in which developers actually interacted with the code. So let's say 10 or 15 years ago, when CruiseControl came along, and that was the mustard at the time. CruiseControl was the biggest thing on the face of the earth, because we could actually do a build after somebody actually checked something in, which was great.

But then we quickly moved into, "Well, that's a very reactive model." Now, the way in which we get quicker is we want to communicate back to developers, fail fast. Looking at tools and processes we can put in place up front for developers so that when they try and commit something, what they get is instant feedback on, does it build?

Fifteen years ago, CruiseControl: developer checks something in, goes on holiday. Then it breaks the build. He's nowhere to be seen. Someone else doesn't know what he's done, et cetera.

Looking at that part of the process and chunking it up into pieces that we would now build up a DevOps pipeline around. From the point of change, looking through that cycle in terms of how can I make the cycle quicker, how can I use automated technologies in order to be able to build it, to examine the code, to be able to run automated tests against it in a fast way, so that we get the quality up front with the developer.

We enforce unit tests, and we enforce automated functional tests, and we enforce some sort of code rules that will be managed as part of, if you like, the build part of the pipeline.

My approach was very much, "What is DevOps, everybody?" And everyone's looking at you for a definition. It's like, well, I can draw the pipeline and what DevOps is before DevOps arrived, in terms of the manual steps, and breaking it into the chunks and the small pieces that I can influence, and then build up the picture.

Within our large organization, we've got 300, 400, 500 developers, and you don't transform them overnight. So pick the small bits of what would be the old manual pipeline and start transforming pieces and bring it together gradually.

What that does then is, we talked about, I use the term seek forgiveness. We went and sought forgiveness for the sort of things that we were doing. We're a large corporate. We have very strict rules about the types of kit we can use, the types of software we can use, and I'm sure this resonates with a number of people around the room. But sometimes you need to bend the rules a little to show people what good looks like. Then that's when you build momentum.

I have great support from our senior leaders within the bank, and they're good technologists as well. They're great people to go and show, "This is what good could really look like." We are on a journey. As I sit here today, we're not at the end of that journey. In our little world in digital, there is more to do, but we are gradually, gradually expanding like a gas outwards, from the real start of someone has an idea to the point at which that ends up in production.

Before I give someone else a go, one thing I think is important and one thing that I always bring: as I say, we are a retail bank in the UK. We have to have the appropriate risk appetite. We don't have developers clicking buttons, committing a piece of code, and it arriving in production. There is nothing wrong with getting to the front door of production as quickly as possible.

A lot of the pipeline techniques that we've got and the tools and processes we've adopted is enabling us to do that much faster, and then gives me the opportunity to go outside of the digital part of the organization into other areas and start helping those guys understand what we've got, how we've broken down barriers. One of the things we'll talk about later on is culture, and in a siloed organization, how do you break down those walls?

Sanjeev Sharma

Excellent. That's the most comprehensive answer I've seen.

I think you've set up my next topic area, so maybe we just move forward to that. What I've drawn here is a typical delivery pipeline. I apologize, I'm having you turn back.

If you look at a delivery pipeline, as Mark pointed out, it's made up of several stages where requirements are being converted to code, and then code is being built, it is tested, there are different kinds of tests, it goes into integration, integration testing, all the way out to production.

If we look at DevOps from a very narrow lens of saying, at the end of the day my goal is to make this delivery pipeline as lean and efficient as possible, then the goal is to identify waste in this delivery pipeline and start working on alleviating that waste, which is exactly what you said you did. You started taking two adjacent silos in this pipeline and said, "How can I make the handoff between the two of them? How can I make the silo itself more efficient? Then how can I make the handoff between them more efficient?"

Something we do a lot with clients, and we've done over 100 of these over the last couple of years, is a value stream mapping exercise. We have a marketing name for it, the DevOps Optimization and Innovation Workshop. That's what we call it. But it's a half-day workshop we do where we actually do a value stream mapping of a delivery pipeline like this, where we look at all the silos you have, all the stakeholders you have in those silos, and all the artifacts that are being handed over from one silo to the next and being moved from one environment to the next, and making that process more streamlined.

Gareth, when you started adopting DevOps at Monitise, how did you identify where to start? Was your pipeline-centric view one of your approaches also?

Gareth Evans

Again, because we were starting a new project, it was going to involve lots of new developers spinning up new teams, even new spaces for people to sit and things like that. So we actually started right at the very beginning, which is with the onboarding process.

When a new developer comes in, how long until that developer becomes productive? And how long does it take until that developer can potentially, I'm not saying they will go to production, but potentially push a change right the way through to production?

That becomes a measurement exercise in trying to work out where those bottlenecks are.

The very first thing that we actually started building was a Vagrant-based development environment. The first thing that you do when you come in, you have your new desktop or your laptop, and this is what you're running on, and there is a `curl` command that you basically pipe to `sh`, and that downloads and installs everything that you need. It's all version controlled. It's there.

Within around 45 minutes to an hour, depending on how the internet's doing that day, you have a development environment with all of your IDEs installed, configured, Git working, everything ready to go. It will also clone the repositories that you're assigned to be working on, so you're ready to go straightaway.

That was the very first thing that we looked at.

Once we'd got that process working, then we looked at things like where we felt that the main overheads were going to be, especially when you're starting off. Quite often that can be in repository creation and Jenkins or build system maintenance and updates.

The second thing we built was a self-service system for the development teams to go in and actually say, "I want a new repository. This is what I want it called. This is the description." Based on a couple of tick boxes, it would automatically create the right build and review process and hook it all in, and then those repositories appear on the dashboards ready to go. So that was the next piece.

Sanjeev Sharma

Excellent. So that was a challenge you identified that in your existing projects, onboarding was a big challenge? And probably your biggest bottleneck.

Gareth Evans

Yeah. We knew that, somewhat embarrassingly, it could take two weeks for a new starter to get an environment up and running where they could actually build the previous version of our application. That was something that we felt was just unacceptable.

So with a bit of a twist in how we do it, or how we implemented it, we took that two weeks, and in some isolated test cases, we have actually managed to get new starters joining the company and pushing to production on the same day.

Sanjeev Sharma

Excellent.

Gareth Evans

It's not part of our banking app, but it was still what we would consider a production change.

Sanjeev Sharma

But it sets the bar. This is the expectation. You should be able to, the day you come, be ready to push something to production.

Gareth Evans

Exactly.

Sanjeev Sharma

It might not be code the customer ever sees, but it sets the bar of what the expectation of continuous delivery is.

Gareth Evans

Often our version of what production is is different from the customer's. We would say production could also include a lot of the support systems that we've built to manage this process internally.

Sanjeev Sharma

Sure. I remember meeting a client, this was a small startup, and they said anybody at the company knows how to push a button, and whatever they push, whatever's in the latest integration stream, gets deployed to production. They said, "Our CEO can do it. Our CFO can do it." Of course, that's not happening at IBM.

But at most organizations, if it's not a regulated environment with SOX requirements and all, you could do that. Why not? In other places you wouldn't, because maybe your business is, even if there's no regulatory requirements, your business might not be ready to consume it.

Mark, let me have you go next about how did you go about... You already alluded to this, but was there one major bottleneck in your delivery pipeline where you said, "We have to attack this first. This is where we waste the most time"?

Mark Howell

Yeah. The first thing we actually looked at was our build time and what value that build process was giving us.

Again, being as we are, we're confessing to all: it took us two hours to build a Java application. We had looked at that, and because it was very CruiseControl-centric, you will only find out it's broken after somebody's tried to build it. Going back to the chap's-gone-on-holiday scenario as well.

I used to work in the area of the bank that used to be on the supporting side of that, and we used to get a ticket raised, dragged out of bed on a Saturday morning: "My build's failed. Can you help?" Go and look at the log, and it's just a compile error. It's like, "Well, I don't write your code for you." We do most things.

But we looked at that process there and how can we get that feedback back to the developers? We can't have them waiting two hours to find out that something doesn't work. That was a combination of the version control tool we used at the time, and that was a combination of infrastructure, build tools, and CruiseControl mainly. ClearCase was the tool that we had in place at the time.

We looked at that experience and thought, "How can we make that better?" One of the things was to migrate to Git. We currently use Git and Gerrit as our version control tools. We have a very large application and in the region of 250 developers changing it at the same time, so we have to rely on our branching strategy to support that.

We had a thing internally within the bank, sort of your proudest moment. I managed to migrate 250 users. On a Friday, they checked into ClearCase. On Monday, they came and used Git. It was seamless. We didn't miss a heartbeat.

The first thing was to transform the user experience in terms of the dev tools, going back to Gareth's point there. Giving them something a bit more lightweight, using things like Jenkins, which allowed us to build much more quickly. We addressed some of the infrastructure issues as well. In a large corporate, you're not walking to PC World and asking for a new box. There are things you have to follow to get kit, but we did that. Maybe a bit of seek forgiveness on the way.

But we then could quickly show that that application could be built in 15 minutes. As part of that, then, is that when we start increasing value to that. So running my unit tests, bringing in my BDD tests, bringing in parts of the pipeline, and actually being able to measure and control the quality of the end product.

Those are the sort of things that we went after first of all.

Sanjeev Sharma

Okay. Rosalind, same question to you. Of organizations you have worked with, I know you can talk about all kinds of organizations, but let's stick to your mainframe clients. What do you think is the biggest bottleneck you see, and where would they focus on?

Rosalind Radcliffe

So it depends. Okay, you don't want that answer.

In most mainframe shops, they have started to take this pipeline and transform on the distributed side to do DevOps. Most of them have started to look at, how do I do these pieces? They've either decided Z is inapplicable, or they've said, "Yeah, I don't care. I'm just going to leave those guys out."

When you look at that, you go, okay, here's my pipeline on the distributed side, and I'm stuck because it's dependent on changes in the system of record, and this doesn't keep working, so let me do something in the mainframe.

The problem we have in the Z side is it's already a system that's completely automated that just works. A build on the Z side is a few seconds because you only build what's changed. It just works, and it gets deployed to production always in a consistent manner.

The problem is the tool that's being used to do that was built in the 1960s or '70s, and if you're lucky, it's been updated in the '80s. But that's the technology. It doesn't handle any of these modern things. You can't do the service interface that you also happen to have. It doesn't handle any configuration. It doesn't, it doesn't.

So it does its little piece, and it does it well-ish, but it doesn't let you do anything in a modern way. You can't plug in your automated testing. You can't deliver to a new test environment because it's got a set hierarchy.

The biggest problem is this difference, that the Z guys, it works, and it is all automated. You'd think it'd be wonderful, except it's done in a way that isn't extensible. It doesn't help. So the biggest problem is helping them understand, how do I move to modern tools, modern technology, get a modern pipeline, get a modern SCM, do things in a new way, so that I can take that environment and I can push a button and deploy it to that distributed side and to the Z side at the same time, so I can do it efficiently across the board.

That's one of the biggest areas.

The other one is testing. Most mainframe shops do very little automated testing on the mainframe. Usually, it's somewhere near 0%, and anybody that says higher than zero, I say congratulations because you're better than norm. The highest I've ever heard is 10%, so that has to change. If you're spending months doing manual testing, this isn't going to happen.

Helping them transform in the automated testing, bringing in automated testing, understanding that you can unit test Z code. Why not? You can. There's no reason not to. You just don't.

Those are the biggest places where people have bottlenecks and need to transform. The problem is we can't move overnight. Our SCM and build tool and deployment tool is one, and so it's harder to do this transformation. It does take organizations longer to do this kind of transformation, but once they've done it, they're now in modern tools. They're now able to take advantage of those tools and add in functions faster and move faster.

The fast and slow, I think, also depends a lot on the culture, but we'll come back to that. I don't want to go into the culture question now.

Sanjeev Sharma

We have done a lot of surveys within IBM. We've done one with Forrester, one of the analyst firms. I know the State of DevOps has come out recently, which Gene spoke about this morning.

One major bottleneck which clients talk about a lot, saying that this becomes the biggest waste for us, is environment availability for dev environment and test environments. It is understandable if it takes a long time to get a production environment up, given the size and the security and the compliance requirements and access control it might have. But for dev and test environments where there's a finite number of people within your firewalls and confines who are going to use it, it seems kind of unacceptable that it should take two days or three days to get an environment. When I mention that to people, they usually go, "Try three weeks."

How did you handle that? As you have the microphone, we'll let you start, and you can talk about LPAR availability and whatever you want to regarding the mainframes. I know you discussed it a little bit in your session.

Let's talk about environments, because that is a major bottleneck for most organizations.

Rosalind Radcliffe

On the Z side, I can tell you that if companies wanted to, they could have always just brought up another LPAR. The way z/OS works, for those who know it well enough, you know if you have z/VM, you could just start another LPAR and it actually takes no time at all.

However, the processes in place say it's probably six months if you're lucky, and the real answer is never, because they're not going to create a new LPAR for dev. Why? Why do you need it? That's the wrong answer. So it's not going to happen.

The real answer we do now is we provide a way of cloud provisioning z/OS running on Intel Linux. I can now build up a golden image of my Z system, and my development teams can now create their own on the fly. Depending on the company, it's maybe 40 minutes, and I've got an entire new z/OS environment for my team. Done.

Wouldn't that be wonderful? It really should be about 10. It might be down to two. I've got a company that wants it in three minutes. Once you get it down to 40, they think I can do it faster. They could never do it before, so I thought the 40 was pretty good.

But cloud provisioning an environment, and what they more often do is provision it next to the distributed environment they're also cloud provisioning. So they can now cloud provision a Z next to distributed, and now I have my entire environment for my development team to work.

Now, this is dev and unit test and function test capability. Once you get to performance test and stuff, you're back on real Z hardware. But this gives you the ability to dynamically provision environments, allow you to run automated tests.

I even have companies looking at dynamically provisioning z/OS as part of the build process and running all the automated testing. Why not? If you can, why not? It's actually not hard to do, but it's the thought process change. It's the thought process, especially in Z space, where most companies have five or six LPARs. Large companies have 13, six. I've heard as high as 60. So some people have larger numbers, but they're tiny in comparison to a distributed environment with thousands. This idea that I can now have more systems is very foreign, but very useful for development teams.

Sanjeev Sharma

Mark, same question to you regarding environments and how are you addressing it?

Mark Howell

One of the things that I am responsible for is all of our non-production environments for the internet bank.

Sanjeev Sharma

Right person to answer that question.

Mark Howell

Right person to answer the question.

Environment availability is my number-one problem, as I sit here today. So what are we doing about it? Let's describe the problem.

We're a large bank. We don't go to AWS or people like that in the cloud to dynamically provision environments. We have LRM things to consider. We've got DPA. We've got a myriad of regulatory requirements about our environment. We can't open the door to AWS and start deploying customer code or customer details into an environment like that.

In a large organization, I think I alluded to this when I started the introduction, we can't go to PC World and buy a computer. We have large parts of the organization who are responsible for provisioning new tin. Because that's the model we have, and I'm sure this will resonate with a number of people in the room, we will stand up a static set of environments that represent each of your boxes there. What we will end up doing is we will deploy them repeatedly. So, the anti-pattern of cloud, if you like.

We have a number of static environments that represent a route to live for the internet bank, and I have a team of people that continually deploy into those environments to meet the business need.

When you deploy, ultimately you're making the environment unavailable. So what do we do? Let's have two of them. We'll have one environment that's up, and we'll be deploying the other one at the same time. That's a pattern that we've had in place for a while.

But as things become more complicated and the deployment takes longer, and there's more and more configuration that needs to be done around the application you're deploying, that means it takes longer to deploy. There's more chance of defects creeping in in the process and defects in an environment scenario.

One of my taglines is, "It's always an environment problem." Doesn't matter what anybody does, it's an environment problem. And guess who gets kicked?

It's a bit of a standing joke, but we realize that environment issues come from anywhere. So what are the things under our control, and how can we be more efficient about how we minimize that outage? How do we make the deployment quicker?

We've invested in IBM's UrbanCode product. We've looked into that and moved away from our current technology, and we're seeing some major benefits in the amount of time it physically takes for us to get change into an environment.

Additionally as well, that's been a great segue, and we're going to talk about culture later on, but how do we then empower the people doing the deployment rather than having to have a process based on a ticketing system, or having to go to another part of the organization with a "Mother, may I?" or "Could you?" Rather than empowering the guys themselves to make the change, to get the deployment done as quickly as possible, so that window is as small as possible, so that the testing uptime is the thing that we're measured on.

Ultimately, project managers, program managers, the business is only interested in, "Have you tested that bit of change?" So the environment availability is through there.

Going back to your picture behind me, I'm sure we're all familiar with the route to live there. We can relate to a number of those environments, and we have a number of releases that have a path through many environments in those sorts of categories. I have a team of people that click deploy.

That's before we've started looking at UrbanCode and how we can orchestrate that release. The internet bank, let's just have one thing called that, one central place and one source of truth for a WebSphere endpoint. Let's make it up: a port for this or whatever. Let the right people change it, so we're not having pass the parcel around a number of teams, and make sure that we can get the change into that environment as quickly as possible.

Sanjeev Sharma

Excellent.

Gareth, let me qualify the question for you, because if I remember correctly, hearing what I read about Monitise and what I've discussed with others from your company before, you are doing 100 to 200 deploys a day. Is that right?

Gareth Evans

Yeah.

Sanjeev Sharma

So obviously, you do not have an environment availability constraint. How did you get to that three-digit deploys a day?

Gareth Evans

You've just taken away my punchline.

Yes. When we started building the FINkit platform, our head of enterprise architecture came up with a series of principles that we were going to start using to help us define some goals and work through it. The two that really matter in this case are: automate everything, that was number one, and the second one was no humans in here.

What that second one really means is that you can't SSH into a box. We don't allow people to log in to a box and make a manual change. It has to go through configuration management. It has to go through a review process and an automated build to be pushed out onto that environment, so we don't end up with these snowflake environments where everything's different.

That comes probably back to your first slide, which one of the boxes on there was the tools. Using those principles, we're able to select tools that work for us. Some of those are Jenkins, Ansible, Cucumber, things like that, standard things.

The other one is Cloud Foundry. We run our compute stack, where probably 95% of change is happening, maybe more, based on top of IBM Bluemix, which is Cloud Foundry. That means that we can't actually get into those VMs and do that.

With a simple bit of scripting that's controlled from Jenkins, the hardest thing about environment management becomes, what do I call that environment? It becomes trivial. And yes, we have gone from 50 deployments a year to 200-plus deployments per day without downtime.

Sanjeev Sharma

Excellent. Perfect.

Let's do a segue into your environment. You drew this out for me.

Gareth Evans

Yeah. It doesn't look as good when it's zoomed in.

Sanjeev Sharma

This is your delivery approach, right? Could you walk us through this and tell us a little bit about how you got to this, and what are you working on improving next? Where do you think the next bottleneck is? Because there's always the next bottleneck.

Gareth Evans

Yeah. The thing about bottlenecks is there's only really one at any point in time, and once you remove that, you then find the next.

This is our standard delivery pipeline that I draw up for everyone that asks. At the top, we have developer. I would use that to classify our Vagrant-based development environment, which is very quick, up and running. It's what the developer does in their day-to-day job on their desktop.

Coming down, we have a build that's often classed as build and review. We use Gerrit as our code review tool. That enforces pre-merge builds on any piece of software that goes through, and it also allows multiple sets of eyes to view every piece of code that goes into that mainline.

I'm not sure if you were in the talk across there about using DevOps in a highly regulated financial environment. Those PCI requirements that, I can't remember his name, he was talking about, we have those requirements. We have to be able to prove that more than one person has looked at a particular piece of code, and Gerrit is the tool that allows us to do that.

Taking that forward, the release process: you want to build a piece of code. We're using Java, or Java and Scala. We're Maven-based primarily, but we want to reduce the time that artifacts spend sitting around before they're pushed out onto environment. So we use what's known as an atomic release process. Every commit actually becomes a release in its own right.

It's a modified build pipeline on Jenkins that just allows us to very quickly, every commit becomes tagged, it's versioned, we know exactly what change was in that piece of code, and it's only ever one change. One commit creates one version, and that triggers one deploy. Those are some simple rules that allow us to optimize around that.

In terms of the deployment, we're not using UrbanCode. We did for a period of time. We're actually now using a customized Jenkins slave. It gives us the network segregation that we need, and it allows us to really pipeline these deployments through. Again, because we're using Cloud Foundry, the two work really nicely together. We use the Cloud Foundry API to push out code onto there.

Then at the top end, we've got monitor. We have a process of continuous test. Whilst we have tests running at post-deployment, we also have a series of tests that are actually running against all of our environments continuously the entire time, producing logs, feedback, everything that we need, and they're all fed into a centralized management solution.

The idea is that at any point in this pipeline, we're getting as fast feedback as we possibly can. The main way to get the feedback to developers is, nobody likes emails because that was like 1990s. The stuff that people like is using the chat tools.

We've developed a series of integrations with Slack and various other components. For instance, every time a Cucumber job runs, the report is published to a Slack channel, providing there's been some change. Every deployment that goes out in the company, whether it's a test environment, SIT environment, or even to production, there's a series of channels that you can go and subscribe to that gives you a big thumbs up and a pint of beer if it's successful. If it's rolled back, it gives you the reason why, and people are then alerted to go and notify.

Feedback is absolutely crucial.

Sanjeev Sharma

Excellent.

Mark, why don't we head to you? I have to apologize because this is how I have to show your picture. I had it taken out and put in a Word document, but then when I tried to recover it in my iPad, it's crashing. Instead of wasting time, I will just paste it in an email with all the text cut out.

Mark Howell

Thanks, Gareth, for stealing my thunder as well.

Email, don't you love it? People just won't quit.

We've got very much a very similar, identical set of tools. Arrived when we did that analysis on how can we make it better, we have Git and Gerrit. As part of our Gerrit workflow, we will have, as a pre-commit quality gate, we'll make sure it builds in Jenkins. We'll run it through Sonar, make sure the "don't make it any worse" button in Sonar. We have a plus-two peer review.

For something that's been evolved over eight years that we sort of turned up three years ago with a set of tools, we're not going to get 95% unit test coverage. What we want to do is start quietly instilling some rigor around the process.

I think there is a button called "don't make it any worse" in Sonar. You turn it on, and let's make it up, there's 50,000 alerts. You are not allowed to have 50,001. It'll break the build. You've got some real solid developers come back, and they've fixed 10 of them. Your new benchmark is 49,990. Just checking my maths.

So you chip away at it gradually, but it enforces the right behavior.

With the pre-checks done, we commit that code onto the target branch that we're going to actually ultimately deploy. Then we have a variety of testing tools: BDD tools, Selenium, JBehave. What else have we got? BlazeMeter, Sauce Labs. We have a number of tools in that space as well that we will use as part of the pipeline that will enable us to get the deployment unit created and with a known quality.

That then is picked up into our environments team. In our new environment, we're building a brand-new environment, and we're bringing in UrbanCode into that environment so that the artifact that's produced is immediately available in UrbanCode. I have the decision whether I actually want to give someone a button, or I want to kick an API.

As mentioned earlier, a lot of our environments are static. They're stood up. We just repeatedly deploy to them. It isn't a problem. But then we make the decision of, does the tester, when they're ready for the deployment, click the button themselves, or do we want to automate it into a sort of what we would call a black/white pattern? It's red/green, choose your nomenclature, but basically there's the one I'm testing, there's the one that's going to be next, and sort of do it that way around.

We're seeing great benefits from using some of these tools. The more and more talks we give, and speaking to people like Gareth as well, we're arriving at a very common set of tools. They give the developers very fast feedback, because it's all about making sure what they write is of a known quality.

How quickly can we get it into an environment so it can be tested as automated as possible? Some things you still have people clicking on screens, but using AFTs and things like that, automated functional test, in terms of, "I can get to an internet banking application. I want to get to an account overview as a customer. I want to make a payment, standing order," the usual sort of tests you do.

Ultimately, then, when we get nearer production, because a highly regulated environment, as we know, so what processes could we put in place? To your point, Gareth, the segregation of duties. The developer not actually putting code into production, but UrbanCode manages that all for us. We have a separate part of the business that own the thing in production. They are the guys to do it. They have rules around, so the same guy who starts it has to be checked by somebody else, and we can implement that within UrbanCode. And it's all auditable as well, which is a very key requirement.

Sanjeev Sharma

Excellent.

One of the interesting things I noticed: both of you took a snapshot, and you drew. I'm pointing that out over and over again. The thing you put in the middle was feedback, because that is what drives the continuous improvement process.

Rosalind, from your experience of working with all the clients, could you comment more on the feedback loop and how you're seeing customers succeed? Are there any stumbling blocks which people should be aware of as they set up some kind of a feedback regime?

Rosalind Radcliffe

Feedback is key, and fast feedback is key.

The comments that have been said about building the pipeline so that it's fast and efficient, I get my feedback as fast as possible. Things like, why even bother to do the build if you don't pass a certain set of quality checks? Putting those things in are the things that I'm seeing clients do so that they get the feedback as fast as possible.

That's cross-platform. It doesn't matter what I'm doing. I can run those reports. I can get code quality reports. I can understand. In fact, a lot of organizations are trying to put on deliver or check-in checks to say, "Are you right in the first place?" You can't deliver your code in the first place unless you have a clean build. You can't deliver your code in the first place unless it has the right unit tests associated with it, et cetera.

Putting that feedback in as early as possible in the process so they get the input.

The other piece about that is there are some things you're going to test from a manual standpoint. UI testing is the worst thing to automate testing around because it's the hardest thing to automate testing around. But last time I checked, that UI is calling a bunch of services. What you can do is automate the testing entirely of all of those services. All of the APIs, all of the back-end systems should have a completely automated test piece, and therefore it's easier to put this whole picture together because all of the pieces need the feedback. They all need the feedback as fast as possible, and then they all need to come together.

You can use UrbanCode Deploy to deploy that thing end to end. I can deploy the Z part and the distributed part and the configuration. I can deploy into a Bluemix environment with OpenStack. I can deploy the whole thing and do the testing and do whatever I need to in a coordinated fashion to get the feedback back right away.

It's all about as fast as possible, all of the different kinds of stages in a form that I can also get user feedback. That's going to be the other point I'm going to make here.

Early in these... Well, this picture doesn't work, but one of the other pictures you had up talked about the different stages of development, and if we look at some of those...

Sanjeev Sharma

This one?

Rosalind Radcliffe

Yes, this one.

If you look at some of those, if I can get it deployed automatically, efficiently into this environment, I can get those QA engineers, the business engineers, the business people looking at the code much sooner, looking at this interface much sooner to validate, not that it's working necessarily, but that it's doing what you want it to do.

Working code, yeah, developers are good at writing working code. But are they good at writing the code that satisfies the business requirement? You need a business person to validate that. The sooner you can get that person into this picture, and if you've got a good pipeline that can deliver the environment into a pre-prod environment on a daily, hourly, take your choice of word, basis, I can get that feedback early, and then I can iterate faster.

Sanjeev Sharma

Thank you.

Let's take the last few minutes as we're going into the home stretch here and talk about the big elephant in the room, and that is cultural inertia. I call it cultural inertia because the larger the organization, the more the inertia. Rosalind and I work for a company which has, at last check, close to 400,000 employees, which is fairly large. I think, Rosalind, on your team, you have people in every continent except Antarctica. Yes?

We tried really hard to hire somebody in Antarctica, but they don't have too many mainframe developers there.

How do you overcome it? I'll give you some examples of cultural inertia. "Oh, this is the way we've always done things here. Oh, it's not broken. Why should I fix it? Oh, not my problem. That's Rosalind's team's problem. Oh, it's the mainframe guys. They're the ones slowing us down." "No, we have an outsourced vendor, and we can't renegotiate the contract, so we can't go any faster."

These are all examples of cultural inertia. Overcoming that requires a lot of change and a lot of investment from senior management.

Can the three of you, in whatever order you want to go, as Rosalind has the mic first, you go first. How do you overcome cultural inertia?

Rosalind Radcliffe

I'll say that I think cultural inertia is worst, and you can correct me if you think I'm wrong, but is worst in the Z space. Because I think we have a set of developers that have literally been working at the same job, doing the same thing for the last 30 years, using the same tools, working on the same application. They have no desire to change, and so therefore it's not possible to change.

I don't know how many developers I walk into, or middle managers, take your choice, in the mainframe space: "You can't do that here because we've always done it this way. It's worked for 30 years." And they're not kidding. It has worked for 30 years. But it's not really working anymore. The business needs to move faster.

How do you get that cultural change? You get that one person, or you get those few people that understand that there's a better way to do it. Or you let too many of them retire and there's no one left, and so you hire... Oh, this isn't the way to do it. But you wait much longer, and you will be doing it that way. You won't have the people left who know it. You'll be hiring new kids, and they'll have to learn it without the old guys like me around, because I am going to retire.

It's important. The way to change this, in many cases, is to bring in the young kids, have them work with the old guys, especially in the mainframe space. They've got the new practices, and the Z guys actually know how to do good development. If you put them side by side, you actually end up with much better solutions in the end.

You get modern practices. You get historical understanding of what the business is. Those Z guys have been around. They understand that business. They understand the business rules. These new kids learn. That's one way, in at least the Z space, to do this cultural change.

One of the key pieces of that is find the right people who want to change. If you can get a few people who see the value and see the benefit and can produce benefit, it spreads like wildfire. Otherwise, well, have fun.

Mark Howell

I echo some of those comments as well. I think you need to create a movement. That's what we did.

There is an element of, we work in quite a siloed organization. We have an operations department that works over there, and the answer, they don't like change. They like stable environments. They don't want change. Developers are cowboys. We don't like to change, but then the business is forcing us to make change.

I think it's bringing all of those guys on the journey with us. For me, it's been a journey of showing people what good looks like, in terms of we can automate this, we can make this easier, we can make it simpler.

From a development community as well, seeding the right people into those teams, again, to show what good looks like in terms of the tooling, the processes. Wouldn't it be great if all this went away for you? Then you're creating that movement, that groundswell of almost, how can I phrase it, the old way's not going to work for us anymore. We don't want to do things like that anymore. We have to start looking forward: how do we develop products and propositions for customers quicker?

Looking across the marketplace and looking across the industry, we are traveling at the speed of light at the moment, and everything changes so quickly.

What we've done is seeded people into teams and to go and show them, "This is our thinking. Come with us on the journey. Tell us what works for you or tell us what does not work for you, because we don't want to build another ivory tower." We have to get in amongst it.

When we have that groundswell, then it takes on a life of its own, and then you have people actively contributing.

One of the things with us as well, and a comment to all of you, my personal view again, is that you need to give people the room to breathe. It's not a side-of-desk activity. It's something they need to take seriously and think about how they can be agents of change themselves and join that movement.

Sanjeev Sharma

Same question.

Gareth Evans

The FINkit business within Monitise is a relatively new business. It's been going for around two years now, so you'd think that we wouldn't have the same problems. But if you have two development teams and one's sat at one side of the room, the other's sat at another side of the room, and they're working on different projects, even though they're kind of working for the same common goal, you create silos. There becomes a bit of a barrier, and this team only really talk to their peers because they do things one way and that team does things another way. So it happens.

One of the things that we looked at from a cultural point of view is how do we ideally stop those barriers, those silos, from forming in the first place? But where they do form, and they inevitably do, and it kind of moves around, it ebbs and flows a bit, really, how do we go about breaking those silos?

At an individual team basis, the first thing we did was we got some Nerf guns in. We did. Because nothing makes two developers talk to each other that normally wouldn't more than being shot in the face during a conference call. You can't respond. It's brilliant.

That's something that just brings a bit of conversation, a bit of banter in the office, that just lightens the mood, and those people will then have a conversation. That was the first thing that worked really well.

The second was we do a site-wide standup every Friday at the same time. Everybody in the office, regardless of what they're working on, whether it's office management, help desk, or development, or something else, everybody stands up and it's like, introduce yourself, these are the new starters in the room. Does anybody have any questions that they just don't know who to ask to go and find out the answer? Those kind of things that report. That gets the conversation going within the office, within the team.

We're still to work out, I think, the cross-site communication, but there's a few things going on there. We've got a team in Cardiff. We have a team in London. You find that communication on Slack, if you're looking at the number of messages, it goes up during the Six Nations. We can't have the Six Nations more frequently, but what we can do is push other things. We can organize events or organize things that happen between sites that allow that communication to happen much faster.

The other one is that we actually got a pool table in the office, and we find that enforcing people to do a weekly pool tournament where they would be playing somebody that they would not normally talk to, so it's all completely randomly generated each week, works really well.

Sanjeev Sharma

Excellent.

Rosalind, I'll hand it back to you because I made that comment about highly distributed teams, because I think that's the exact opposite of... You cannot do Nerf guns between Bangalore and Haifa. How do you handle highly distributed teams? Because you have people in multiple continents, multiple time zones.

Rosalind Radcliffe

I call one of my teams the poster child for geographically dispersed teams. In this case, I'm going to limit it to about 20 people, and the team of about 20 people is located in Perth, Australia; China; West Coast of the US; East Coast of the US; Toronto, Canada; Pornichet, France; Paris, France; England; and I'm missing somebody.

Sanjeev Sharma

Haifa? Bangalore?

Rosalind Radcliffe

I don't have Haifa in that group. I don't have Bangalore in that group, but I know I'm missing some. Whatever. You can get the hint that I'm around-the-world development very simply. That's just one of the teams that I deal with, but that literally is a team that one would've said should've been a two-pizza team, co-located, whatever, because this team works together on a daily basis to do their job.

We work through a combination of tools, and it really is tools and support.

Before you think IBM's absolutely nuts for building a team that's that small, that is that geographically dispersed, I'll tell you I care about skills more than I care about location. I've actually built a team of excellent skills who happen to be located all around the world. They are experts at what they do, and so it works really well because this team of people knows that the person in Australia is excellent at what they're excellent at, and we need them.

This geographically dispersed mess works really well because each person has their own skill. We use a set of tools to collaborate and work together. In our case, we're doing Z development, and we are using RTC, and we can communicate. We can collaborate. There is no question about builds. They just run. We can connect to our systems. We don't have to worry about it.

We do use Slack now. That's a new tool that we added into this team. RTC works better for these guys because that's where they're used to collaborating and communicating and getting all their stuff. But we use tools to be able to do it.

We don't have standups. I'm not going to do that to my team. There is no time that's nice. None. Zero. There's no time that's nice for this world, so we make Australia miserable every time. That's our rule. If it has to be a team meeting, Australia's miserable, and we'll find a time for everybody else, which means 9:00 East Coast time. Sorry, California. Sorry, Perth. Sorry. But it works-ish.

If we have meetings, however, between two sites, they have to pick a time that's not miserable for anyone. If I meet with Australia, it's their day, not mine. So you work out what works for them.

I would never, ever suggest building a team like I did unless you're doing what I did, which is skills. I think that is the real key. If you're doing it because of skills, not because of silliness, it does work because people want to collaborate and want to work together. But you have to have the right tools to do it. The desire to want to work together does a lot.

The other thing I'll say is I do cheat, though, so this part probably shouldn't end up on the recording, but oh well. I bring my Australian developer around the world twice a year, so he gets to stop by other areas. We do get face-to-face collaboration time for him around the world. He gets to see people. We get to have team meetings, team dinners, team whatever, and we get to collaborate twice a year face to face. Makes a huge difference, because everybody's met him, everybody's seen him, everybody gets to deal with this. When you can do that, you do it.

Sanjeev Sharma

Excellent. With that, we are ending up in the last two minutes. Thank you, panel. Thank you so much.

Is there any ending statements? Anything you feel you would like to add as we wrap it up?

Gareth Evans

I think the one thing that we've really taken from the DevOps transition that Monitise certainly has gone through is that DevOps is not a team. DevOps is a culture. If you can get everybody bought into that culture, I'm not going to say it's going to be plain sailing, but relatively speaking, you're going to start seeing change. Once you start seeing change, it motivates you further to progress, really.

Sanjeev Sharma

Excellent. Thanks.

Mark Howell

I'd echo that.

One thing: don't try and boil the ocean, because you will just get nowhere. Have a look at where your biggest pain point is and go and attack it. When you make change there, you give the others around you confidence that change is good. Change can help me and make my day and make my developers happier. Then you go after the next thing and then build the picture up from there, and that's the way to do it.

Again, when we looked at our proposition, we looked at where's the biggest pain point? Let's go and make a real difference there, and then we'll move to the next, move to the next. Don't get hung up on the word DevOps. Look at where on your build cycle and your deployment cycle, where are the areas of focus.

Rosalind Radcliffe

Culture. Culture. Absolutely change the culture. Figure out a way to get people to understand this is a positive thing.

For some teams, it may be, you're wrong. You do have to change. You have to accept that change sometimes does just have to happen, even if the team doesn't want to.

Sorry. Life isn't easy. There are things that have to change. Just because it worked doesn't mean it's working. Business is changing. Today's world is not even yesterday's world. We need to deal with what's happening today and deal with that change. Honestly, once you get past the culture, you get people bought in, then they see it and they go on.

Sanjeev Sharma

Excellent. Thank you, panel. Thank you so much. Let's give them a big hand, guys.

That was phenomenal. I hope somebody was live tweeting. I'm sitting here going, "Man, that's a tweet right there."

I'm sure we'll all be around for the reception, so please come grab us. Rosalind and I will actually be signing books. She's the author of Mainframe to Mobile DevOps for Dummies. I wrote DevOps for Dummies, and you can grab a free copy of the book and have it autographed by us, if you would like it to be lowered in value by my autograph, at 5:30 in the reception area.

Thank you for your time, and thank you, panel, especially, for your time and wisdom.