Mirco Hering

So I'm going to talk about DevOps for systems of record. And I will explain a bit, if you don't know what that term means, and we will get through this.

I had this presentation in Australia earlier in the year, and I had 45 minutes. Now I have 25, so I have to move relatively quickly. But I was promised, because we go into lunch afterwards, that we can have a couple of questions at the end. So we'll cover whatever you're interested in afterwards.

Okay, so just quickly about myself. I'm looking after the Agile and DevOps practice for Asia Pacific for Accenture, which basically means I'm playing in all the big enterprises. Usually you don't get the luxury to play in a pure digital space. I have to deal with systems of record, and that's what I'm going to talk about.

I do blog relatively frequently and tweet most when I'm at conferences. But through that, you can ask me questions afterwards if there's anything else that you have interest, more I'd like to share.

So what will you learn from the session today?

I will quickly start why I think systems of record and commercial off-the-shelf products are important for your DevOps journey. There's an argument to be made that you shouldn't consider these, but I think it is important to do that.

Three simple steps that you can follow when you're working with those systems. Talking a bit about how to handle the code, and yes, those systems do have code. And then I go into my real-life experience, a bit of the benefit that we got with some of the implementations I've done, but also the challenges. Some of the things that I, quite honestly, haven't solved yet. And obviously interested in, if any of you have solved some of these problems, I'm more than happy to leverage.

So systems of record, what is that? Very often when I talk to people, I call it legacy. Legacy, some people find offensive, so if you want to be kind, you can call it enterprise applications. But it's basically the systems that don't play well. The systems that just don't behave like the Java application that you've heard a lot about in most of the talks. The things where you can go on the internet, and you can figure out how to do continuous integration with them. Those systems, you usually don't get that answer.

And if you want another shorthand, think about the Siebels, the SAPs, the mainframes of the world. Although we've heard this morning that mainframe is actually apparently quite good in this space. In my experience with mainframe, when I came from uni as a Java developer, my very first job was to code some COBOL. I was pleasantly surprised that it was just custom code. No matter what people told you, it is just custom code.

So if this is your world, does that look familiar to some of you? Is that what you see in your organizations? The green screen, working with database-based systems, Siebel, Salesforce. That's a reality.

And the reason why I think that you need to consider DevOps for those systems is the amount of money and effort that you spend on these. So even if they will never get to the point that you deploy them 100 times a day into production, the amount of effort that you spend there, any benefit that you can get will make a huge difference. And I will show you some of the numbers, and it's just astonishing, I think, what you can do.

But there's another aspect. And the other aspect is that those systems still need to work with your digital applications. So to me, the analogy that you see here, those two gears, is a really good one. You have this fairly fast-moving small gear and the slightly slower large gear, but they integrate at some stage.

And if you think about your banking application that you have on your mobile, ultimately you can get an update to that mobile app every couple of days. But what is required is still a mainframe server probably at the back end that will give you your bank account statement. So that's the way that this works. And for new features to really come out, you will have to consider both sides.

And it gets a bit worse, because what often happens today is that the speed of your delivery of new features is determined by the slowest-moving part. They have these integrated projects that have to go through the whole stack of applications.

As you go down the DevOps path, you will work on disintegrating them or decoupling them so that you can move your front-end system separately from your back-end systems, like the whole thing when we talk about APIs. But a lot of the core functionality still sits in your back end. So whatever you can do to optimize that, to make it faster, and to take cost out of that, and to make it more reliable, you're still going to get a lot of benefits.

So that's the two reasons. One is the scale, the sheer scale that you have for these systems in many organizations, and the other thing is the speed to market even for a digital space.

So my view of DevOps is that it's a direction, not a goal. And that pretty much every one of you and every organization is on that journey, because DevOps is about improving your IT delivery. And it doesn't matter where you start and what you do, all of that is part of the DevOps journey.

Some of them have a bit more path security. They know a bit more about what the next few steps are. Others are kind of stumbling their way through that. But we've always been on that journey, and I've, for my whole career, done this.

And one of the best things that happened to me is that all of a sudden it became sexy. Because when I used to talk to people about configuration management and build automation, no one wanted to listen. And nowadays, I say it's DevOps, and everyone wants to talk to you. So I think that's brilliant.

So what I will focus my talk on is where we've actually been really successful with it. And so that's the configuration management space, the build and deployment automation.

We've done some work in the test automation space, and I will talk about my mostly negative experiences so far. And we are doing some experiments in the provisioning, and can we create a Docker container with Siebel in it? Those kind of things are currently in the experimental stage. But I certainly want to see that we can bring the systems of record into the same mechanism as the other applications.

But is it technology or culture that holds us back? And I think on the first day, one of the keynote speakers said it so nicely: "DevOps is not a technology problem." And I think that's absolutely true.

I was glad that the lady from IBM, I can't remember her name, this morning talked about the mainframe, because one of my earliest experiences with DevOps were working as that COBOL programmer right after uni. And not sure how many of you have done some COBOL coding. A few of you? Yeah. Good.

In COBOL, you have to start coding at a certain column in the text file. And the way that it used to work at the company that I was working for was you upload it to the mainframe, you try to compile, and then it tells you, "Damn, on line 36, you started in the wrong column." And that's a 10-minute process.

So coming fresh from uni, having worked with Eclipse as an IDE for Java for my whole student life, I thought, "That's stupid." So I went to the development manager and said, "Look, I think we should have some IDE that can test for these things before we actually upload it to the mainframe and save us the 10 minutes and the CPU cycles."

And he said, "Mirco, you don't understand. We have been doing COBOL development for 20 years with TextPad. Our developers know how to code this. They don't make these mistakes."

And that conversation was over. There was no investment being made in these kind of tools. And I think that's the culture, the perception of what good looks like that they don't necessarily know. And I think we heard that in the keynote this morning quite nicely that this is different. You can actually do this, and you need to explain to people what good looks like.

There's a technical term for this, and it's called Dunning-Kruger effect. If you don't really know what good looks like, you think you're pretty good at it. And there's quite a few examples here that prove that point.

And I do think that is a real problem that we have in IT. I'm a consultant. That means I spend a lot of time going into organizations and doing some kind of assessments, and I do have a very ambiguous relationship to maturity models.

But one of the things that I encounter is you go in and you start talking about some of the core DevOps practices. And as Jeff said yesterday, one of the hardest one, in my view, is continuous integration. I do think it's really hard to get that right.

However, you wouldn't get that feeling when you go into organizations because many claim that they do continuous integration. And the conversation usually goes like this.

"Do you do continuous integration here?"

"Yes, we do."

Because we are curious, we ask a bit further.

"So what does it mean to you?"

"Well, we have a Jenkins continuous integration server."

"Okay. What do you do with that?"

"Well, it sits on the server and we run it every Sunday."

And you're like, "Yeah, that's not strictly speaking continuous integration." Installing the server doesn't make it continuous integration.

And so I was struggling with that and had used lots of different maturity models. And one day on my way to work, I remembered something that I spent a lot of time at uni with. And I thought, "Man, this has real application in my day-to-day job." And what that was, was playing Civilization.

Because in Civilization, they have this technology tree that means that if you've discovered certain practices, then you can move on and build more advanced practices. And so what we've done is we've built a technical dependency tree that puts all these different technical practices together. It shows you that to do continuous integration, you have to have automated builds. You have to have automated tests. You have to build a check-in, yada, yada, yada. It goes all the way down to continuous delivery with all the practices that we associate with that.

For each of these practices, we have a definition, we have metrics, we have measures of progress, and we use that to educate people on what good looks like. And that's one of the ways that we use to overcome this kind of cultural barrier that a lot of these systems of records have. Because you can actually get to quite a significant piece of detail, and it's a self-learning tool. It doesn't require me to be there. But as so often is the case, if you have a technical coach, you're going to be much faster.

So this is my simplified view of the DevOps software supply chain, and that applies to anything: systems of record, custom development. It basically goes from SCM, where you have your ingredients, everything that you require to make your software. That's your source code, your test scripts, your infrastructure as code if you do have it.

Then you need to figure out your IKEA manual. So how do you make out of all these pieces the application at the end, or the application package? And as is so often the case with IKEA manuals, you have a couple of missing pieces that you still need to find. But that's the idea behind this.

And then from there on, you have to understand the path to production. How do you get this into operations and make this environment agnostic and so forth? And those rules still apply for systems of record.

This is a slightly more detailed reference model. The only reason why I have this in here is to show you that we are just talking about a relatively small part of the overall ecosystem, but the whole ecosystem needs to come together, from environment provisioning at the right bottom to development analytics on the left bottom, where you take your data exhaust from all your different systems and use it to gain information about the health of your delivery.

So the three steps that I encourage everyone to follow is look under the hood of your system, find those moving parts. And it's not necessarily easy.

When you look at Siebel or a system like Siebel, it's not easy to understand what is the actual source code because a lot of this is kind of hidden. So you need to figure out ways of getting it out of those systems. In the Siebel case, a lot of this is actually XML. And XML you can treat like proper source code, similar to for Pega and Salesforce. It's kind of a Java derivative. So you can usually find these moving parts somewhere. And you need to bring them into the same rigor as you would for any other source code.

When you recreate the IKEA manual, well, you have to be a bit creative because some of these tools require you to call them APIs to actually do things that are expected to be done manually by clicking a button, and you need to find ways of doing that. We are working with a lot of the different vendors to find APIs for all of those steps that would otherwise be manual.

And annoyingly, and I will tell you that later in the list of my challenges, not everything is automatable at the moment. A lot of these large systems of records vendors haven't really caught on to opening up their systems for full automation. They do believe that you would run a pure Oracle shop or a pure SAP shop or a pure whatever shop, and they don't allow you to integrate with everything else.

And then you need to understand the path to production, which again, for systems of record, is often different because you can't afford to have production-sized systems. So you need to find scaled-down versions. You need to figure out what variability you allow and what not. And you really need to understand this to make the right choices.

And then the quote that I like is, "Only tools buy a tool to solve this problem." The tool themselves doesn't make much of a difference. I've worked with lots of different tools, and they all have their benefits and challenges, but at the end of the day, it's the principles that matter. And you can probably make it work with any tool. As a consultant, when I go into companies, I often have to work with the tools they have anyway. But that's not necessarily holding you back.

Why is COTS software different? So I already mentioned that it's kind of hard to get to that source code, and that's one of the problems.

Another problem is that people who work with those systems don't often consider themselves to be developers. So when you talk to them about merging, when you talk to them about general development practice like unit testing, that's not something they are necessarily familiar with. They assume that it doesn't apply to them.

The way that you build these systems are usually their application architecture is relatively monolithic. It's not easy to get it to a modular level. You have to deal with a lot more overlap when you try to build different features that people have to touch the same piece of code or the same modules in the application.

The performance is different. When I compile Siebel at one of my clients, it takes eight hours. How am I going to do this in continuous integration? How do I do trunk development when I have 500 developers on this one system that creates one binary out of 40,000 files?

So you have to think differently about those problems, and you need to, unfortunately, find some branching strategies and other things that allow you to deal with that.

Now that you got the code, so if you're actually getting to that level that you have a source code available, what you need to do is you need to get it out of those systems and put it in a proper SCM. Most of those systems of record provide their own mechanism for version control, but it usually doesn't integrate with anything else. It doesn't have proper baselining. It doesn't support merging. So you're going to really struggle if you keep it in there. So take it out, put it in Git, and you're going to be much better off.

Tightly integrated with your IDE. One of the biggest challenges in DevOps, in my view, when you introduce these practices is that it becomes more difficult initially for developers. And so if I ask someone to make a change in Siebel and then also check it into Git, they're not going to do that. They will try, but they will forget once in a while. So integrate these things that it happens automatically. Make sure that that supply chain is integrated and that you have a closed-loop system.

And then, unfortunately, you have to solve for merges. One of the biggest benefits that we got out of one of the recent projects I did was looking at the structure of source code and identifying what of that is actual payload and what of that is useless metadata. And using a merge tool that allows you to identify those two different areas and automate the merge where you only have metadata differences, it made a huge difference.

So you need to think about those things differently than what you would do with a normal Java code. And you need to just go that level deeper, and I encourage you all to understand what these systems do, understand the application architecture behind it.

So let's put some numbers to this. This is one of the projects that I've done. We moved from four releases a year to 11 releases a year. That was tick the box. We're now doing monthly releases.

The unfortunate truth was that each of these releases still took six months. We just had more releases. And I do see that a lot in organizations, to be honest. What that means is you all of a sudden have a lot more open code branches, and the retrofitting of those code branches, bringing them back together, the effort that goes into that are quite significant.

In our case, it was a two-week-long exercise that we had to choose when to do, and the effort was very significant. So by automating a whole bunch of different practices over that supply chain, the integration with configuration management, the build automation, deployment automation, and the merge automation allowed us to very significantly reduce the effort.

The two-week-long exercise is now three days long. The overall cost of that process is about halved. And the scale that we're talking about, we're taking 3,300 days out of the annual budget. You can multiply that with your daily rate that you're using in your organization, and you will see that it's real money.

And this is money that we can reinvest now in some of these experiments that I mentioned, that we can actually spend on building functionality rather than-- But it's really waste. Because none of these activities add any functional value to the application. They're just to get them into production.

So all is good. Have I solved all problems? I don't.

So these are the things that I keep experimenting with, and I don't yet have the answers. So hopefully next year, I'm going to be back and give you a few more answers.

But unit test automation for some of these applications is pretty hard. In Siebel, there's a technology, Open UI, where you can use something like Selenium, and that's good. Great. If you use the older system, it's really hard to find a way. You can try to do some stuff with direct API calls, but it's quite tricky. We have found things that work, but are too expensive.

There's unsupported activity that I mentioned for which there's no APIs. Continues to be a real problem and continues to be something that I don't see necessarily all vendors jumping in to make sure that they're helping me. So some of the stuff I just have to accept. In our specific case, fortunately enough, these are activity that happens once or twice a year, and I'm going to just bite that bullet at the moment.

Performance, I mentioned. Compiling, deploying these things take longer than I would like to see.

Configuration management skills. We ran configuration management training for all these used-to-be configurators that want to become developers. And that's a good part of that journey.

Mentioned common objects and operations. When I say I want to do these things automatically and put it into production for a system of record, I get even worse conversations than you get for other systems, right? You don't do that to mainframe. You don't do that to Siebel. But we can. And bringing these guys on board has been a real challenge.

And I recently went to the operations unit of one of my clients, and we sat down, and he said, "I get what you want to do, Mirco, but I'm not going to allow you to just deploy more stuff into production that doesn't work."

And you're like, "Yeah, well, if you help me, we will make it work." But they don't see it that way. They just see me coming in, and I'm coming from a development side, so we have that very true dev and ops conversation.

So top five takeaways. I do think that many organizations will get benefits from looking at DevOps practices for these complex systems: sheer scale, or if you're looking at the speed of delivering functionality.

Three simple steps. They're the same steps as for custom development, but here you just have to do it a bit differently. Find the code, create the IKEA manual, and understand the path to production.

Treat the systems of record code just like any other code. Don't let them tell you that it's different. It is not different. Somewhere there is a text file. And you have to be creative. Sometimes we have created tools that extract database configurations and put it in a text file or in a custom-defined XML file if you have to. So find creative ways of doing that.

And then the benefits are just really, really significant. It changes the way your organization talks about these systems. So all of a sudden, you can still use them for some differentiation, and they're not just a system that are kind of dragging along with the digital guys.

What am I still struggling with? And this is not necessary to do with systems of record. But not a lot of people talk about how do you successfully partner in organizations. Obviously, the organizations that I work with all have system integrators and vendors, and how do we make sure that the culture, the DevOps culture in my organization and the DevOps culture in my client's organization actually aligns?

How do we align incentives? The contracts that I'm working with, DevOps works despite those contracts, not because of those contracts. And I think that's a real problem to solve.

How do we shift culture at scale? The development organizations that I've been working with have all been around 500 to 1,000 people. And 500 to 1,000 mainframe, Siebel, SAP developers, to shift them, it's quite hard.

How do you safeguard against regression? One of the worst experiences in my life is when I go back to a client a few years later, and we're basically doing the same thing again. And it just happens way too often.

And it's not because people make bad choices. It's because people make what they consider to be the right choice in their paradigm. And unless you have established that culture and the right idea and the right, "What does good look like?" they will not make the right choice.

And then last but not least, how do you measure the benefits? In our case, it was really simple because we had such a long process that we just introduced by doing more releases. But if you don't have it, if you don't have that really standout pain point, how do you still measure benefits?

How do you put numbers and dollar figures to speed? But if someone can give me that answer, brilliant. I think you're onto something really big in the industry. But I don't think anyone has solved that problem yet. But I still keep trying. I keep writing business cases, and at some stage, I will have a good answer.

Thank you very much.