Chris Kernaghan

My name is Chris Kernaghan, and today I'm going to talk to you about doing DevOps in SAP landscapes.

So, a little bit about me. I am a husband and father. I'm this T-shirt today. My daughter is wearing the other one, which is "apprentice," so I promised I would wear this for her.

I'm also an SAP Mentor, so I exist with a group of people who represent the top 1% of influencers within the SAP ecosystem.

I work as the regional COE lead for database and technology for Bluefin Solutions as part of Mindtree. And I'm really, really lucky that my job actually fulfills all my passions, because the SAP technology platforms are actually quite diverse, so it's actually really cool.

My journey in DevOps started five years ago, and it's all the fault of a good friend of mine called Simon McCartney. And what I'd really, really like is, his Twitter handle's down there, and if somebody would tweet him and say, "It's all your fault that I'm listening to some crazy Irish dude talking about SAP and DevOps," that would be awesome, because it's 9:00 his time and he would get a kick out of it.

But I spent four years out of the last five trying to convince people in the SAP ecosystem that I wasn't mad, and it's only been in the last year that people have actually started to come back and go, "You may have been right." Which is a nice feeling, but it still took four years to get there.

So here's the background of my ecosystem. SAP is probably the biggest software company that people haven't heard of. I think it's number three in the world in terms of software development. It has 85,000 employees and a market capitalization of 116 billion euros. And it's famous for, or not so famous for, running the world.

Depending on the metrics, or who you listen to, 75% to 80% of the world's financial transactions touch an SAP system somewhere in their life cycle. 95% of the world's automotive manufacturers, similar of oil companies. The one I love is 85% of the world's brewers all run SAP, which is great, and 80% of the world's chocolate manufacturers as well.

The company that I work for, Bluefin and Mindtree: Bluefin was started in 2002, Mindtree in 1999. Bluefin was acquired by Mindtree in August 2015. Bluefin is famous for pushing the boundaries of SAP technology. So when SAP want to do something really challenging as a demo, or a keynote, or with a partner, they come and speak to us, and we've been doing that great work for ages. And Mindtree is actually famous for running the Azure platform for Microsoft.

So this is the opposite side to a presentation I gave about a month ago in our industry conference, and I was bringing DevOps again to the SAP community and trying to teach them and show them what DevOps is, why it's important, why it's really cool, and what they can gain from adopting it.

What I'm here today to ask all you nice people is: how do we introduce DevOps to SAP teams and get them started? Because I've seen so many of the names on people's badges, and these are all SAP customers. So I've talked to people, Disney, for example, or NetApp, or Lockheed Martin. These companies run SAP. These SAP systems, they run finance, they run HR, they run warehouse management, production planning, sales, and distribution from these systems, but they don't know about this ecosystem.

Again, how do we introduce DevOps to our SAP teams and get them started? Because we're starting this journey from practically zero.

So, is DevOps possible in SAP?

DevOps in SAP is hard. I think we all know that DevOps is hard. We've all heard many reasons why DevOps is hard. Culture's a big one. But because the SAP ecosystem's really starting from scratch, I'm going to put it out there and say DevOps in SAP is hard.

Over the last four years, I've spent a lot of time with customers trying to bring them along, and I've distilled it into four messages, four key areas that really challenge them a lot. The first one's culture. I'm not going to labor culture too much, because I think we all know that culture is really, really hard. But I do talk to them about what is culture, and how does culture enable DevOps, and why tools are not the be-all and end-all of DevOps.

CI/CD pipelines: what is a CI/CD pipeline, and why is it important? How does it increase the velocity of change without increasing risk?

Automation: what do we mean by automation? What can we automate? And why it doesn't necessarily mean that lots of people are going to lose their jobs.

And measurement: what do we measure? Why do we measure it? And how do we use those feedback loops to make our processes better, to make our software better, to make our lives better, improve the quality of our lives?

So why is DevOps so hard in SAP landscape?

Well, SAP isn't just one technology. It's actually six distinct technology stacks all packaged as coming from one company.

We have ABAP, which is our proprietary platform. It's a 4GL language. It's a business process language.

We have things like Business Objects, which is our reporting and consumption layer, or one of them, again.

We have HANA, which is our in-memory database platform that encompasses a columnar in-memory database, a web server, data science, predictive algorithms, all those great things.

We have a J2EE-compliant stack called NetWeaver Java, which is really annoying because you can't run anything on it except SAP applications.

We've embraced JavaScript for our UI5 front end. And we also have a cloud platform, which is a Java platform.

We also have multiple UI and UXs, our user experience. We have the proprietary SAP GUI. Hands up, who has actually entered travel and expenses on an SAP system? Anybody? You know that user experience is not great. It's not great.

We have Office tools like Business Explorer, Analysis for Office for reporting. We have a consumption layer called Web UI. We use HTML5, and then we have a set of design guidelines called Fiori.

We have so many different moving parts. We have different application architectures, and we also don't have a single unified development toolset, which makes stuff really hard.

Although, I was in the mainframe session, and I was listening to the nice lady from IBM, and what was actually really interesting was they didn't unify their development toolset. What they did was they provided a tool that would actually do the continuous integration and continuous delivery pipeline instead and linked it into all those development platforms, which was actually really smart.

But those are all reasons that lots of people, lots of different places in technology, have issues with. But why is it actually so hard in SAP?

Well, SAP's special. We're special. We're snowflakes.

Normally, we don't sit under the direct remit of IT. We consume IT services, but we don't sit underneath IT quite often.

A lot of my customers have multiple vendors. So where other parts of IT through technology have problems to scale and increase reliability, SAP threw people. You've heard of business process outsourcing and things like that. So it's not unusual for very large companies to have multiple vendors serving and supporting their SAP technology.

As an example, I work for a customer, they have HP looking after the infrastructure that sits in the data center. They have Mphasis supporting the technical runtime of the application and the databases. And then they have, I don't know, IBM supporting the application. So it's really difficult to get these people to align on the values and start to adopt DevOps processes, because there could be commercial implications to each of those different vendors.

The next two are sort of aligned in the fact SAP created a concept called NetWeaver Certified. The problem with NetWeaver Certified is it keeps resolving problems that are already fixed.

So if we take monitoring, for example, or deployment. Within monitoring, we have a great tool called Solution Manager. Fabulous tool. Really like it. Sorry. Pain to implement, but really, really cool. And one of the things that it does is it monitors the system and collates all this information. We have Tivoli as well, for example, from IBM. But we also have open source technologies that we can use, like Nagios or Sensu, which have plugins for SAP.

Many of the companies that I work with, other parts of the business use Nagios and Sensu and things like that, but the SAP people won't let them use the tools on their SAP landscapes. Why? It's not NetWeaver Certified.

So culture. What do we need to learn about culture in the SAP ecosystem?

We need to change our attitude to failure. Now, when we have multiple vendors all incentivized with commercial agreements that punish failure, it's really difficult to get them to change their mindset around failure.

We also over-engineer solutions to try and beat off the mean time between failure. What we need to look at is mean time to recovery.

We saw it in one of the first presentations on Monday was the three-tier support model that is absolutely ingrained in SAP. And we don't use things like swarming and things like that, push the ability to solve the problem closer actually to the problem. We tend to move back up the stack.

In terms of collaboration, many of these organizations and outsourcers, because SAP support is off to one side and not within IT, we're not truly multidisciplinary, and we're not T-shaped employees. We're T-shaped within the SAP ecosystem, but not really outside of that.

Top-down permission. Again, many of the companies that I've talked about get the idea of DevOps and why they want to do it and why it's important, and then the people at the bottom get it as well. But the squishy layer in the middle, those managers who either on the customer side are in charge of the SAP landscape and measured against business SLAs, and the multiple vendors that they're working with, and the commercial agreements, and the SLAs that they're working to, it's really, really difficult to change this culture. So it is.

And then value alignment. Again, when you have multiple vendors pulling in different directions, it's really hard to get them to align to the values of what the company, what the customer is trying to get to.

In terms of tools, since we don't have a consistent set of tools across the different parts of SAP... I work in SAP landscapes, and as a technical consultant, I'm working with HANA as a database, I'm working with the ABAP stack as a business transaction layer, I'm working with Business Objects as a reporting layer, and I'm working with mobile guys working on UI5 and mobile deployments. And there's no unified toolset across each of those people. So it's really difficult for me to find a common language with these people.

You really have to work hard, and I can do that because I'm usually with these people. But when I'm working with people who are 4,000 or 5,000 miles away, getting that connection, getting that understanding is really, really difficult.

And then community. We have a really vibrant community. We have a fabulous community. We're really, really strong. But we don't share enough. And we do have great empathy.

So moving on to CI/CD pipelines.

Continuous integration, continuous delivery in SAP is hard.

That's wrong. That statement's factually incorrect.

This is what it should be: continuous integration and delivery in ABAP is hard.

Because we come across so many technology stacks, saying "in SAP is hard" is wrong. UI5 is JavaScript. You can do continuous integration really easily. Well, really easy, if you know what I mean, compared to ABAP. Anything that I say is easy, I'm comparing it to ABAP. So just bear with me.

But it is hard. When you look at the properties of CI/CD, you can see on the table there, what appears on the far right more than anything else? ABAP.

The unit test tool that we have in ABAP sucks. Really sucks.

The regular merge to trunk sucks, because branch-based development is really expensive in ABAP, and I'll go through that in a minute.

Automated testing in ABAP is really difficult as well because of the proprietary nature of the SAP GUI technology.

So what are the properties? One of the great things is we have a single code line in ABAP, so everything that you do in development should go right the way through to production.

We don't tend to do branch-based development because it's really expensive, because in order to do it, you actually need a whole copy of the SAP system to do that branch-based development.

We don't do local-based development because the IDE and the code repository is in the ABAP system itself. I've got to keep saying ABAP.

And the other thing is, when we save and activate a function, we don't have the concept of feature flags. So once you actually activate the change, it's there, and it's active across your whole landscape.

Tools. This one's really interesting for me because we have had this great ability to deploy change for the last 20, 25 years, and it's automatic. But what's interesting about it is the smallest unit of change when you're doing JavaScript development, for example, is a commit. A Git commit. Or if you're doing SQL development, it's a SQL commit. So it could be one line of code, it could be hundreds of lines of code.

In ABAP, the smallest unit of change that we have is a transport. And that transport could be one object, one line of code, it could be hundreds, it could be thousands. So when we actually look at the size of the unit of change, the larger the unit of change, the greater the risk.

The SAP GUI technology: it's actually really difficult to automate because it's proprietary. It's a proprietary front end. It's not a web front end. It doesn't have an API or anything like that there. It just doesn't work particularly well.

The ABAP stack does have APIs, but when you're starting to look at reports and the running of reports through the GUI, you can't use the API. You have to use the GUI layer.

And people don't use the tools very much. As I said, the ABAP unit test tool, people don't use it enough. So the feedback loop back to SAP to improve the tool doesn't get triggered.

So, languages and in testing, generating test data is really hard. Because the ABAP stack comes across so many areas of the business, HR, finance, production planning, warehouse management, sales and distribution, there are graph relationships in the data model itself. But when you're looking at a data model that encompasses tens of thousands of tables, generating test data in that environment is really hard.

And testing's really hard. There's a reason why we have an entire ecosystem based around testing, and it's not baked into our development process as well.

But how can you do a CI/CD pipeline in SAP?

You can use ABAP Unit, the unit testing tool. As I said, we have a tool called Solution Manager, called component-based testing automation, which will do this whole end-to-end test going right from the back end through the front end into mobile, into reporting layers, and things like that. It's fabulous.

And then within our cloud platform, as I said, we have DevOps tools. Because it's a Java-based and JavaScript-based runtime environment, they have a whole section set aside for Jenkins and Cucumber and things like that, and Selenium. And again, with our new front end in UI5, our JavaScript front end, we have web testing tools as well.

So you can do these things. It's just that people are choosing not to do them, for those reasons I said before.

So automation. Why don't we automate much?

Poor tools. The SAP tools are not built for integration into other automation tools. It integrates really well into Solution Manager, but it doesn't integrate in the other tools.

So when I take, for example, environment provisioning, the SAP way, supported way to build your environment is to use what's known as the Software Provisioning Manager. Now, I can build a Bash script that'll install a Business Objects server, or a NetWeaver Java server, or an ABAP server, but it's not the supported way to actually install the system.

I also can't script that Software Provisioning Manager to any great degree and link that to something like Chef or Puppet. I can't do that particularly easily.

There's a lack of trust. People don't trust the test automation, especially in compliance environments. They would rather be hands on the keyboard actually doing it so that they can verify it.

Scale. We keep trying to boil the ocean, and we're not starting small. Because of the way the SAP system itself cuts across so many business functions, it's really difficult to take an isolated area of the business and test our DevOps processes.

And then job protection. As I said, we outsourced an awful lot of our SAP activities and support to other economies. So there's a lot of people who potentially could be out of work.

But what can we automate? This is stuff that I've automated in the past in varying degrees of complexity.

I've done server builds. I've done server builds in Amazon, in Google, in Azure, and on-premise using VMware. Great fun. Really, really enjoyed it.

Change movement, so the ability to move change through landscapes automatically.

Moving on to things like, in medium complexity, monitoring and alerting.

Moving on to higher-complexity stuff like regression test packs. So using tools like LoadRunner or HP QC, or, again, Solution Manager to automate these things. But it's really hard. So it is.

And then environment provisioning as well.

In terms of measurement, the five W: who, what, where, why, when. Evaluation, so setting thresholds, evaluating security and compliance, evaluating information use. That's stuff we do really well. That's stuff we do really, really well.

What we don't do particularly well is actually analyze that, gaining insight, pattern recognition, seeing: do we see the same problems at the same time every month? Do we have the same problems at month end? Do programs fail at the same time? Those types of things.

And then because we don't do that insight step, it becomes really difficult to actually do conclusions, to actually improve the run of the system. For many people, close enough is good enough, but it's not really. We have to continuously improve.

And again, when we get into this multi-vendor environment, because different organizations are in charge of different parts of the SAP system, it's really difficult for other companies to say to their counterparts in those other organizations, "We've seen this in this part of the system. Can you fix it?" And the other vendor can quite happily go, "No, I've no commercial agreement with you. Why would I want to do that?"

So we have a tool, as I said, Solution Manager, and it's awesome. I've seen so many slides with so many moving parts on CI/CD pipelines: Jenkins, Jira, Git, all those sorts of things, and they're great.

I actually have a single application, a single monolithic stack, that'll monitor my application end to end. It'll do version control for me. It'll do change management for me. It'll do incident management for me. And it's all there in one data model that I can then report from and do that insight and analysis really, really easily and support it much, much more effectively than those 10 or 15 moving parts. It's just a pig to implement.

No, it really is. It is. It sucks. So it is. I love it, but it sucks.

So here's some of the interesting things that I love to measure.

I love tracking developers. I'm an infrastructure guy, an operations guy, and I love monitoring what developers get up to. So when they actually start to deploy code, you look at the return codes. I used to run a competition on one project. The developer who got the lowest total return codes, or most successful changes through in the week, won. And the one who got the worst changes through in the week had to buy cake.

So every Friday, we had cake, and somebody had to buy it. It was awesome. It was great.

We monitor things like the database size, which doesn't seem like a great metric, but when you're working with an in-memory database that's constrained by its hardware, you absolutely have to monitor the database size. Otherwise, you're going to start to run out of memory, and the system's going to crash.

Stuff we monitor in medium complexity: the number of successful test runs. I can monitor that through Solution Manager. The number of high-utilization users, and the number of program errors that we get.

And then in terms of high complexity, I can do some really, really cool stuff about predicting the load of a process. So when you get to quarter end, month end, year end, customers are typically running really, really heavy stuff. I can go back using tools like Solution Manager and actually pick out all the performance information for that time, run some predictive against it, and then predict how much load I'm actually going to need on a customer and actually provision that hardware for them to make sure that they don't run into any issues.

I can also do things like predict peak load times.

So my ask, the purpose of what this was, and I know that it sounds very, very negative, or it has sounded quite negative as I've gone along: we don't do this, we don't do that.

What I actually want to do is to help you all, when you're having conversations within your organization with the special SAP department, and actually understand where they're coming from. To try and help build that common language and give you the ideas and the knowledge, or a brief introduction to the knowledge, of things like the ABAP stack and why it's a challenge.

Most of the SAP customers run ABAP. It's the holdout. It's the one that when people say, "Oh, yeah, we run SAP," they're probably talking about the ABAP stack.

So when you go and speak to these people, your CIO or your CEO says, "Right. We need to do DevOps." And they get everybody in a room, and the SAP manager's sitting there and he's going, "We can't do that." And you can actually go, "Well, actually, do you run UI5?"

"Yes, we do."

"Well, then you can. Do you run HANA?"

"Yes."

"Okay, there's another bit. Do you run ABAP?"

"Yes."

"Okay. We'll leave ABAP for the time being, and we'll concentrate on everything else and make it work."

So what I wanted to do was help you to help them find a common language, common understanding. Help cross-train teams. We are famously myopic. Because we come across such a large swathe of technology, there's so much for us to learn, we don't see what everybody else is doing.

Show us the possibilities. Show us what you have done. Show us what you gained from it. Every slide that I've seen up here over the last number of days that has, "We managed to decrease our error rate by this and increase our code deployment by this," I've taken a picture of it so I can take it into my customers and go, "See? This is what you can do."

And also teach us about culture. Teach us how to share better. Teach us how to do these things. Show us the culture, show us how to embrace failure, and show us that when we say fail fast, we don't mean fail fast in production. We mean fail early in our process and how we can fail early in our process.

And that's me. Thank you very much.