Guilherme Sesterheim

Hello everyone. My name is Guilherme, and I am a DevOps and SRE engineer for SAP inside ProServe on AWS.

Today we are going to talk about a very interesting subject for enterprise users and enterprise professionals. A very large number of enterprises use SAP as probably the backbone software for controlling many processes inside their organizations. Today we are going to talk about how to modernize the installation of one SAP software installation, which has not changed much, from something that can take up to two months down to two hours.

Let's discuss our agenda. We will cover what SAP is. For those watching, please join me on Slack and tell me if you already know what SAP is. I want to know whether the audience is aware of SAP software.

The second topic is that SAP installations have not changed much. Then we will discuss automations in Bash, the automations that are already available out there. Then we will discuss automations with Ansible, which is what this project is all about. We will cover how this automation works, the challenges, which are the most important and most interesting part of the process, the benefits of evolving and modernizing the way we install software in our organizations from more traditional ways to a more DevOps-like approach, and finally the help I am looking for from the community listening to me.

01What Is SAP?

For those who do not know, SAP is probably the fourth or fifth biggest IT company worldwide. They became huge because SAP is the most famous and most effective ERP solutions company. ERP stands for enterprise resource planning, and SAP has software with the same name that controls everything for your company.

If you are a manufacturing company, SAP will control everything from quotations with suppliers, to tracking logistics when you place an order to purchase goods, to receiving them at your branch, putting them in the warehouse, using those goods in your manufacturing or production line, producing whatever you produce, and then selling it. It controls the sales orders too, while still controlling the controller areas, HR areas, and project areas, for example if your manufacturing line demands very specific projects for each item.

SAP is this huge company that controls all of that. When you put SAP inside your company, you will be controlling all your processes with this world-class knowledge, and it always brings a very good benefit for your organizations.

Here is a big number to show how important SAP business is. Using the source from their website, nowadays 77% of the world's transaction revenue touches at least one SAP system. This is huge. Seventy-seven percent of all the world's revenue crosses at least one SAP system. This is very impressive.

02SAP Installations Have Not Changed Much

Now we know what SAP is, but I am going to tell you that SAP installations have not changed much in the past decades. SAP was created in the 1980s, if I am not mistaken. They became pretty popular in the 1990s and also in the 2000s. But SAP installations did not change much.

Of course, they are much easier than they used to be. But still, when you are installing SAP in your enterprise or company, you are still going to receive media files from SAP, and you will have to take several steps to prepare your operating system and then install SAP. You have to pick between Red Hat or SUSE for your operating system. Then you have to create several folders, connect your instance to the network and probably a shared file system to control your data, and pick the database you want to use with SAP.

It is an extensive amount of work to install SAP in your organization. That is not fast, and in many cases it can take up to two months to install your SAP landscape in your organization.

This is pretty much how an SAP installation looks in AWS. This is the average; there can be differences from company to company. As I mentioned, the process of installing that can take up to two months. Documentation for just one of the SAP systems can be about 30 pages. This is very big, and these are already the easy documentations to approach.

In a typical SAP landscape, we will have at least four servers if you are going for high availability: the database in high availability, the ASCS communication server, the PAS or primary application server where users actually connect, and the AAS or additional application server, which makes the SAP installation highly available. You also connect to Amazon EFS or Amazon S3 to save your data while in transit or at rest for use with your SAP environment.

Still talking about SAP installations, they are very error-prone. If you are doing an SAP installation manually, you have the problem of not having things in code. You will never repeat exactly the same process. It is very error-prone. You are centralizing knowledge in just a few folks' heads, typically the SAP Basis team.

It cannot be repeated because you do not have things in code. Every time you do a new installation of SAP software, you are using another new two months of work. That takes a lot of work from a very specialized team.

Since you do not have things in code, you cannot audit what has been done. Of course, you can log into the instances and see what is sitting there, but you cannot understand the process that led the software to be there in the way it is.

From the experience I have acquired, it is usually very poorly documented. If you are trying to follow a step-by-step guideline, unless it is for operational day-by-day operations, you probably have very poorly documented assets. Then it is very hard to do, once again, the same installation you did a couple of months ago.

03Automations With Bash

Let's talk about some automations that are already available out there. Mainly the SAP Basis people, which is the role that typically places the installation for us, come from an IT operations background. They are very familiar with Bash and other higher-level languages like Python.

But Bash is still a very low-level language. To build very good, reliable code in Bash is going to take a lot of time. You will be using Bash basically for creating the folder structure. There are several folders you have to prepare on the system to have SAP. You will use Bash for package managers. There are several packages you have to install with Yum, APT, or Zypper, depending on the OS. You have to configure hostnames, and you also have to configure the hosts between all those instances.

These are some automations already available, but they are very small compared to the size of an SAP software installation.

Still on Bash, the actual SAP software installation is left behind. I just mentioned prep activities that have to be done on the operating system, but we are still not tackling the installation itself. The installation is still done manually, either via an interface SAP has or via your terminal.

Even with those automations, the installation is still very error-prone because Bash is not a language designed for being self-healing or solving issues by itself. Bash is too low-level, so it demands a lot of work to achieve small steps in automation, and we are not using the right tool for the right job. Bash is something you really use for very specific needs, for example calling another language. That is where this project comes in.

04Automating With Ansible

This is the internal project we have been developing within AWS ProServe, and it was released in early 2022. There are a couple of customers already using it in production. I left the URLs here in case you want to take a look at how it works.

Basically, we automated the whole process of installing SAP from the infrastructure part, using Terraform for infrastructure creation on AWS, all the way until Ansible SSHes into the instance and does all the prep work: changing hostnames, doing mappings in /etc/hosts, package manager work, preparing folders, logs, and several other activities that used to be done manually. Ansible prepares all of that and finally triggers the installation of our SAP software within those instances.

The pipeline at the bottom of my screen is a Jenkins pipeline showing the complete step-by-step process. If you look at the fourth bullet, there is create environment. That uses Terraform to create the infrastructure on AWS. Install HANA and install ASCS/ERS are the first two SAP software installations. HANA is the database, and ASCS/ERS are part of the communication server. The next one is PAS, the primary application server, and then we are done.

This takes around something less than two hours to complete, and this has been proven very successful and very easy to repeat when we apply it.

05How Does It Work?

There is an open source repository on GitHub under AWS Samples. You will find the link in the blog post; I shared the link previously.

The user initiates a local agent on their own local instance, fills in parameters like where the SAP media files are in an S3 bucket, and fills in other SAP-specific parameters: whether we want the installation to be high availability, what hostnames we want, the SID, which is the SAP ID for each instance, instance numbers, and the size of each instance, such as the type of Amazon EC2 instance for memory and CPU configurations.

The user also fills in other infrastructure specifics like credentials, memory configurations, the amount of EBS volumes, and which EFS will be used for that installation. You fill this in and give it to the solution as parameters at the very beginning.

Once you are done, you trigger the installation, wait about two hours, and then this is the output: one beautiful SAP landscape installed with SAP HANA on high availability, ASCS and ERS on high availability, and PAS, which is the software your users connect to.

I am showing two additional images for folks watching who are used to SAP software. This is basically the health check we do at the end of an SAP installation to check if the software is actually installed properly. This is just for the servers of HANA and the primary ASCS. You can see that all the listed processes are green, which means one SAP software is running on those instances.

06Challenges Throughout the Process

We have automated a lot, and we have covered automating all the activities that could take up to two months for a large environment down to two hours. That is a big achievement, but it was not easy. It took us some months of work. I want to share the two main challenges we had throughout the process that were interesting for understanding how they impacted the installation and how to overcome them.

SAP has its own libraries and packages that you use to interact with the SAP system on the terminal and command line. Examples are HDB for applying maintenance to HANA, hdbnsutil for interacting with HANA, and hdbuserstore for creating users for your installation. During the installation, you have to create a couple of users for post-installation activities, configuring your SAP software before releasing it to your users.

One main challenge was how to make the packages and libraries visible on the path for Ansible to interact with them while calling regular shell commands. When you SSH into an SAP instance, you have to switch user to what we call the SID ADM user: your SID user, the SAP ID, plus ADM for administrator. You switch to that user so those libraries become available on your path.

Unfortunately, in Ansible that does not work straight like that. The team had to find a way to import those libraries into the path so the Ansible user could find them. But just importing them was not enough. We had to switch users from task to task. That took a lot of time because some libraries are not available at some points in time. We had to source all those variables inside the path. It can seem trivial, but on SAP, since you are always switching users, it took us a lot of time.

The main challenge was that the regular Bash script SAP uses to source the libraries into the path had one or two mentions of tset or reset commands. This is a command that demands user interaction on the terminal. When we triggered that command using Jenkins and Ansible, it would just hang the installation.

That took a lot of conversations with SAP people and understanding more about how each step inside that source command worked. There are several steps inside that source file. We had to understand each command and what it does, what it impacts in the way we interact with SAP software, and realize that it was safe to omit or hide these tset and reset commands from those files. Then we could source our file and interact with SAP in the proper way.

07Benefits

We automated all of that, and I wanted to bring some benefits for folks working in SAP environments who have not thought much about automating all of this, or who are still working on projects for that.

The benefits around code are that Ansible is a very high-level language, different from Bash. Ansible is a language where you describe the states you want the OS to achieve, and under the hood Ansible does whatever it takes to achieve that state. Ansible can retry your commands, and on SAP we have to use that several times because some commands might fail the first or second time you try to apply them. Ansible helps with that, which is one less headache while installing SAP.

We can put things into code, and we will not forget what we did a couple of minutes ago or a couple of months ago. We also have intelligent checks over parameters included as inputs. If you are using this solution, several of the parameters I mentioned are checked on AWS: whether your bucket exists, whether your EFS exists, and whether you entered a valid password for Jenkins to use as the default password on your SAP installation.

You also maintain a single source for repeatable portions of the code for different SAP products. Whether you are installing SAP HANA, ASCS, ERS, or PAS, you have a single source of code that you can repeat and automate.

Think about creating sandbox environments for your SAP users to test new features from SAP. You can automate all of that, put it inside Jenkins or whatever CI/CD tool you are using, and your users will have it much easier to train and get up to speed with what is out there for them to know about SAP.

With Ansible there are also benefits for the organization. You create a repeatable process. It makes it easier to use once you have a repeatable process. You just play that same process and tweak things when they change. You are not rewriting everything from scratch every time you have to do an SAP installation.

It dramatically reduces errors in the process. Since we have everything in code, once we have tested all of that and keep the same versions for OS and packages, it will work very smoothly compared to doing all the work manually.

It will dramatically reduce the specific need for specific knowledge from special people who know. The SAP Basis team will be less required for these activities and will be able to focus their knowledge more on complex tasks involved in the SAP environment. They can ask for help from other teams, such as L1 teams, to perform tasks like installing SAP software.

It also allows audit. Once you have everything in code, your code can be audited, and it also serves as documentation. You know what is there because you have the code that is doing whatever is there.

It also increases traceability in IT operations. Once you automate all of that, you have logs for whatever Ansible did on your behalf, and you increase your traceability.

08The Help I Am Looking For

This is what I had to share with everyone today, and here is the help I am looking for. This is a very common SAP use case. When you have SAP software installed, it is very difficult to switch the AMI, switch the base OS image, and update it to a new one. These are very common policies in companies.

If you have already played with something like this or automated something like this, let's catch up and discuss how we can switch the base OS or upgrade the OS version without impacting SAP, easily, avoiding errors and avoiding more disruptions to customers' businesses. This is something pretty interesting we have been working on.

Thank you for watching the session. This was DevOps for SAP, going down from two months to two hours of installation. Thank you very much. Please get in touch about the presentation if you have any questions. I will be here on Slack answering any questions that come with the presentation. I have also shared, once again, the URL for everyone to look at the blog post that explains how this whole solution works. Thank you very much.