Chris Boorman

This is the Automic sponsor session. And what we would like to do today is talk about how we have transformed our own engineering organization from what I would consider a traditional, ITIL-orientated engineering organization to a very dynamic, agile, continuous delivery model of delivering our own software.

My name's Chris Boorman. I'm the Chief Marketing Officer, and I'm joined by Josef Puchinger and Christopher Hejl, who are both from our engineering organization over in Vienna. Thanks, guys, for joining us as far as today is concerned.

They are going to do the majority of the speaking, because what I'd like to do is have a little bit of a dialogue with them and ask them some questions about the journey they've gone on and how they've got to this world where they are now delivering production software in two-weekly sprints with zero downtime, versus the old world, which was every six months there's a new release comes out and so forth. So that's the journey they've been on and Automic's been on.

But before we did that, I'll start with a little bit of a, "Hey, I'm the CMO." So for those of you who are unfamiliar with who Automic are, I thought I'd start with a little bit of a brief introduction about Automic, just to set the scene.

We are a software company. We focus entirely on business automation. The entire reason for the existence of our company is to help our clients automate processes in order to drive efficiency, in order to drive agility. That's what we do.

We basically were formed about 20 years ago. We operate across the world. We have about 600 employees around the world, operations in the U.S., across Europe, Asia Pacific, and we work with about 2,700 customers. And the key thing that we deal with is trying to firstly focus on automation.

We offer three portfolios. One is workload automation, another is service orchestration, and a third area is release automation. And rather than go through and just give you a bunch of logos, a little bit of background in terms of ourselves.

About 2,500, 2,700 enterprise customers use our software. When you wake up in the morning in your bedroom and you look out with your bleary eyes, maybe you put your shoes on from Payless, you pick up your phone from Carphone Warehouse, you pick up a book from Barnes & Noble. You look at your holiday that you book through TUI, or you draw back the curtains from eBay. These are all organizations that rely on Automic from an automation perspective.

When you go into the kitchen in the morning to get your orange juice, maybe you get it from Ocean Spray, or you use your cooker from Electrolux or Bosch. Your drinks, your food in the fridge, these are all organizations that rely on Automic from an automation perspective.

For those of you who came here from abroad, maybe Gene Kim or others came over via planes from Boeing Corporation or Airbus or across Europe. Maybe you came by car from Toyota, Porsche, BMW, Fiat, Mercedes. By train, or if you came across from mainland Europe, people still do that, Brittany Ferries. Again, customers who all rely on Automic.

From a financial institution point of view, we have about 750 financial institutions that leverage and use Automic software. A whole range of them here who are relying on Automic providing workload, release automation, or service orchestration across their environment.

Go and get some food. Maybe you go to McDonald's or get a drink from Carlsberg or go to Carrefour to pick up some food, Costco, 7-Eleven. Again, these are all the sorts of organizations who use and rely upon Automic.

And finally, for the fashionistas in the room, maybe Chanel, L'Oreal, Saks Fifth Avenue, Adidas organization, Polo, Kodak. An organization like Adidas uses us to basically replenish all of their stock in their shops. So if you go into their shop and buy a pair of training shoes, immediately you've done that, the entire manufacturing process is restocking that environment. That's the sort of world in which Automic works in.

And what is obvious, I think, to everybody is that we are in a massive period of change. Our view is that every industry is being disrupted. Our view is that if you are not moving faster, then you are going to die. You are basically going to suffer from those people around you and those companies around you who move faster.

And so what we do is our goal is to help you, help our clients, drive agility. And DevOps and the whole world of release automation is the space that we see a lot of activity happening in there. And from our own point of view, we've had to eat our own dog food to work out how we achieve that.

So from that point of view, it's all about, everybody talks digital transformation, everybody talks about agility. But what these guys will tell you is it's all well and good being agile, it's all well and good being fast, but this is what people rely upon as well. It has to end up being quality and reliability. And I think you'll find it's an extremely exciting story that we're going to go through.

So what I'm going to do is I'm going to pass over to Josef first and say, welcome, Josef, Head of Engineering for Automic. Continuous delivery, what is that to you?

Josef Puchinger

Okay. Thank you, Chris. Hello, everybody. Good to have you here in the room.

What do we talk about today? Maybe let's start with velocity. Who feels the pressure that you have to increase the velocity? Can you see your hands? So I would say it's the majority in the room.

So what do you need to address when you have to increase the velocity? From my point of view, there are multiple things we have to address. One thing is that you have to apply agile methodologies. Another thing is that you have to apply DevOps practices. And the third thing is that you have to focus on reducing your software or application complexity.

We cannot cover all of that today, so let's focus on a few things. Let's focus on DevOps practices, and here we focus on continuous delivery today.

So what you'll hear today is you will hear us talking about how we have applied continuous delivery at Automic, because Automic tells and sells this story to the customers. Automic tries, or makes business, with continuous delivery, with helping our customers to do continuous delivery. And that can only work when we, as Automic's software delivery unit, are champions and leaders in continuous delivery as well, and when we use our own tools to do continuous delivery.

First of all, maybe let's start again with a question. Who has a complex and heterogeneous application landscape and technology stack to deal with? Okay. Some in the room. Oh, heck yeah. Would've expected more.

And who of you feels the business pressure that you have to deliver faster and more often software to production? Okay, so some majority.

And who has the time and the budget to re-architect and rewrite all your software from scratch? Okay. So for the millions of people who will watch the recording afterwards, in the case you can't see the audience, I couldn't see any hand.

So nobody has the time and the budget to re-architect and rewrite the software from scratch. So what is the consequence of it? The consequence is that we have to bring agility and DevOps practices to the software as it is of today. This does not mean that we should not change that software in the long run, but we have to solve this problem for the software as it is today.

Good. So what is continuous delivery? What is our understanding? I think it's not new. It's something what we are doing over years already. We build, deploy, and test and release software. A big difference is that we do it permanently. We do it continuously. We don't do it after six months of developing software. We do it every day, multiple times. So that's the big thing here.

And when you develop software for yourself and you develop the software and operate the software, this cycle even works with releasing to production. So continuous delivery means then that you release to production permanently.

One challenge for us as Automic is that we have more than 2,500 customers worldwide who run our software on premise. And these customers would not be so happy with us delivering new software every two weeks because they cannot upgrade that often.

So the challenge for us is that on the one hand, we want to do or we need to do continuous delivery for several reasons, which I will explain later. On the other hand, we cannot deploy to our customers' production at any time. So what we have to do here is that we have to find customers, internal customers, friendly customers who work with this model.

In our case, we just have decided that we continuously deliver to all the instances of our software which we use internally. So we have customers, we get immediate feedback, even if we cannot roll out to our customers at the outside world. This does not contradict with doing continuous delivery as a practice.

Okay, so what are the key aspects when it comes to continuous delivery? From my point of view, there are three things, three very important things.

One is the culture. So Dev leaders and Ops leaders, you have to make the value of continuous delivery visible and tangible to everybody. And how can you do that? Find the champions in your organization, make small steps, create value quickly, because you will see as soon as the value is seen, there are more and more people joining this continuous delivery gang. This is something what only works when you can show value very quickly.

The second aspect is the software architecture. As already said before, you most likely cannot rewrite or re-architect all your software, so you have to make it work with your existing software. And you have to fix your software over time, so this of course is also important, but continuous delivery and your business cannot wait until you've re-architected your software.

And the third aspect for sure is automation. When you do that continuously multiple times a day, then you cannot afford having manual steps in this cycle anymore. And the funny thing is we thought that we had automated everything already, and once we started shortening the cycles again and again, when we started running it every day or multiple times a day, then we found out that there are still manual steps which need to be automated, because otherwise you cannot keep this cycle time.

So that's our understanding of continuous delivery, even if you cannot deploy to customers' production at any time.

Coming back to what I said, we do continuous delivery with our tools. So Christopher, I would be interested in what did developers say at Automic when they heard that they have to give up all the tools they loved so much and that they had to switch to Automic tools for continuous delivery? How did they react?

Christopher Hejl

Well, basically they loved it, mostly because they didn't have to switch.

I think when you break up silos, which we talk about all the time when doing DevOps, you have to kind of keep the freedom of choice in tools. Because at the end of the day, you don't mind what tools you use, you just want the problem to be solved and the task to be done. Whatever the tool fits best in what context, whatever team.

So similar, as you can see, a ticketing system, which is the basis of communication, collaboration between teams and people. You have the Automic solution, which is the basis of communication and collaboration between systems and applications to bridge those gaps and break up the silos.

Josef Puchinger

Yeah. Okay. So the short version is developers don't need to change the tools they have. Automic just orchestrates across the tools that developers have in use. And when they use Jenkins for build, when they use Git for source code management, whatsoever, developers do not change their tool stack. They just get another layer which orchestrates across this tool stack.

Chris Boorman

Can I just ask a quick little question?

Josef Puchinger

Sure.

Chris Boorman

Just to be clear, from an Automic point of view, are you using things like Git? Are you using things like Jenkins?

Josef Puchinger

Exactly. We are using Git as a source code repository. We are using Jenkins and also TeamCity for doing builds. We have some homemade solution for doing builds for platforms which are not supported by Jenkins or TeamCity. So it's really a mix of tools which is in place, and we use the Automic solution to orchestrate across that. So developers did not change the tool stack.

Okay, so what are the benefits? Managers usually like to hear that there are three big benefits of continuous delivery.

One is quality. Of course, the quality of software increases when you get feedback much faster. Just comparing that what we have today, that we get feedback after every sprint from users using the software in production and from professional services guys or pre-sales guys doing demos with it, is a big value to us, and it really improves the software. Compared it to former times when we developed nine months, then we released the software, then it took some time until customers upgraded to new software. So it could happen that there were 12 months between developing something and getting feedback. This does simply not work anymore.

The second thing is risk management. And don't get me wrong, I don't want to say that we should not take risk when we develop software. We definitely have to take risk, otherwise we will not make step forwards. But we can and we must avoid unnecessary risks. And with continuous delivery, you always deploy small increments, and with that you reduce the risk tremendously compared to having these big yearly releases.

And last but not least, time to value. Again, our example here is that we can present, we can demo, and we can POC new features immediately after they are developed. Immediately after the end of the sprint, we release internally to production, and these features can be presented. So we create the value immediately.

But what does it mean for developers? So how does this look like for developers? Christopher, can you explain that a little bit more, please?

Chris Boorman

AKA, the real version.

Christopher Hejl

Yeah.

So we develop software. And we use Git, as already mentioned before, and basically every time we do a code change, we branch off. It doesn't matter if it's a feature, a bug fix. We make a branch, we hack away. When it looks good, bless you. When it looks good, we have a peer review and we merge it back to our stable branch.

So the typical development stuff and the tools that you expect to see here is your typical development tools, your IDEs, your build systems, your debuggers, and that kind of stuff. And we have most of that, but we also have our own tools in here.

And what's kind of fun is at the build tools, besides using Jenkins and TeamCity, which we both use, we also use our own solution, which is kind of weird because we're not a build tool. We don't sell ourselves as a build tool. So why is that? Number one, because we can. Because a build tool is basically just some process automation. But mostly because, which was hinted at before, turns out it's kind of hard to find a build tool which supports all the different platforms that we do. I tried yelling at Jenkins, "Build me the AS/400 and the BS2000 stuff." Yeah, no luck there.

We also use it for test execution, for example. Well, we don't write our tests within our tool. We use JUnit, NUnit, TestNG, Selenium, Cucumber, Robot, what have you. Freedom of choice for all the tools. But we need to execute all of those at a certain point in time in the process, and this is where we do the test execution orchestration with our tool to span that.

And up there we also have the ARA self-service, which I'm going to come back to in a minute after I explain the next thing, which is what we call the continuous delivery of products. So when we do the code changes, we work on code of individual components, but we have actually hundreds of components which match to 100 different lifecycle entities with different versions.

And then somewhere we make, out of components, products, sorry, which is just a combination of different components together. And those products can actually change kind of often due to some marketing reasons or pricing reasons. So we have a separate system which actually has this configuration, the dependency manager, which defines what components in what version form the product. Which means we have to test all of these combinations.

Every time we do a change in component one over here, we have to build the whole product and all the products that it's involved with and test it. And the testing means not just deploying to some servers and website, but actually deploying to all the platforms that we support. There's multiple operating systems from Windows, Unix, Solaris, what have you. There's multiple databases that we support. And then we run all the different test suites and the different smoke tests that come for different components by different teams.

So there's actually a lot of stuff that needs to happen, and it needs to happen every time. And I thought that this was actually the difficult part. And don't get me wrong, it was kind of difficult, but it worked out pretty well. What's kind of interesting is I had a hard time visualizing all that stuff because everything that I just described was just one iteration, and you have to do this every time that you change one component. So that's a different talk, though.

And we do all of this stuff with our own solution, with our deployment solution, with our ARA. And when everything looks good, all the tests look green, we push it forward into a binary repository, and from there we do a deployment to our dev pod, which is short for development production, which sounds kind of weird, but it's the production system used in development, so kind of makes sense. But also to our Automic IT, which is a completely separate department, and they do their self-service stuff with that.

And we do this deployment step here every two weeks at the moment. We do zero-downtime upgrade, again, with our solution.

And to come back at the first point, when we do all this stuff with this continuous delivery with the product installations and tests, this is great because this is where we find, when some developer changes component one over here, that there's actually component 107 over there, which he doesn't even know that maybe exists because it's seven interfaces away, but you see that maybe something broke, and you can stop putting it in production and fix it before.

But essentially for a developer, that's already too late, because I've already merged my code into my stable code line, and I would like to have this even before. I want to be the only thing at this whole process that I'm actually going to look at the very first box. This is the only step that I do manually.

Before I merge back into some stable code line, I want to do all the things that I just described here, and I want to do this with every commit, which has some challenges, similar to the SAP talk that you heard before. Mainly, you have to do this in roughly 15 minutes because developer's not going to wait for some feedback on a code change for the next day. It's not what you want to do.

So this is also something that we're still improving on.

But mentioned before, we have still a release cycle of six to nine months. So these versions that we have up here are still not the versions that are GA for our customers. Not yet. Maybe in the future, fingers crossed.

So before we have the final release for the GA version, we still have a stabilization phase. Important part, there's not a separate QA department. There's not separate people that do the testing. It's the same people that do the development. We just basically do a feature close. We stop adding new features, have a couple of weeks on adding this little extra of quality before we push it out to the download center.

Chris Boorman

I just wanted to ask just a couple of questions because this seems as though this is the core of what is actually happening.

And you mentioned you're using things like Jenkins and TeamCity. Can you clarify, is it the heterogeneity that Automic needs that basically is why you're using the Automic technologies?

Josef Puchinger

Yeah. That's definitely the point here.

Just when you look at this box, product feature branch. Simple box, looks very simple. But whenever you create the feature branch, your source code repository system allows you to do just a click and you have your feature branch, you have your code line. But you also need a fully functional test system for that.

And just in case you have a simple web application stack, this is an easy one. For our case, where we have a complex and heterogeneous technology stack, we support different operating systems, we support different databases. We have C and C++ in use. We have Java and .NET, so it's not so easy anymore.

And the big challenge here is really to fully automate it, spin up and provision a test system for this heterogeneous world. And for this problem, we have not found another solution which we can use. So here, that's the main reason why we are convinced that the Automic product is the best choice here.

Chris Boorman

And this is the thing as far as Automic is concerned, is that our clients work in this, a lot of traditional technology space.

And the other thing that I'm intrigued by is the orange part in the middle there. So we use, I think it is Salesforce.com for our customer-facing support portal. We use ServiceNow for our internal service desks. So we actually have two cloud-based environments.

And I think what you're saying here is you are deploying every two weeks into that production environment within our own environment, with zero downtime deployment. Could you just clarify the zero downtime? Because this is something from an agility perspective that we are increasingly seeing the need and providing to our customers is, quite simply, you don't need downtime to deploy new upgrades, and that's where our roadmap to continuous delivery for our clients is all about as well.

Josef Puchinger

Yeah. I would even say you must not have downtimes to upgrade, especially when you do it more often. When you do it regularly every two weeks. Of course, this requires you to find a solution to do upgrades without the downtime, and there are different concepts in place.

In our scenario, our workload automation software supports zero downtime, so we are able to use our release automation software to deploy workload automation without any downtime. This is something what must be supported by the software itself, and without that, there is no way.

Chris Boorman

So one of the things we've also heard yesterday and today is a lot about the culture, a lot about the challenges.

So Josef, could you share a little bit about, okay, this isn't easy, this wasn't easy. Congratulations on getting to where you've got to. What were the challenges?

Josef Puchinger

Yeah. First of all, we are for sure not done. I think that's very important to mention. This is a long-term transition project, and it is never done. You always will find things to improve.

But nevertheless, I think we have reached a lot and we have made huge steps forward, and we had a lot of quite big challenges.

So what are the challenges? First of all, it's a culture mindset, especially for developers. You must understand when developers go from... Just need to go back to this picture.

When developers merge from the feature branch to the product master, this is the only place where manual interaction takes place. Anything else is fully automated. So for the developer, it's very important that in his mindset, merging something from the feature branch to the product master means that it goes to production.

Of course, in reality, there are some stages in between, but from a developer's perspective, there is nobody else looking at it anymore. So it's not like, "Let's deploy it, and there is a tester who will look after it." All of that must happen in the feature branch. Once the developer merges it, the standard case should be that it goes through to the production.

And this mindset is not easy, especially when you have to scale. Just to give you an idea of scale, we have around 150 engineers in our department, so it's quite tricky to convince everybody that this is the right way to go. And we for sure have not convinced everybody, but we have convinced the critical mass, and now it starts improving very quickly.

Another challenge is, we also heard it at one of the keynotes today. Another challenge is to keep the feedback loop short. If a developer checks in and gets feedback five hours later, the developers will not work with the results. They will simply ignore the results.

So interestingly, we came up with the same maximum time of feedback as we heard it in the presentation before. We said the feedback to the developer after doing a check-in or a merge must take maximum of 30 minutes. That's already too long. So we have to shorten that for sure. And this is something what is really important.

The same is valid also for feedback from users, from real users. The one is the feedback from tests, from automated tests, from manual tests, and the other thing is feedback from real users about the usability of the new feature.

Then another challenge is effective test automation. And believe me, we had a lot of discussions. What is the best test automation tool? Is it this one? Is it that one? And then there are preferences. Developers do have preferences, and we really, really wasted a lot of time having this discussion until we realized that the tool is not so relevant. The tool is not so important.

There are many other aspects for test automation which are important. The first thing is stability, because if you have too many cases where an unstable environment or a wrong test case breaks or creates a red result, then developers stop working with it. They do not look into it anymore. So you need stability, that you really can get results out of it.

Second thing is you need to run them fast, short cycle times. And the third thing is that you have to have self-contained tests. You write them once, and suddenly with this approach, you run them in many different environments, and the tests must be designed in a way that this works.

Once you have solved this, I'm convinced that the test automation tool is a minor issue. We in our organization have even decided to give a lot of freedom to the teams, and they can choose their test automation tool within some given boundaries. But nevertheless, we have many different tools in place, and it's not a problem at all.

Another challenge is the full-stack application provisioning. As said before, as long as you have a simple web application stack, it's easy, and you get it out of the box, maybe even from Jenkins, it's solved. But for complex and heterogeneous environments, it's not solved so far. And we had to go through that, and that's something what we definitely could solve with Automic's products, as they exactly support these heterogeneous environments. They are designed to cover that.

Then the next is you need a customer. You need somebody who uses the increment of the sprint. If there is no user, it's not worth to deploy it. If you don't get feedback, don't waste your time with deploying it. So all of this only works if you have somebody working with the software and giving you feedback immediately.

And on the other hand, it's also a challenge for us to communicate within Automic what is the change about, what does the latest sprint increment deliver. And this is something where we can write up long emails, you can communicate via conference spaces, and you will see that nobody reads it and nobody knows it.

So what worked at Automic is that we create an eight-minute movie after every sprint, which contains a little bit of fun and a little bit of what is in the new release. And we saw that these movies are watched. So this is a much more successful way of communicating the news about the new software increment.

And last but not least, very big challenge is you will always, or I think you might have people in your organization who say, "We can't do continuous delivery because it does not work for BS2000. We cannot solve it for z/OS." I don't know.

So there are always edge cases, and it's very important that the approach should address 80% of your problem. Don't block yourself with the other 20%. I don't care if we have to take manual activities or actions when we change a BS2000 agent. I don't want to say that BS2000 agent is not important, but we don't change it every day. It's not our 80% problem, and I don't care if there are manual steps. This should not prevent us from automating all the other stuff. And I think that's also a question of mindset, how to get there.

Chris Boorman

So as far as that's concerned, what you've gone through are some big changes from the culture point of view. There's also clearly some big capabilities from a technology point of view. So what are the benefits you're actually now seeing to you as a team?

Josef Puchinger

Luckily, I've prepared something for this question.

What are the benefits? So as said before, we are not done. We will never be done. It's a long process. Nevertheless, we can see good benefits already.

And one thing is the features and the usability of the product is better because once we deliver something internally, we get immediate feedback. "Oh, you cannot use this feature. You have to redesign it." And at this point in time, change is still cheap. Because we have not rolled out to the real customers. We have not GA'd it. We can simply change it in the next sprint.

Another thing is minus 30% of customer tickets in eight months. I think that's really a great success. And this proves that the quality, that all these continuous delivery things we have implemented do have a direct impact to the quality. Of course, it's not continuous delivery only. But the result is really good. We could decrease the number of customer tickets heavily.

Another point is install and upgrade. Install and upgrade is not really a standard use case for software. Install, you usually use once, upgrade maybe once a year, once a month. So it's not the standard use case. And this also led to the situation that install and upgrade was not perfectly solved in our software in the past.

And the continuous delivery implementation forced us to invest into that, because continuous delivery doesn't make sense if you cannot install it automatically, if you cannot upgrade it automatically, if you cannot do upgrades with zero downtime. If you can't do that, forget it. So this continuous delivery initiative also forced us to invest into very important product features.

And last but not least, just imagine if you go through these two-week cycles, just imagine that releasing software, GA-ing, generally making it public, could be a pure business decision. Because at the end of the sprint, from an engineering point of view, we are done after every sprint. We release it internally, and once product management says, "Yeah, that's cool now. We can release that because that makes sense for our customers," then we could release it. And I have to admit that we are not there yet, but that's the vision we target for.

So in the bottom line, I can say that our products are better because we do continuous delivery, definitely. I can see the impact. I can see the facts. And I also say that our products are better because we use them to do continuous delivery. We use our own tools.

So these are two very important ingredients if we want to build good software. Do continuous delivery to get fast feedback, and use your own software. If you don't use it, you will never understand if it is a good product or not. And we definitely use it. We eat our own dog food. We use our release automation product to implement all of that.

And that's how I see it. And I really want to encourage all the dev leaders and ops leaders here, continuous delivery is a challenge you have definitely to face. You have to solve that in your organization, and you have to solve it together. Otherwise, for marketing reason, I will not finish this sentence.

What happens otherwise, I just ask everybody here in the room to ask yourself, do you think that you can survive without continuous delivery?

Thank you. Back to you.

Chris Boorman

Thank you very much indeed. Thank you very much.

Gentlemen, if we could just have a round of applause for Josef and Christopher.

We are running out of time as far as that is concerned, but I hope you agree. It's lovely to see, and we've heard a lot today and yesterday about organizations who are moving to continuous delivery. And from an Automic point of view, how we have done that, the end result from my point of view is better quality and faster delivery of new capabilities, which are absolutely brilliant.

If you have any questions, please hang around afterwards. But with that, thank you very much indeed. Thank you.