Host Intro (Gene Kim)

So, one of the topics that has come up so many times over the last decade is: how do I do DevOps for COTS applications, such as SAP or Siebel or Salesforce, Workday, NetSuite, ServiceNow, and so forth?

Various people have come up with answers to this question, but they all seemed rather unsatisfactory to me. That is, until Andrew Davis, senior director of research and innovation at Copado, gave a talk called "DevOps and Salesforce," which makes sense because he's author of a book called "Mastering Salesforce DevOps."

And the insight he gave seemed universal to all of these applications that I had mentioned above. Well, not really. He had to explain it to me a couple times before I finally got it. This seems like a problem that almost every large enterprise faces, and how you think about these applications will limit the impact you can have.

For that reason, I am so delighted that Andrew will teach us about DevOps for low-code platforms, why this is important and why it's on the rise, and what makes those applications uniquely challenging, and what we can do about it.

Andrew Davis

Hello, and welcome to DevOps for Low Code. I'm Andrew Davis. I'm the senior director of research and innovation at Copado, also author of the book "Mastering Salesforce DevOps."

So first of all, I want to establish what is low code again. Low code is application development using visual or declarative techniques instead of code. Just to make sure we're all on the same page about that: first of all, I think we all know what infrastructure is. We all understand what code is. So low code are systems that enable you to create a user interface, like you see in the upper right, or create a flow of business process logic, like you see in the lower right, using visual or declarative elements.

Now, when we're talking about DevOps for infrastructure, we're talking about all these cloud platforms, AWS, GCP, technologies like Kubernetes, and so forth. When we're talking about DevOps for code, we're typically talking about build pipelines and CI/CD and build scripts and so forth. What are we talking about when we're talking about DevOps for low code? That's what I'm going to cover today.

So first of all, we need to understand a little bit more about low code, because low-code platforms can come from two very different sources. The first is as a way of simplifying code-based systems, and the second is as a way of making commercial apps more flexible. They're a little bit different, two very different streams.

The first one, simplifying code-based development: this is equivalent to moving up through these levels of abstraction. They say the whole history of computer science is all about increasing levels of abstraction, from machine language to assembly language, to higher-level languages, to low code. And so you have a bunch of low-code systems. There's something like 200 different low-code systems that are all allowing developers to build applications with visual declarative elements instead of having to write code, so to speak, on the command line.

What are the benefits? With each level of abstraction you're moving up, you get greater simplicity, which means that the systems that you're building at each of these higher levels of abstraction are easier to maintain, but they also have coarser granularity. You're dealing with bigger chunks of things, bigger components. And there's a couple of downsides. It means you get less flexibility as you move up through the stack, and potentially less speed as well.

Now, the other very different approach, different direction in which low code arises, is in the direction of making commercial applications more flexible. So, the first thing you can do to take a commercial application and make it your own is to put your own data in there, and then all of a sudden, it's yours. You've personalized it. The second thing you can do is change configuration.

But we know that humans have infinite desires and infinite different interests, and there's no amount of configuration that can address them all. So at some point, these commercial applications need to develop no-code ways of customizing the application if they want to meet the diverse needs of their customers and potential customers. And at some point, they may even need to enable code to be written on their platform.

So if we look at some of these big commercial applications like ServiceNow, Workday, Salesforce, SAP, and there's many more, they've all enabled increasing levels of flexibility and configurability on their platforms. Why? Greater flexibility means that they can meet more needs for more potential customers, and it also enables them to make changes, fine-grained controls and changes to an application, rather than expecting all of a company's thousands or tens of thousands of users to change their behavior. So some of these small tweaks really aid a lot in the adoption for the applications.

With anything, there's trade-offs. When you move from a pre-built, pre-packaged application to something that's heavily customized, there is more work required to set it up and get it running, and it's more complicated. But one way or another, these two trends have converged: this trend of simplifying code-based systems, which may be something you run on premise or in a cloud-agnostic way, and this idea of making commercial apps more flexible. They both converge in this idea of low-code systems. So, just to clarify up front.

Now, low-code systems are very much on the rise to the extent that Gartner is predicting that by 2025, the majority, about 70% of new applications developed in enterprises will be based on these low-code or no-code platforms, and that at least half of them are going to be from outside of traditional IT organizations. They'll be coming from marketing departments, finance departments, customer support departments, and so forth. Gartner and Forrester, for many years, they've been studying these low-code platforms, ranking them, rating them, very fast-evolving space.

Salesforce is sometimes called the OG of the low-code world. Its origins, it started enabling code in the platform in 2008. If you really want to understand Salesforce, though, you really need to transport yourself into their world, because Salesforce has a very different world and very different culture. Not only do they host some of the world's biggest technology conferences, they have some of the world's densest population of stuffed furry characters at these tech conferences. You won't find stuffed furry characters like this at the DevOps Enterprise Summit or most other tech conferences because Salesforce is really trying to connect with a very different audience.

The talent pool in Salesforce looks very different from the talent pool of traditional application development. So I want to explain that a little bit more.

Let's look first at traditional application development. You want to build an awesome app. How do you do it? First, you need to build custom code and infrastructure, and that's going to cost you some money, of course. But you need a software developer to do it, and they're going to cost a fair amount of money. They're hard to find, they're expensive to pay, and they're hard to retain: 13% year-over-year attrition in the tech industry. They, in turn, need to get their education from a computer science degree, typically from a reputable university, and that, in turn, requires a lot of money. So some of these things act as limiters on the talent pool.

What Salesforce has done is they've really emphasized a different way of building applications using declarative development: clicks, maybe a little bit of code. To do this, you can use Salesforce admins. Now, you still have to pay them, but perhaps not as much as your traditionally trained software developer, because they've come from non-traditional backgrounds. They have migrated into the tech world later in life, potentially, or without necessarily the benefit of a traditional university education.

They're likely to have gotten at least part of their education on Trailhead. One of the biggest, most impactful things Salesforce did is they launched this free, gamified learning platform, Trailhead. Did I mention it was free? And made it as fun and engaging and open and inclusive as they possibly could to bring in a very different and much bigger talent pool. There are currently millions of people who've trained in Salesforce development. And if you're wondering what to do with that extra cash, you'll need to send some of it over to Salesforce.

Now, what is different about DevOps for low-code systems? Disclaimer: my background is mostly in Salesforce. This is going to vary a lot depending on what platform you're looking at, but hopefully some of it'll be relevant to all these different platforms.

Salesforce, and in fact all of these low-code systems, make it easy to build. Easy is great, right? But there are some limitations to easy. The trouble with easy is that easy means fast. You can get a lot done in a short period of time, which means you build more. And when you build more, things become complex pretty quickly. If you get enough moving parts in any system, things tip into complexity pretty quickly.

Now, the vision in the Salesforce platform is that it's like we're building with Legos. You can take these small components, you can assemble them, you can build this magnificent structure. Your Salesforce org or environment looks like this Salesforce tower made out of Legos. But the reality is a bit more like building a Jenga tower, trying to deal with a Jenga tower in many cases. Teams, as they get bigger, as the orgs scale, it becomes increasingly risky to make changes.

So I want to step you through this. This is kind of a summary of the evolution of Salesforce. It's a summary of the evolution of all of these low-code systems, and it's often recapitulated in the experience of individual development teams.

You start with just having one production org, your SaaS application. Life is simple, and you spend your days like this. No problem, just relaxing. Everything is great because other people are taking care of infrastructure and so forth.

Now, gradually, you begin to pay the price of success. The price of success is that over time, you get more and more users using the application that you've built. This has two consequences. First of all, your applications become more and more business critical because more and more people are using them and needing them for their daily work. The second is that all of these users create a massive demand for changes. They all want something different or customized for their needs.

If you take these two together, business critical but high demand for changes, what that means is you're facing a very high risk of changes, and your life begins to look more like this. There is an enormous amount of stress and pressure every time you want to try to change this org if you're doing everything just in the production environment.

And so you embark in this amazing adventure called the software development life cycle. That may not be new to you, but for a lot of the people who are building in the Salesforce platform, that's the first time they've ever done anything like that. I can't overstate how different that development community, the Salesforce development community, is from traditional software development. So you embark on this journey. You've got dev, test, and production environments. How hard could it be? There's a big adventure lying in wait.

This is where things start to get complicated in a different way. You have these new developers who are often relatively early in their skill development. They've learned on Trailhead. It's kind of like somebody who's learning to drive. But you get these large, complicated Salesforce environments, and what you need is more akin to traffic engineering. You need the ability to orchestrate and conduct changes at high scale and high speed.

The challenges that begin to emerge when you have multiple environments is you have trouble keeping those environments in sync with each other. You have trouble tracking changes. You have trouble propagating those changes systematically. There is a risk of interactions and side effects, and you get this accumulation of technical debt. The net effect is that many times, your org or your environment becomes more like this: tangled, convoluted mess that is very risky to change.

So when I'm talking about orgs or environments, you need to understand that Salesforce controls the underlying environments, whether that's a production environment. Because this is a SaaS system, it's wonderful they're managing the infrastructure for you. That's a huge category of headaches off your plate. But it does make things a little bit different. The sandboxes you're developing in, they are also hosted by Salesforce. And there's a new kind of org, short-lived scratch orgs, but these also are hosted by Salesforce.

Now, what is Salesforce anyway? Basically, it is one big monolithic database on which there is the Salesforce platform. Salesforce provides both the platform and the database in the cloud. They also provide these standard apps that just kind of work out of the box. You can also install these third-party apps that greatly expand the range of capabilities with little or no effort on your part, and you can build custom applications that are specific to your organization.

The powerful thing about this is that all of these apps are naturally integrated with each other because they're on the same platform and they share the same database. That's one of the very compelling things about Salesforce: everything is in one integrated database, although as we know, that can bring some other challenges.

Now, if you want to move changes from one environment to another, let's say you've got an identical environment, you want to move changes, there is one gateway through which you can take changes out and send them in, and that is called the Metadata API. You can pull a set of related metadata changes describing the changes you made in the org, and the idea is you just send them over through the Metadata API in the target org.

Except you encounter deployment errors because you were facing a missing dependent field, for example, or you were lacking test coverage, or you had malformed XML, or my favorite, unknown error. The list goes on and on. The powerful thing about this is that it does allow you to begin to track these changes in version control, and once you've got things in version control, you can begin this journey of deployment automation. And that's where things really get fun.

But you may be familiar with this periodic table of DevOps elements, thanks to Digital.ai. Because Salesforce works in such a different way from traditional infrastructure or code-based systems, the vast majority of these tools do not work on Salesforce. In short, your DevOps mind tricks won't work on Salesforce, and so you've got to have a different approach.

One of the reasons it's a bit tricky is Salesforce relies heavily on these huge XML files. Some of these can be 100,000 lines or longer. Now, what happens when you try to merge XML files in Git? You might think, XML merge, how hard could it be? The answer is: you don't want to know. Because if you merge these, you get "how you hard don't could want it to be know," which is probably not what you intended.

This is a picture of an actual Git merge conducted in Salesforce. This is from a project that I was on a couple of years ago. As you can tell, it's kind of tricky.

So the limiting factor on building your own Salesforce DevOps solution is that you need people who know Salesforce. You also need people who know DevOps, and you need people who have time to write scripts. And so it's a little bit of a narrow intersection between those, which is what makes this a little bit of a tricky challenge.

There are two CI/CD and testing options for any low-code platform, and it's important for you to understand about the trade-offs between build and buy because you, as a DevOps expert in your organization, are probably going to be the ones that people turn to. People who are building on these low-code platforms are going to be needing advice, and you're going to have to understand the trade-offs and the pros and cons of different options.

So if you choose to build, it's good to know, certainly for Salesforce, the dev tools have improved greatly, and I think all of these platforms are trying to provide more options for developers. You'll need to commit ongoing time to building out the scripts, building those capabilities that your teams need. What this means is that some of the costs when you build your own are hidden. There are two costs. One is the cost in terms of the time that it takes to actually explicitly build the tooling. But the second is if you've not built the most effective tooling, there's a hidden performance cost that the organization will face.

If you're going to build your own, this is easier with teams that have a predominance of coders, people who do come from a more traditional software development background. Certainly in Salesforce, you're going to need to brace yourself for merge conflicts and some of these other strange idiosyncrasies of the platform. And you'll probably need to train these non-coders in how to use Git.

If you opt for the buy scenario, there's a growing number of commercial options. I work for Copado. It's one of the leading commercial options for Salesforce DevOps and testing. But it still requires time spent to train people in process. Remember, in many cases, these are people not from a traditional software development background. They're going to have to be guided through the process of why you do certain checks at certain points, why you need to track everything in version control, why you can't access production directly, and so forth. But the costs when you buy a tool are a bit more explicit. You pay the licensing fees, and you can get all the support and so forth that you need.

This tends to work better for teams that have fewer professional coders, more click-based developers. As an example, this is a screenshot from Copado. Copado allows you to see all of the different environments that are involved in the Salesforce development life cycle, or whatever platform you're working on. It allows you to view the history of the deployments and allows you to actually take action and move changes between one environment to another. As you can see, it's a low-code screen that fits the expectations of these low-code developers building on it.

This is a testing tool called Copado Robotic Testing. On the right, there's a screen where people can actually click through and record their actions. Any user of these systems can record their actions, and they're recorded on the left in fairly human-readable text. We have to always, with any systems we're building or supporting, think about the usability aspect based on the skill sets of the people who are building it.

Delighted to have a chance to share this with you. Let me ask you for the help that I need. I'm very much still getting up to speed on some of these other low-code platforms. If you're working on any other systems, ServiceNow, SAP, and so forth, especially if you're facing new challenges, I'd love to know more about what it looks like in that world. And if you are struggling with DevOps for Salesforce, please do get in touch. Happy to help. Thank you so much. Grateful to spend this time with you.