Andrew Davis

Hello everybody. So I'm happy to have a chance to share with you today on DevOps for Salesforce. And the full title of this talk is "Why the business has been bypassing IT for decades but is now in way over their heads, and how you can help without going insane." My name is Andrew Davis. I'm the senior director of research and innovation at Copado. I'm also the author of the book, "Mastering Salesforce DevOps."

01Salesforce context

So I want to level set. I want to make sure that everybody here knows what Salesforce is, how it works, basically why it exists, and why you should care.

So I want to go over upfront, Salesforce, the business, the technology, the acquisitions, the community, the parties, the mascots, and the real estate.

So I want to start first of all with sales. Why sales? Salesforce. You're not in sales. Why do you care? So why do you start with sales? Why did they start with sales, and why does it matter? First of all, profit. Let's start with profit. To get profit, you need revenue. To get revenue, you need sales. To get sales, you need customer data. And to get customer data, you need a Rolodex. And if you want something that's a little bit easier to update than a Rolodex, you can use Microsoft Excel. So this is the origins and the roots of Salesforce.

But prior to Salesforce, people had realized that maybe there were better ways because there were issues with the data attrition. Salespeople would leave, and they would take the Rolodexes with them. And there was entropy. People take their Rolodexes, they get wet in the rain and so forth. And there's management. Management wants to see the data. They want to see the customer information, and more importantly, they want to see your forecast. So they want to get all that data in one system, and that system came to be called a customer relationship management system, CRM, and the famous players prior to Salesforce were Oracle and Siebel, and then Oracle bought Siebel.

So customer relationship management. I want to explain to you how to get started with Oracle or Siebel in less than three years and under $5 million. So first of all, you need to start with budget, CapEx, capital expenditure. You need to get all the money committed up front, and then you need to get time and attention from IT. This is an actual diagram of an actual IT org right here. And then you need to install the CRM, which involves a lot of CD-ROM disks, and then you need to host your on-premise CRM, which involves a lot of servers.

But Salesforce comes along, and they think, "We got a clever, disruptive idea. What we're going to do is we're going to set this up so you only have to pay monthly or yearly with a monthly pricing point." That's operational expenditures, not CapEx. And even better, you can bypass IT, which as we all know, is the root of all trouble and delays is the IT team. And there's no software. There's not going to be any software, and there's not going to be any servers. So who wouldn't love it?

So Salesforce was pushing the cloud before the cloud was cool. This is 1999. These people are in the cloud way early. And so summarizing the benefits of Salesforce, they host all the infrastructure. There's no AWS, there's no GCP, whatever that you have to deal with. Major upgrades three times a year. They provide the support, the security, training, all that stuff. Application itself, built-in out of the box, addresses most of the common needs, and it is highly customizable. Just got a lot more customizable over the years.

So I started with the question, why start with sales? But then the question becomes why stop with sales? Things have worked so well so far.

There is one negative byproduct of sales, and that is that it generates customers. And the problem with customers, as everybody knows, is that they complain, and so you need customer support. So what Salesforce did is they thought, "We'll help with the customer support as well. We'll provide a Service Cloud." But then they got an even better idea. It's very hard to provide customer support. Let's let the customers support each other. Let's have a community supported by the Salesforce Experience Cloud, where the customers can help each other and get all the information they need. And since it's going so well, let's get earlier in the whole process and let's get some prospects with the Salesforce Marketing Cloud. So as you can tell, Salesforce has really spread out to address a whole lot of the business needs. And Salesforce has done a really good job with this. They've received a lot of industry accolades and are leaders in many categories, a highly regarded company, very successful financially, and high customer loyalty.

Not only was Salesforce pushing the cloud from way back in the day, they were pushing low code. Nowadays, everybody's talking about low code, but Salesforce has been in this business for a good long while.

So understanding low code, first of all, think about traditional application development. The way this works is you want to build your awesome app, you need to start with some custom code and infrastructure. That costs some money. And you need software developers, and they cost a lot of money, right? And they need a computer science degree from a reputable university, and that costs a bunch of money as well.

But Salesforce had the idea, "Hey, we bypassed IT once before. Let's bypass them again. Let's do the Salesforce development the low-code way." Clicks, maybe a little bit of code.

And for that purpose, you can use a Salesforce admin. Now, you do need to pay them money as well, but it's less than you would typically pay a full software developer. And partly because they come from non-traditional backgrounds. It's a good compromise. They can get paid more to be a Salesforce admin, than in their previous job, but less than a full software developer. And there's no university degree necessarily required. You can get trained on Salesforce's own training platform, Trailhead, for free. Now, there's one thing that I should mention is that you do also want to pay Salesforce a bunch of the money. So if you were wondering what to do with that excess cash, Salesforce has a suggestion.

So custom application development is, I believe, the fun part of it all. And this is the way it works in Salesforce. You've got a customizable database. You can build business process automation. There are graphical tools to build the business processes. There are code methods to do it. Customizable UI, again, with graphical methods or code-based methods. But also there's a huge application ecosystem. It's a business app store, and that's quite cutting-edge back in the early 2000s when they started this program. The applications can have all of these aspects just built in naturally. And if that's not enough for you, you still need something more clever, then you've got a robust versioned API that you can use to plug in your external systems.

I mentioned these Salesforce admins. So I wanted to talk a little bit about the community and the culture and the world that Salesforce has built up around them. And I really mean a world. It is huge. Dreamforce is the largest single-vendor tech conference in the world, about 180,000 people back when you used to do that kind of thing. They've got a lot of cute, fuzzy mascots appearing in lots of different forms and a lot of focus on personas, especially people coming from non-traditional backgrounds, coming into the Salesforce platform and becoming awesome admins and so forth. And the Salesforce community is, shall we say, dedicated. Highly committed community of people building on the Salesforce platform.

So that was just getting you up to speed on what Salesforce is.

02DevOps for Salesforce

Now for the fun part. Let's talk about DevOps for Salesforce. What does this look like? How is it different, if at all? Which it is a lot. So I want to disambiguate two things. First of all, Salesforce and Salesforce. People get these two confused. Understandable. And then I'm going to ask you to check all of your DevOps tools at the door because they're probably not going to work in the Salesforce platform. I'm going to talk to you about Salesforce metadata. I'm going to talk to you about XML Hell, an initiative called Salesforce DX, and as well commercial tools. And there are commercial tools that are available and do work for Salesforce.

So in terms of building on the Salesforce platform, I mentioned low-code. This is a picture on the left of building the schema. On the right, it's building the business processes, one of many ways to make it easy to build on the Salesforce platform.

So easy is great. But some conditions apply.

The problem with easy is that easy means fast, and fast means more, and more means complex, actually.

So the idea with Salesforce is it's like building with Legos. You get all these individual nice building blocks, and the vision is you can assemble all of those building blocks into an amazing structure that is your Salesforce org, which looks basically like this Salesforce tower made out of Legos. But oftentimes, what teams end up with is something a little bit more like this Jenga tower, where there is a significant amount of risk in making changes to that system because it has become so incredibly complex.

And so Salesforce has gone from a world of easy to a world where we have so much easy, we're really struggling, and it's not quite as easy anymore. So the development life cycle in the Salesforce platform is where things start to get kind of complicated.

Now think about all of our awesome Salesforce admins. Now there are a lot of Salesforce specialists who do have CS degrees and a lot of IT architecture background and so forth, but it's a broad mix. So you have to think about the blend here. You've got all these people who are learning to drive with Trailhead, getting started. But you've got large organizations where an increasing amount of their business processes are running on the Salesforce platform. Some organizations going all in. Massive portions of their applications are low-code on the Salesforce platform.

And this is much more like traffic engineering as opposed to learning to drive. So very different scale. Now the challenges that you face include things like keeping your different orgs in sync with one another, including your development, testing, and production orgs. Tracking changes that you've made, propagating changes systematically, the risk of interactions and side effects, accumulating technical debt. This probably sounds familiar to all of you from traditional systems. And so what you get is often something that looks like this. So this is often the way that the Salesforce orgs end up being very tricky to manage.

And so when we talk about Salesforce, this is their marketecture. It looks like this. You've got all of these applications that do all these different things. But only a subset of this is what I think of as the Salesforce platform because my skills are on what you call Salesforce Core. And the underlying technology on Salesforce Core is covering part of this, but not all of the different capabilities. So when we're talking about Salesforce here, just understand it's not all of the applications that they've acquired.

03Platform architecture and tool mismatch

So I want to take you back to a topic that you know and love, the difference between on-premise infrastructure as a service, platform as a service, and software as a service.

So infrastructure as a service, as you know, they take care of some underlying aspects like the virtualization and the underlying networking and so forth, but you have to build on top of that. Platform as a service. They take care of more of that stack. The hosting provider takes care of more of that stack, but you still need to provide the code and database and so forth. Then software as a service, basically, they take care of everything and all you have to do is configure the application and supply a bit of data or a lot of data.

Now Salesforce is interesting because it functions like a platform as a service, but the way you configure it is basically like a software as a service in the sense that everything that you do on Salesforce is configuration, what Salesforce calls metadata. Even the code and so forth that you load onto the Salesforce platform to cover the UI and the business processes and so forth, they're all treated like configuration. It's very different from just pushing files up to a service. So understand that difference.

And what that means in practice when I ask you to check all your DevOps tools at the door, this is the periodic table of DevOps elements, DevOps tools, thanks to Digital AI, that we know and love. Huge pantheon of different tools that serve all different aspects of the DevOps process. The problem is that almost none of these are relevant to Salesforce. There's only a very limited set that actually will work on Salesforce. Version control, a little bit of build scripting, maybe your CI tools and so forth, but it's a limited subset.

So we want to think about-- I often get involved in conversations where we have two communities that are meeting each other for the first time, the DevOps community, you folks, and the Salesforce community. Again, the Salesforce community is amazing. DevOps community is amazing. And so I'm delighted to be in the middle of these conversations where these two communities are meeting.

But you have to understand there's a knowledge gap, and there's a difference in the expectations. So the DevOps teams that the Salesforce folks are often meeting for the first time when they're starting to get into things like version control and deployment automation and so forth. The DevOps teams make a bunch of assumptions about how applications work, and these assumptions are simply not true on the Salesforce platform.

First of all, these DevOps teams kind of assume that everybody can just use Git. But on the Salesforce platform, that's not a comfortable experience, not a familiar experience for these low-code, click-based admins.

The DevOps community tends to assume that an application's just a bunch of files. You can push them around. But Salesforce, remember, it's software as a service. It's not files, it's configuration. And you tend to assume that if you need to merge the files, you can do so easily. And I'll explain why it's not nearly so easy in the Salesforce platform to just merge files. You tend to assume that you can use comprehensive build scripts. You can build something in Java. There's tons of options for build scripts, but that's not true in the Salesforce platform. Many more limited options. Tend to assume that you can build, test, deploy things very easily and automatically. Also not true. The deployment jobs and test jobs can run for a long time. You assume that you can build, run, and test on any computer, but you can only run these applications on Salesforce itself, and that you assume that you can define an environment to run the application. But again, only Salesforce provides that environment.

So in short, your DevOps mind tricks won't work on Salesforce. So I want to help you to understand how things do work.

And with Salesforce, we start with environments, the operating environment. So Salesforce controls the underlying environment. The production orgs are hosted by Salesforce. The development and testing orgs, called sandboxes, they're hosted by Salesforce. There's a new kind of short-lived environment, scratch orgs, that can be used in some circumstances, but again, they're hosted by Salesforce.

So every Salesforce instance is a big, monolithic database. This is Salesforce Core, and the Salesforce platform lives on top of this. Originally, they were Oracle databases, and Salesforce is evolving that a bit. Salesforce provides these, maintains them, does all of the database admin stuff, all that kind of stuff. And they provide a lot of standard apps. And these standard apps, they just work out of the box. You want a customer relationship management system? There it is. Service Cloud. They just work out of the box.

And there's also these third-party apps that you can install from the AppExchange, Salesforce's app store. And you can build your own custom apps. And all of these things are integrated naturally because they're sitting on the same platform. They have access to the same data. They can interoperate with one another. You can trigger one process in a third-party app from inside your custom apps and so forth. So nicely integrated, takes away all those integration problems. That's why people want to build a lot on this one platform.

Now, where things start to get a little bit tricky, and again, I should say it's quite easy, really very easy, relatively, to build on the Salesforce platform. What gets trickier is when you want to begin to move things between different environments. Where you have a development and a testing environment, you want to move your changes from that environment over into your production environment.

So each Salesforce org has one doorway through which you can get changes out or in, and that's called the Metadata API. So from the Metadata API, you can pull a set of related metadata. So you've got a little bit of user interface, a little bit of data layer, you've got a little bit of business processes, maybe some code, config, whatever, your related metadata.

And on your production org, you've got the Metadata API. So your vision is that you just take it out of one org and you send it to the other org, and that basically works.

But it's very easy to have a lot of deployment errors. And the deployment errors are things like you're missing dependencies. You missed a dependent field. You're missing test coverage on part of that metadata. You've got malformed XML, especially if you're processing it by hand and trying to use version control. We'll come back to that. And you can get these unknown errors. So I personally have addressed tens of thousands of Salesforce deployment errors, I'm very proud to say. It's not exactly that fun, though. And the errors can go on and on and on. So it's quite a pain.

Now, over the last few years, really just the last five years mostly, people have been beginning to try to track this stuff in version control for obvious reasons. And once you've got it in version control, it's not a big step to begin to do deployment automation. That's where I kind of got involved in this whole activity, just trying to make the whole deployment process easier. But this is the big picture. Now, Salesforce provides one built-in method for doing this in a declarative way, in an admin-friendly low-code way, and that's called change sets. Change sets were introduced 13 years ago, and they basically haven't evolved much from then. They've got a really, really rough user interface, not what you want to be doing your work in.

04Salesforce DX, metadata, and XML

So, a few years ago, Salesforce launched finally, because of this popular demand, an initiative called Salesforce DX, which stands for development experience. So it's kind of a moonshot project, really, really ambitious, and they thought way outside the box. They thought, okay, in an ideal world, how would or could Salesforce work? And came up with brilliant ideas, and they launched it in 2017.

One of their brilliant ideas was we need more funding for our development tooling teams. So those developer tooling teams increased significantly, and so that allowed them to get a lot more work done. They began to improve the Metadata API and a lot of the other APIs. They adopted Visual Studio Code as the IDE of choice, moving away from Eclipse. They have a new command line interface where you can do some scripting. They created this new kind of org, scratch orgs. They created a way of packaging for enterprises. And an easier way of tracking your changes, the changes that you made in one org. Just imagine, you're a low-code admin. You don't know anything about the Metadata API. You're making changes. How do the changes that you make in the nice Salesforce user interface translate into metadata? It's a very tricky challenge. People struggle with that.

Now, in 2016, 2017, when Salesforce DX came out, that basically created a renaissance in terms of open source activity on the Salesforce platform, where they began to really grow the activity, the enthusiasm for things like version control. So the dotted line here indicates the percentage of GitHub repos that are related to Salesforce. And as you can see, when Salesforce DX was announced and in the years following, things have really started to tick up. So a larger and larger proportion of activity on GitHub is done by Salesforce community. And the red area down here are repos that mention SFDX, which is an indication that they're using the Salesforce DX scripting lines. As you can see, this is growing actively these days.

So how Salesforce scripts work. I mentioned there's this Metadata API. There's just a couple ways to work with this. The traditional way is using Ant. Now, some of you may know Ant from 2001, not the world's easiest tool to script in and so forth. And so the Salesforce DX initiative launched this new command line interface, SFDX.

But both of these fundamentally are dealing with a large amount of XML. All this metadata is stored in an XML format.

There is no other way to move changes out of a Salesforce org or into a Salesforce org. Doesn't matter what kind of tools you might throw at the problem, commercial tools, build your own, and so forth. They're all going through the Metadata API. This is a little picture of Salesforce metadata. This is inside Visual Studio Code. But imagine some of these files are literally hundreds of thousands of lines long of XML. Highly complex.

Now, the problem with dealing with XML, I mentioned about XML hell, that if you really want to manage XML properly, you have to parse it. So imagine I have these two snippets of XML here up in the upper left. How hard could it be? The answer is, you don't want to know, because when you try to merge them, you come up with something like, "How you hard don't could want it to be no." And that was not your intention in merging them.

So Git doesn't naturally do a great job of merging Salesforce XML, which leads to a lot of pain. So I'm going to show you an actual merge of metadata in Salesforce. These are two XML files being merged in Salesforce.

So a little rough. Now, that's just for two files. When you have a lot of files, this is what it looks like. So this is an actual project that I worked on. Huge number of XML files being merged simultaneously, and as you can tell, it's pretty rough.

So if you really want to get Salesforce DevOps working, you want to have a DevOps process version control that works on Salesforce. What you need is the intersection of these three things. You need somebody who knows Salesforce. You need somebody who kind of knows DevOpsy stuff, whatever that means, broad term. And you need somebody who has time to write scripts. So as you can tell, it's a somewhat narrow intersection of those three categories.

05Scale, packaging, and commercial tools

So I had the good fortune with Copado, my employer, to run a couple of research reports. We had the very creative, clever name, State of Salesforce DevOps Report. Not ripping off anybody else's ideas, totally our idea.

The technical practices of DevOps, we found, are less mature than in the broader IT industry. The adoption of version control much lower and so forth. But the software delivery performance is actually comparable.

Because Salesforce is an inherently stable and fast platform. You can go a long way just building in a haphazard way before you really encounter problems.

But teams really struggle as they get bigger, and a lot of the companies that we work with at Copado, they have 100, 200, 500 Salesforce developers and admins working together in the same org. And one of the conclusions we noticed in here is that as teams scale up, their lead time really starts to tank. And the lead time, of course, that means time to value. That means a delay in delivering real business value on that Salesforce platform, which is the opposite of what Salesforce is supposed to do.

So one of the reasons why you have such a long lead time is because it's all one big monolithic database, and so everything gets really tightly coupled, and all of the components become highly interdependent. And so this is an actual graph I created of Salesforce components and how they're interrelated with one another and I tried to pull it apart with some colors, but as you can see, it's very complex.

So Salesforce launched this initiative of unlocked packages promising modularization. The idea is take that, what they call the happy soup of Salesforce metadata, I refer to this as the treacherous spider web of doom, and refactor that, sort of rationalize it as this package structure where while still complicated, it's not nearly as complicated a structure. Once you've divided things into packages, at least you can be clear what the interdependencies are. And so the idea is you take all of this mix of org metadata and you gradually fork it off into packages. Package one and package two, and you reduce the loose org metadata. And these packages basically operate the same way that any partner apps you might install from the Salesforce AppExchange would work. And you have less of this loose org metadata that's uncontained.

But the challenge is that the obstacles to packaging are legion. So it requires refactoring much of the code base.

And as with any refactoring effort, there's no immediate and obvious value add to the business. It would either stop or slow down other business initiatives, so it's hard to get investment and buy-in from the leadership teams. And it requires some sophisticated software engineering techniques. Now, there's brilliant people in the Salesforce world who figured out how to do this using solid principles, object-oriented design, and so forth, but those are not necessarily the majority of Salesforce developers. Or the majority of Salesforce developers are not necessarily at that skill level. And the development environments, just setting up these scratch orgs can be hard to provision in some cases. It's not necessarily an easy lift for all teams.

And then once you've got things into packages, you have a lot of difficulties with package interdependencies. Unlike something like Node.js, where every package has its own dependencies in a separate folder, you have this diamond dependency problem, as it's known in the Salesforce world, where you can only have one version of the base package installed in any Salesforce org. And so anyway, it's a little bit tricky to sort out those dependencies. And then even if you do have it figured out, then you have all these other problems with multi-pipeline packaging builds that are a little bit tricky to manage.

And so it's for all of these reasons that you have a number of commercial DevOps tools for Salesforce. So it's quite a niche. It's an unusual platform to develop on. Traditional tools don't work. So there's a unique market for commercial DevOps tools that help to address some of these challenges.

One of the main challenges that is commonly addressed is providing a graphical interface for these low-code Salesforce admins to track their work. So you can use version control even if you're a low-code Salesforce admin. That's very powerful and helpful. They provide a GUI for managing the flow of changes, for visualizing which change is in which environment. They handle the Salesforce metadata, all of that XML in a very intelligent way.

And it's important to understand that while Salesforce itself is sophisticated, there are a number of very sophisticated applications that are in turn built on the Salesforce platform.

So these applications built on the Salesforce platform, they have their own configuration. But that configuration is stored as data, as complex relational data. So they have their own networks of this complex relational data. So if you want to have a development life cycle for these complex applications for CPQ, configure price quote applications, or industry-specific apps, you need a way to move that data in a smooth way. These commercial tools, they also tend to provide tools for reporting and team collaboration, and some such as Copado, my employer.

This DevOps tool is actually a Salesforce app itself. And I know what you're thinking, that's a little bit meta, right? So it manages the Salesforce development life cycle, but it lives inside of Salesforce, so it's a Salesforce app. So it is customizable in a low-code way, the same as any other Salesforce app.

And so I'll give you a little bit of a visual of Copado in this case. You may not recognize it, but the screen that you're looking at is Salesforce. This is Salesforce itself. This is roughly their user interface. But the central diagram is your delivery pipeline. Your development environments, your testing environments, your production environments. So visualized in here, along with visualizing the flow of packets of changes, user story, quanta of changes, and the history of your deployments and so forth.

This also is a Salesforce user interface, and this is a value stream management tool actually built on the Salesforce platform to track metrics on your development progress in Salesforce. So how long do these user stories spend in the development phase and testing, and how often are they sent backwards and so forth. So as you can see, these are familiar concepts to the broader DevOps world, but they've been ported onto the Salesforce platform for the Salesforce platform. So lots of layers there, but hopefully that's giving you a picture.

06Closing

So if you're curious, if you like this topic, if you want to learn more, one great way to learn more about Salesforce is to go to Trailhead, Salesforce's free learning platform, trailhead.com. Awesome learning platform, gamified, fun, very accessible. Really excellent tool in terms of learning.

And if you search for DevOps, you'll find these two modules that I wrote together with Copado to explain DevOps for Salesforce in a little bit more detail. But there's all kinds of great content out there.

So now, thank you so much. Really grateful for your attention and time. Here's the help that I need.

If you're working on Salesforce, let me know. I'm really curious to know who here has been faced with this challenge of DevOps for Salesforce or managing Salesforce deployments.

Also, if this is sounding similar to any of the other platforms you're working on, SAP, Oracle, Siebel, ServiceNow, any of these other low-code platforms, let me know as well. I'm curious to see how you're handling those changes. So thanks so much for your attention. Really a delight to be with you today.