Andy Domeier

Hey everyone. Thanks for joining me today. My name is Andy Domeier. I'm the Senior Director of Technology Operations Engineering at SPS Commerce. I've been with SPS Commerce for about 18 years now. That's been quite a ride, and we've gotten the chance to solve some really interesting growth problems, both with our technology because our business has been really successful and grown over time, but also with our organization in terms of process and how you scale teams out, adopt DevOps around the globe, and a lot of really fun and interesting aspects to that journey that I'm hoping to share a bit of what we've learned on our journey at SPS, and hopefully it's of use to you as well.

What I'd like to do today is get into some details about something that I've found has helped us be really successful as an organization trying to tackle some of our technology goals throughout the last few years. And to do that, I think it makes the best sense to just talk through what SPS Commerce is and some of the things we've tackled and why I've seen them be really valuable for our organization.

Please keep the chat on the side in Slack active. Happy to chat through questions as we go through the presentation here. Hopefully we can have a lot of great hallway track as part of this as well.

So really quickly, SPS Commerce. Our mission as an organization is to connect all retail trading partners through the easiest-to-join-and-use network, freeing them to focus on what they do best. And I italicize that part because I want to tie back into it later in the presentation today. But at the end of the day, the easiest way to think about our business is we're a communication network for supply chain data.

And as you can imagine, there's a lot of complexity that goes into that, both with regards to availability and scale as you grow, and just complexity of navigating everything that goes into successfully communicating and organizing supply-chain-related events.

The story for SPS specifically is really a great success story. We have been growing consistently year over year, quarter over quarter for a very long time. And a big reason for that is we really solve meaningful customer problems with higher quality and better cost than they could solve themselves. We work really hard to make sure that the customers that we obtain stay with us for life, and a big part of them doing that is because we're going to offer really meaningful value. We're going to have great reliability and availability with our services and security as well. And this is all the same reasons that new customers choose us.

And so this idea builds on top of each other, and at the same time, while we're doing all these things, as our network grows, the opportunity for us to offer new services to the existing customer base that is a value-add for them, there's really a lot of significant opportunity there for us as well. And along the way we try to stay really responsibly profitable as a growing organization. We still work really hard to make sure that we're managing our costs. So I'm sharing this with you not to brag about SPS, but really just to help you understand what kind of culture and what kind of environment we're driving on as an organization and what we're building on.

And share a little bit about our technology journey. So like everybody else, back in the 2000s we're in physical data centers. We're trying to figure out how to scale out standard infrastructure patterns, getting VMware up at scale so that we have some more flexibility, pretty monolithic in terms of application architectures. And as our organization continued to grow, being in retail, one thing that you really need a lot of is ephemeral scaling: being able to scale into compute for the holiday seasons or specific major events, but not necessarily wanting to provision all of that hardware. So pretty stereotypical use case for trying to get to the cloud.

And in the process of doing that, we lifted and shifted a lot of our basic architecture in 2014 into AWS. We started dabbling with orchestration engines. We were using Mesos and Marathon for a while there. We weren't really containerized, so we were doing a lot of everything that we had inside just a JVM that we were scheduling. That really helped unblock our ability to get into the cloud and start thinking about scheduling workloads differently.

And in 2017, as most folks have gone through that cloud adoption fun, as they start to see that really big challenge on their operational expenses, 2017 brought a new focus for us to make sure that we remained responsible with our run rates in the cloud. And so we really needed to get more effective in being able to schedule workloads and be more effective with our compute. So that drove us into containers and serverless-related items. I say AWS adoption up here, really in general just the managed services that are offered in any cloud platform that help you be more efficient and cost-effective over time.

And just recently now, one of the aspects of our organization and business as a communications network, API strategies are really important to us. And for a lot of different reasons, but at scale, when you start to talk about opening up your platform via APIs, the need to have consistency, the need to have availability and reliability, both internally and externally, really challenged us to go back and visit some of our platform technologies. And in this space, it's really hard to argue that going the path of something like Kubernetes doesn't set yourself up for a better success trajectory. So we made some investments into Kubernetes, Istio, service mesh, to start to build out essentially what we viewed the next-generation compute platform for our organization to be.

As well with that, one of the things that we were targeting is, how do we be more resilient in the cloud, particularly at a region level? We were pretty well distributed across AZs, but pretty heavily tied into a specific region.

Along the way one of these comments that came out, I don't know if it was from our CTO Jamie. I give a lot of credit for this one. It's cited a lot, and I don't know if he had it before the day that he brought it up, but we were navigating trying to solve particular technology problems at scale. And a phrase he used that ended up being a really great guiding principle that we ask often is just essentially: where you choose to solve the problem matters.

I'll get into more details here, but I think you could probably step back and think about that in the context of your own technology and think about the different problems that you're trying to tackle, whether it's CI/CD or observability or compute in general or security. And where you decide to solve that does matter. That's a meaningful conversation to have, and I think that's what leads us down the path of what I want to talk about next here.

In the process of doing this, one of the things that's unavoidable in your discussion is going to be priority friction. So pretty obvious on paper: if you're trying to drive an organization where there's a great sense of technical pride and accountability, success requires your developers to be able to deeply focus on customer problems. That's something that we need them to do to make sure that we're driving business and delivering for our customers. But it also requires our same developers to deliver really highly available, secure, and cost-effective services.

So I call this friction instead of conflict because it isn't necessarily conflict in the sense that you have to pick one or the other. They're direct friction with each other. You have to do them both and figure out a way to make progress on that if you're going to grow, meet customer demands, while meeting all the operational basics that you need to deliver on.

So if you have that kind of friction now, what do you do as an organization that's either trying to scale or trying to transform digitally and see this inherent with this kind of friction? What's next?

The way that I would encourage folks to think about this is a new dynamic. It's a new customer dynamic that you want to consider injecting into your organization if you haven't already. So tying this back to that statement in our mission statement earlier, freeing them to focus on what they do best, part of our SPS mission. In our context, a lot of what we've done is looked at our organization. We have a really highly talented product engineering organization, and they've done a lot of really great things to deliver very valuable products to our customers.

As we've gone through this journey towards more of an operational model where our delivery teams or service-aligned teams own what they build and run what they build, we've seen this opportunity for us to start looking at those organizations as customers, and to start looking at what you could call foundational technologies or shared problems in more of a product-development viewpoint.

So now you have this new customer dynamic internally where you're viewing customers being the product engineers, and developers being those folks in the organization that are helping produce compute patterns and CI/CD patterns, security requirements, and creating this idea that there is an aspiration to deliver value to our product development team with services that solve these problems, but then also get product feedback from these developers as well to make sure that we have a nice iterative cycle.

So in the context here, the new product that we're talking about is really undifferentiated engineering. And that's a bit of a tongue twister, and you've maybe heard that phrase before, but maybe not, and so I'd like to unpack that a little bit. My perspective is that undifferentiated engineering is engineering effort that must occur to some degree for service delivery.

What I mean by that is in order for you to deliver a service, you have to deploy it, whether that's copy-paste, click and drag, or whether you have automation built in with GitHub webhooks and everything else that would go in with that. You have to get what you're building deployed somehow.

It has to be deployed to something. So this is where you could call infrastructure, compute. There's a lot of interchangeable words there, but at the end of the day, you have to run it from somewhere. It could be serverless in the cloud. That's still something you have to set up. You have to have networking to access those items. It could be just EC2 servers, or any server anywhere. It doesn't have to be in AWS or any cloud, for that matter.

You have to offer some security. If you don't, there's going to be problems. And of course, monitoring. And this is kind of funny. You might be able to argue down a little bit and say, well, it doesn't have to occur in order for you to deliver service. But I would argue even if your customer calls in to tell you your app is down, while that's a really poor form of monitoring, it is a form of monitoring.

So when we take that into context and recognize that everybody that's trying to solve customer problems has to solve for this type of engineering effort, one of the positions that I have on this is that undifferentiated engineering becomes one of the most valuable solutions to share within an organization, especially as you scale.

Thinking about it just from the perspective of deployment: if one team over here has their own Jenkins server, and somebody over there started on Drone way back in the day, and you've got some folks over here using GitHub Actions, there's a lot of discrepancy in the tools, but you've also now incurred a bunch of operational overhead because each of those teams is going to have to operate the deployment strategy that they've built to deploy their specific code. And there'll be a lot of complexity in collaboration with other teams. It's hard to share new ideas, new features, and functions if you haven't shared some of the core engineering foundations to solving deployment.

That's just one example. I think you can think of examples for infrastructure and everything else as well. And I think that if you look around the industry and look at the high-performing technology organizations, it's pretty clear that organizations that do this really well, that have maybe it's the centralized deployment pipeline or centralized compute network, you see them with a competitive advantage. They move faster. They solve problems with you to deliver value to customers, both in the context of the availability of their services and the security of their services, but also in the context of being able to react to the market. If they've got to get a new feature function out or have an idea that they want to bring a new product to the customer base and try it out, these organizations tend to have a lot more agility in that sense.

Before getting started, if this is an area where you feel like you don't have that product mindset around this undifferentiated engineering, a few things that I would really encourage you to consider that were very meaningful in SPS's journey of this.

The first is leadership alignment. That can sound pretty obvious and basic, but it is really important at a particular level in your organization to decide what competencies you want to invest in, what are things that your organization needs to be strong in today and tomorrow. And then think about what kind of partnerships you have today. Are there partnerships with SaaS tools that you plan to build on? Are there strengths in your organization today, whether it's with really, really advanced infrastructure cloud engineering teams that you can build on, that you think are core competencies you want in the future as well, that you might want to scale out?

Then think about future business priorities. Something that really resonates with me is that region aspect, region resiliency aspect for SPS, that future business priority. For us, it made a lot of sense in the context of being able to invest more in those compute platform building blocks, being able to recognize that there's a business priority that we need to continue bringing more resiliency to our applications and doing that within the compute layer is pretty obvious. You're going to have to build to do that. If everybody is operating their own compute layers with their own approach to where their services run, being able to then, at scale with your organization, offer that kind of large event for region resiliency is really complicated.

Budget's another one, but just another good example. That was the original reason we really drove into containers as hard as we did. So those future business priorities that are coming up can help you align some of those targets.

And then the other one: we can all fall into the trap of what do you want versus need to be good at. There's a lot of great, cool technologies out there that make a lot of sense for us to go deep in or make investments in internally. But at the end of the day, making sure that we are aligning our investments, both in our time and energy, to things we really need to be good at versus things we want to be good at is important. And it's important to do that all the way up the leadership ladder. You don't want folks really inconsistent on that, or I think that it can really create the struggle.

The next thing, this is something that you have to be really mindful of up front if you plan to try to target investing in the mindset that you want to create around this change. It's really easy to write a few of these things down. At least what we found is, on paper a lot of the stuff is going to make a lot of sense, so everybody's just going to love it, right?

Making sure that there is clear reason why you want to align the organization around sharing particular problems and working to establish this aspiration that these folks that aren't used to being customers are going to have a feedback loop, they are going to be users of a new product internally: it takes a lot of time to build that. And to start incubating that mindset, start that communication process, get folks' heads wrapped around it, if you haven't started that already, it's a really important foundation. It's not going to happen overnight. It's going to take a lot of time.

So some success factors for us. I wanted to just share a few things that I think went really well in our journey so far. I don't think we're done. I don't know if anyone ever is in these types of investments.

The first is just being able to align vision and value to some context of the functions that you want to make investments in and then commit. This probably sounds kind of basic, but if you think about it and you go through this process of different aspects of your organization and what's important to you, I kind of joke with this priority column on this table, the idea that everything's a top priority because everything's always a top priority, right? But there's a lot of different things that your organization may find valuable. These are some. I'm sure there's tons that you could think of that are on here, but operational expenses at the bottom is always top of mind for a lot of folks.

But if you can build out this list and really think about which priorities are, and then maybe think about the undifferentiated engineering you have in your organization, I like to stick with these four examples. You probably have other examples, or you may even want to break these down into more specific buckets. But it's a nice easy way to get that out there and have a really meaningful discussion. Well, do we need to ship features faster? Do we need to have a different strategy for how we expose our APIs publicly? Do we need to continue investing in what kind of availability we have? Is the business growing? Do we need more scale? Of course, OpEx is there. But having that conversation in a framework like this can help you decide where does it make sense to invest first, or where does it make sense to share first in some cases.

Maybe it doesn't make sense to double down on CI/CD if your main concerns are operational expense and availability. There's probably a lot of things you could do with your infrastructure and observability tools that's going to take you a long way. So I think making sure that you have some alignment to that value proposition that you need is really important.

The next one I think is pretty sneaky. So this is my son. He can be a little boy, but he could also be a tiger. The point here is that it's really, really important to recognize that good product development requires a number of roles. And I think that this is something that is really easily lost or skipped in this effort.

At the end of the day, if you think about it, when you're doing product, you really need some function of product management or product ownership. You need to launch things that you build. You need to establish some sort of relations with your users. I think DevRel, in the sense of a lot of this housed out there these days, is kind of an interesting analogous role to this kind of function internally. Customer success: this is an interesting one too. How did folks get questions answered? Is it really friction-filled? Is everything a ticket? How do you like it when you work with organizations where it's fire a ticket away and hear back in three days when you just have a simple question? Those are things internally in your walls you can really start to think about.

And so the one thing that I think prevents folks from taking a step forward, in my experience, is that they feel like these functions have to be job titles right away. Yes, you definitely want to work towards that. But if you're not doing this kind of work today, if you're not thinking about these internal engineering efforts as something that is a product and you want to start that process, you're going to wait a long time if you're going to try to advocate for a technical product management or product owner role that's never existed in your organization and you want it to start.

That's going to set you back a long time. If you just start to think about what you want the accountabilities of that role to look like and then think about who in the organization might be able to start stepping in and helping out with that. We saw at SPS our managers made great product owners. Our on-call engineers make awesome customer success partners. It's not often that you want to pile on a lot of planned work in an engineer's role if they're expected to get disrupted a lot, and so having them kind of double down on customer success during on-call shifts has been pretty successful for us.

And then just establish some reasonable expectations. What does success look like? This isn't your full-time job, but let's just be explicit. Hey, technical manager, we really need you to be the product owner for this effort that we're leading. Or hey, principal engineer, we think that you're in a great position to really help lead the dev relations aspect of what we're trying to roll out and get these engineering teams all rallied around sharing.

Another success factor. This one's probably pretty obvious, and you probably hear a lot of it at this kind of conference. This is not a waterfall IT project. Don't try to run it like one. Implement a product development strategy. Think about how to create a roadmap that's transparent. Establish clear means for customers to give you feedback. Measure the impact of your work. Run it like a product.

Set some expectations as you get rolling on this. You're going to have some debt you're going to have to navigate if you are going to start building some new services that aren't out there yet, or maybe try to start to centralize something that isn't centralized. You're going to have to make some decisions on that, and just acknowledge that there's going to be some challenges out of the gate.

And new tech is going to have a learning curve. If you're asking folks to move off of, let's say, somebody's been using Jenkins for the last four years for CI/CD and you want to get them into something else, that is going to be an event that is not going to feel better at first. Making sure that you're giving them confidence that you're there to help them through the process, but then also a good support channel.

Next, this idea of gravity I think can be really important in this process. Create gravity with your technologies. The idea here is that you want to create services and efforts that pull folks into your tech, not necessarily push them. Think about ways that you can start to build these technologies to solve business goals.

Like I mentioned earlier, the region aspect. We were asking everybody to really focus on region resiliency. Well, if you're offering a platform that has that baked in, now there's a really meaningful reason to adopt those platform technologies along with trying to solve for this larger technology goal or initiative.

There's a lot of headwinds you're going to face. A few that I feel are worth calling out that we dealt with or try to navigate: one is priority conflict and friction. We talked about that a little bit. It's unavoidable. Be empathetic. It's going to be hard. You're going to be asking folks to learn something new in some cases when they have competing priorities. And so that's going to be a challenge for them, trying to navigate learning those things while the business is asking them to deliver on something. It can be hard.

But once you get over that hump, you start to see some really meaningful things. Their velocity is going to increase. They're going to start to share their experiences with others. You're going to see some momentum build.

And then from a culture concern, common negative reactions here can be: shared patterns and tools, but my way is better, right? The perception of a loss of control frustrates anybody. And again, be empathetic to that. Similarly, that loss of control: does it restrict my ability to be innovative? Of course, we want innovation, but innovation with purpose.

I think here's where I would say innovation with purpose is something that we've really latched on to at SPS. It's not that we don't want you to be innovative. It's just that we want your innovations to be applied to customer problems, not necessarily some complex monitoring tool if you're a product engineer.

And some tailwinds that come in with this is the larger and more talented the team is, the more costly it is to work on the wrong thing. Most folks can respect that. Fighting the current is not productive. Duplicating effort for something like CI/CD tools or compute platforms isn't productive. Momentum really simplifies everything. Once you have that current that's pulling you down the river, it's a lot easier to go.

So a few things just directionally that I encourage you all to be mindful of in this journey is that it is really important to be explicit on what you choose to share. It's really hard to create a product if you're not explicit about what the product is. Have that up front. Be really clear what you're trying to deliver and what you're asking folks to share in terms of internal products and solutions.

Explain the importance of the requirements you're asking. Does it connect to a larger business goal like cost or security? Does it connect to a larger business goal that you believe over the course of time this is going to help you with feature velocity?

Communication is not precise. Don't expect everybody to read the first email or watch the first PowerPoint and just know for the next 18 months that you're on this journey, two years or whatever it looks like, that they're just going to remember why you're here or listen the first time. It takes time. It takes repetition. But focus helps build momentum. So if you can align everybody on a particular goal, whether that's compute or CI/CD or observability or whatever those items are, you'll see momentum and it'll be really meaningful.

I'm going to stop there. I have so much more to say on this topic, but this was at least a high level enough worth sharing, and hopefully you all found some value out of it. I'm happy to continue the conversation on Slack, on Twitter, or LinkedIn. I think this is a really interesting and important space for all of us to be sharing what we're finding success in and growing together along the way. So thank you so much.