Randy Shoup & Scott Prugh (Las Vegas 2022)
EXCLUSIVEAn exclusive interview from DevOps Enterprise Summit Las Vegas 2022.
Full transcript
The complete talk — auto-generated from the talk's captions.
Hi, Randy. Hi, Scott. What's Going on? Not so much.
Oh, so we wanna talk architecture today, I Guess. Sounds good. Yeah. Um, so I think, you know, one of the things I've been trying to think about is how, like architecture plays in more of our decisions in like organizational design and, um, beyond just kind of the, the technology.
Like what, what are your thoughts on that? Like, how, how it plays in? Yeah. 100%.
I think, I think a lot of times in our industry we think about architecture as like this very limited part. Like it's, oh, this is about designing systems in the large, or something like that. Do you know what I mean? And I think what you're, where you're going, and I 100% agree with it, is architecture.
The practice of architecture is really like sociotechnical systems thinking. It's really, and, and that system is often a distributed software system. And like, we should, we could talk about that and like, that's maybe the core idea, but it's also about architect, uh, about, uh, organization. I think it's also about like how the different parts of our organization or our process work together.
Yeah. You know, you said very well in your talk, uh, you know, we wanna think about architecting the system, we wanna think about architecting the organization when I think about architecting the processes and practices and Yeah. I think 100%. Um, yeah, and I've had, um, I've had architect in my title for like, maybe half my career, and there's a part of me that's a little bit embarrassed by that, frankly.
Um, because I think we, and I'm putting myself in this, we have made architecture like this very limited, like side ivory tower thing, and, and, and I think it should be, uh, no, this is about like, making the system better. Like what's the job of architecture? Yeah. Even if nobody has the title architect.
And that would make me happy, frankly. Um, we have an architecture of our organization and our system and our practices. Yeah. Um, and our job, whoever is gonna be architecting that, and hopefully that's a lot of us altogether, is we just wanna make it better.
Yeah. And I, I think, I mean, my, my thought is that it's why it kind of like the idea of architecture and kind of transformational leadership kind of pops up top as one of the key, um, I guess high performance capabilities because it drives so many things, right? So when you get the siloed, like, Hey, we've got a bunch of architects that are designing microservices for the teams or whatever, off to the side, it doesn't, if it doesn't take into account those other concerns, like, for example, the organizational structure or like the process and work structure even in, in like, how do you distribute work or even how you decide like, can the work get done Right? Then those types of things kind of create all these de optimizations because all that hasn't been designed together to kind of think about, you know, okay, can we write the code, but can we actually, you know, get the work done across the organization in a way that's even effective?
Right. Yeah. I love that. Um, yeah, I mean the, the several things that makes me think of is the purpose of all the things that we're doing is like driving business and customer value and like, you're not suggesting otherwise, but like none of our work, whether we call it architecture or organization or HR or whatever, like none of our work matters if we're not driving, you know, the ultimate goals of the business and, and the customer.
So that's number one. We all should have that same shared outcome in mind. And two, which I, I love your connection between architecture maybe on the one side and like transformational leadership on the other. And I think what you're saying is that you can't have either without the other.
And I so strongly, I so strongly agree with that. Like, if you try to transform an organization without understanding the structure of it, like whether that, again, that structure is organizational or technological or whatever, like you're not gonna be successful. And conversely, exactly what you were saying, if you go in and say, I'm gonna re-architect, I'm gonna reorg, or I'm gonna re-architect the system, and you also don't take into account the context of all these other things, you're also gonna fail. Right?
So it's all about the, I mean, it's the systems thinking like in every one of these dimensions, do you know what I mean? All together Yeah. AG agreed. And some of the, uh, the other stuff, I was talking to Scott Ella at lunch about just work management and how much of that ends up being a challenge of trying, one is deciding the priorities, but then the other is really understanding I can, I call how the work works.
And then, you know, if, if you have work initiatives or you know, some folks home epics, whatever they are, if they take large scale dependencies on tens or hundreds of teams and you don't understand first understand that it's happening, right? Then that's the first problem. But then if we are not then looking at like, how do we redesign either the work process system, right? Do we limit lip wip?
You can only put so many things at a time, or the team structures or how people do the work or distribute the work, you're then always bound and coupled by this, you know, incredibly burdensome set of coordination costs, and then it's just impossible to get things done. Yeah. And I think, um, uh, kind of riffing on that point, I think just to say what you said in a different way, I think a lot of the reasons why large organizations find things challenging is because they have a lot of dependencies. They don't manage them, or they don't even see them.
Yeah. Right. Or to be fair, the they in an individ, the individual teams know them, but it's not surface to decision makers or like, or the decisions are made without regard whoever's making the decisions and in what form and in what order. They're not made with the explicit understanding that, oh, by the way, if I ask your team to do this thing, Scott, you know, there are other Scott and, and Bob and Susan and Courtney and all their, you know, all their teams also have to do this thing.
Yeah. And if you like, pretend that, you know, ah, like pretend that that doesn't exist. Like no wonder things don't work. Yeah.
And, and, and so I think, I think there's something in there about like the average complex organization, like not doing that particularly well and not thinking that, and that it's not because you're large and complex that you can't do it, but I just think that it ex I just think in the world as it exists right now, yeah. A a lot of people in organizations aren't thinking carefully end-to-end systems thinking, et cetera. But those outlier organiz, I mean, the high performing organizations that are at large scale, like they do think about that. They do think about, you know, the Amazons, the Googles, the Netflixes of the world, like they are explicitly and implicitly inculcating into the way that they work.
Like Yeah. You know, the teams are don't have much resources individually, they're divided in this way. They interact in this like very clean mechanism. Yeah.
And so it's not like you can make, you're not suggesting this, it's not like you can make dependencies go away. You can't make them go away, but you can refactor the organization to, to minimize the impact. Right. And maybe sometimes make some of them go away by combining or moving things around or whatever.
Yeah. I, well, I, I do think, I mean, the, the one thing I was kinda getting in my talk is that there are, you know, self-service is one of the most powerful things to get to actually remove a, a dependency. And just when you talk about, you know, simple things like technical coupling or a combination of technical and process coupling that, you know, a good example is patching a server. So patching a server usually involves the essays, the system administrators, someone and, uh, generally the application owners.
And so those are coupled with a dependency that now you have to coordinate two groups to arrive at the same time, but with self-service, allow people to patch their own servers. Now I remove kind of that dependency. So that actually makes the dependency go Away. Right.
It, it's interesting 'cause or at least it's A lighter weight dependency. Yeah, Yeah, yeah, yeah. Yeah. Okay.
So that, that's, that's my reframing of what you say. I 100% agree on the technique and the result. But the other way I would say it is like, oh, I mean, like, my service still depends on your other services. Sure.
Like, I, like, I still run, you know, my service still runs on the operating system. It needs to be patched, and that's managed by this other team. But what, but, but the self-service is, you've, the dependency is still there, but you've made it not a problem. Right.
You've broken the, I don't know the word for it. Like you've broken the coupling. The coupling, exactly. Right.
Yeah. The dependency is there, but they're not coupled in time and space. Yeah. 100% Coupled in time and space to arrive at the same time at the same space.
And self-service allows you to choose which time And which space want To, you know, arrive at, right? Yeah. Yeah. 100%.
Then you Don't have to coordinate those two people. Yeah, 100%. And I think, sorry, just to riff off that, and I think that's such a great more detailed explanation why, again, a bunch of these like really effective, high performing, large, really large scale organizations work. It's not that they don't have dependencies, it's that, that it's that they've worked outweighs that those dependencies don't block you and aren't, and don't couple teams, right?
So, so much, you know, self-service and good documentation or just com just complete automation. Like another way to solve the patching problem, which is common in lots of places, is like, you know, there's this pipeline of like, we're gonna deliver the software and we hopefully deliver the software in some regular cadence. And then that patch just gets injected into like, oh, this is part of your image. And like, chug, chug, chug, chug, jugg, we run the same old damn thing.
Yeah. But, but that in essence is architecture too, right? You're architecting that concern of patching into some other process that now becomes a regular predictable cadence that does not have to cause this disruption. Like, oh geez, you gotta patch this on Wednesday night Exactly.
This time. Because, you know, there might be a Yeah, I think that's ex I mean we're, we're agreeing a hundred percent. I think that's exactly architecture. I mean, and the, and it's funny because if you went to a lot of people that have architects in their titles, and like, I think maybe there should be less of us in the world.
I think everybody should be doing ar not because I don't think architecture's important, but because I think it's so important everybody should be doing it. Yeah. In the same way as like, I don't think we should have a lot of agile and DevOps in people's titles either. But that's another argument.
Um, uh, I think a lot of architects would say, oh, that patching thing, that doesn't, that's not what I do. Yeah. I'm like, wrong dude. Yeah.
Well, I mean, that comes maybe a little bit to this whole siloed nature that I think folks are, and you know, Jonathan Smart talked about it a bit, right? The, there was a lot of siloing skills in all these kind of different areas. And, and so many things fall into that today. Um, and if you don't have kind of broad thinkers, by definition, it's gonna be very difficult to actually solve the, the problems at the larger scale, right?
Because if you're just solving architecture as, can I design a database or can I design web service a p i, they're they, they do their job, right? But if that doesn't fit into the higher level organizational constructs in a way that are effective, right? You design some service that now requires dependency coordination across four other teams, you could have actually made the system worse. Yeah.
Because now you have the, the, you know, the either singular or bi-directional coupling, like a changes and BSS to change and vice versa. And you have to regression test all that stuff together to reach kind of coherency. Now it's worse. Yeah.
Now I have another team, another technology, all kinds of other stuff to manage that may not have actually approved Overall. Yeah. And I think that when we practice, when we practice architecture in the way that we're talking about, again, I think it's systems thinking at the, at the, at the broader level, um, exactly what you're saying where like, oh, I made a micro, maybe optim, maybe optimization to this particular, uh, component of the system, but actually I didn't appreciate or didn't care about or, you know, whatever didn't, didn't consider, um, you know, the other, the other effects. And like that's, I'm saying what you're saying, like that's a fail.
Yeah. Right? And so the architecture sensibility, the, like sy you have to be a systems thinker. You have to go, I'm not just solving this, like whatever the example you were giving the patching problem or something like that.
Like no, I need to think about it in, uh, in, in context. And it's not just you were saying that, it's like, I could, I could maybe I'm not considering those other things. I think, I think it's not just that we want to consider those other parts, but like they're part of the system and maybe we could change those ideas. Yeah.
So the example I always, I always use is when I used to run engineering at Stitch Fix, like clothing retailer, we had physical warehouses around the US Cool. Um, and we had people that ran those physical warehouses and many times, and one of my teams built software for those warehouses, um, which was for a lot of fun, a lot. We would be constantly hearing, you know, we need to change this, we need to do that. And a lot of times the the, the request would come back, Hey, can you like please add this button or add this new phase?
And we're like, okay, but let's have a quick conversation about like what you're, what problem you're trying to solve. And a lot of times they knew exactly what they wanted and we just gave it to 'em. But non-zero amount of times we would just be talking to them and it's like, well, you know, what, if you like just changed your business process in this other way, you wouldn't need that and it would be simpler for these other reasons too. Do you know what I mean?
Oh, yeah. And it was like, it was this col and like sometimes they would do that to us by the way too. So it wasn't just one directional, but like systems thinking, like thinking, seeing the whole board and going, well, what if we did that? Wouldn't that relieve your problem?
I'm like, oh my God. Yes. And, and that also might be a, a variant of kind of this information distortion problem. You know, I, I kinda mentioned like how, you know, tacit information lost through handoffs, right?
And I, I've just seen a lot of this where you get some requirements and right. They kind of come all pre-canned and someone spent weeks on elaborating all these hundreds of scenarios and they finally make it to the engineering teams. You know, six months later, engineering teams get 'em and they dutifully like go out building a bunch of features and they can deliver to the customer, and it's some really awkward right? But it kind of represents the requirements that they had and all that stuff.
And then you end up go talking to the customer, it's like, well, what do you, what? Like, what is wrong? Like, what is your problem? And they're like, oh, it's this.
And you're like, well, like we have a flag for that. Or like, uh, you could have just turned this on or, but that kind of distortion and the inability to get feedback on just even like what the problem is because it's being transmitted. The, the requests are being transmitted through so many different handoffs, and then everyone has a different understanding of not just the problem space, but also the solution space, i e the system of actually how it really works. Right.
And, and, but it could or could not do. And then you get this like preconceived, uh, you know, and I think Paul Gaffney has some great discussions. This like, you get this preconceived, like you need to replace the warehouse management systems because you need inventory. And it's, you know, really all they needed to do is to have, you know, better, you know, cards on all the, um, all the, um, pallets so that they knew how much was there and what it was.
And that was something that could be done in a couple days as opposed to the preconceived notion was replace all the warehouse systems. Yeah. There's there, like you say, there's no substitute for really understanding the customer problem. How does one do that?
It's proximity to the customer. Like if I'm not the actual customer, which I rarely have been in my, in my career, sometimes if you're developing developer tools, like you can kind of understand that you're the, you're the customer, but like 99% of the time, like we're not the customer of the software that we build. And so we have a limited understanding of the problems that Yeah. And the only way to get that understanding is to as, as best that we can kind of go direct to the customer and like ask or watch or observe.
Yeah. And if we take some of that back up to the, the architecture kind of thoughts we started with, it's that, you know, the design of kind of the organization and the information flows was inadvertently or inadvertently either way, like designed in a certain way to get that result Right. You know, to to pass the information through three or four groups before it gets to the, the group that can actually act on it. Yeah.
You know, and without addressing that design flaw or that architecture, it continues to be extremely difficult to get the fidelity or the information in the system that you actually want. So there is a good piece of that comes back to architecture at all those different system levels that the process, the organization and even, and the software technology of course. Um, but also the, the reality of those coordination costs and not like trying to design with those in mind, right? Yeah.
They have a, they have, they have some, they have a cost. Yeah. I mean the, the, the talk that you gave, I thought was so revelatory on really digging into like, we kind of all have this sense of like, coordination is a challenge, but like really being clear on every handoff is adding, you know, a huge amount of information loss and a huge amount of additional risk to the project. And I thought that was every one of those that I'm just gonna reflect what you said is we're losing half the information at every, at every stage, and then we're doubling our, we're doubling our risk at, at every stage, which, which is, is sobering, right?
Um, and also like, again, that's what, that's what I think a lot of us experience in organizations that, you know, are large and are, for lack of a better word, like not well designed nobody, 'cause nobody took the time to like really do the systems thinking idea. And, and, um, yeah, I mean, kind of like what you're saying, like we, we've all, you know, we all have this like, um, uh, a lot of us have this, um, have this, uh, idea that the most efficient way to do things is to like, um, have everybody do like one little, like, you know, industrial thinking. Yeah. Like we each, I'm gonna screw, I'm gonna put the screw in and you're gonna like turn the screw and then we're gonna, someone's gonna, you know, hit something with a hammer or whatever.
And then, and um, and maybe that's not the most efficient way to even build cars, let alone build software. Yeah, no, it's a, it's a challenge with those, uh, that solid handoff. 'cause it's really hard to get the, the more kind of system level optimization and get the learning kind of, uh, uh, uh, across those folks. 'cause the boundaries prevent kind of that learning.
So. Yeah. Yeah. I mean, I think the, just to double on something that you said earlier, you know, um, just the, that, the, that the only way we're gonna, we're always gonna be imperfect.
Like we're never, you're never gonna like design something that's perfect and like never needs to be changed. I don't think, I don't think that actually really exists in the world. Um, there's only, I tried it and then I can make it better and I make, make, make it better and I make it better. Um, but how do we know how to make it better?
Or how do we know when we have to make it better? It's 100% what you said, which is the feedback, right? We need to be whoever we're, you know, building this thing for, we need to be hearing that feedback from them and uh, and then, you know, using our understanding of the system. Like, okay, well how can I adjust it to make it better?
You know? Yeah, For sure. So, alright. It was a good time.