Log in to watch

Log in or create a free account to watch this video.

Log in
Las Vegas 2018
Share
Download slides

Crowdsourcing Technology Governance

In the past, architecture choices, ensuring quality, and security and compliance technologies were the ways to enforce governance through tollgates, centralized approvals, and silos. Three years ago we started recommend_tech, which uses crowdsourcing to make technology choices at Target.


Dan Cundiff (@pmotch) is a Principal Engineer at Target Corporation. For the last 5 years he worked on api.target.com. Today he's building Target's IoT platform. Dan is an advocate for the use of and contribution to open source software at Target and was one of the founding members of Target's open source office. Previously Dan has spoken at GitHub Universe, Linux Foundation Open Source Leadership Summit, API Strategy Conference, Cassandra Summit, Jenkins User Conference, RSA Conference, Gartner AADI, and more.


Levi Geinert (@levi_online) is a Director of Engineering in the Target Dojo but recently transitioned from Principal Engineer and is responsible for supporting, and refining the agile and DevOps software engineering practices and patterns. He advocates for Developers, and also helps to operate the Target Dojo which has helped drive DevOps adoption across Target. Previously Levi has spoken at ChefConf, #That Conference, multiple smaller venues and internal Target conferences.


Lucas Rettig (@LukeRetigg) is a Principal Product Owner at Target Corporation. He currently owns Target's Item-Location service, the Item and Location Grouping service, the GraphQL platform, and Autobahn, a data movement and caching platform. Prior, he was the Product Director of Supply Chain at Infor Retail and before that spent several years in engineering at Target. Luke has a deep-rooted passion for Product Management and is an advocate for engineering enablement.

Chapters

Full transcript

The complete talk, organized by section.

Levi Geinert

Good morning. How's everyone doing?

I'm very excited to be here today to talk about something that's very near and dear to me. A little bit about me: I'm a director of engineering in the Target Dojo, where we help teams with guidance on engineering choices as well as changing the way they work.

I'm going to talk a little bit about how we moved from a governance model to a guidance model, and how that's helped support the empowerment of our engineers and our business to make smart decisions, take smart risks, and be empowered. We're going to talk about a real example of how we've implemented this by crowdsourcing some technology guidance, and then we're going to talk about how that helps us create awesome products.

A little bit about Target: we're big. We have a lot of stores. We give back to our community in many ways. The interesting thing to note here is that we have a lot of team members, over 300 product teams, as well as 4,500-plus people in technology. This is why guidance is important to us.

A little bit about our evolution: we started our journey with DevOps by rebuilding our engineering culture. Ross Clanton and Heather Mickman presented on this in 2015. Feel free to check that out at some point. It was grounded in empowerment of our engineers, DevOps, Agile, and so forth. This had a very big impact on our engineering capabilities at Target. DevOps and automation were key components of that.

After the DevOps movement, if you will, we got into changing the way we work. The Dojo has supported this. Many of our business today is from business teams that want to change the way they work. They're seeing how our engineering teams are empowered and taking risks, and they're being supported in doing so, and they have interest in that. They also want to understand how to interact better with their engineering teams that are operating with Agile, Scrum, or Kanban. Today, many of our challenges are focused on business challenges and not just DevOps.

Finally, we realized that to have great products, we needed to have our customer at the table. Why are we talking about this at a DevOps conference? I'll show you a tweet that's one of my favorites. Caitlin mentioned that it sucks when you have an idea for a product, but it's much worse when you build the wrong product, right? And so aligning on our customers, internal or external, was very important for us, again, to build those awesome products.

All right, so we talked about governance and why that was something that we wanted to get out of the way. We don't want to require teams on how they should work, how they should operate, what tools they should use, because when we empower them and we make them accountable, then they choose the right tools. We know this because we have research to support it nowadays. Thank you, Dr. Forsgren. And I would extend this to not just tools, but languages, frameworks, and all the choices that teams need to make to accomplish and build those awesome products.

Now, it's great to remove governance, but what happens when you don't provide guidance? We know that it can get very noisy in a tech landscape. If you do any searching of images around tool choice, it can be overwhelming. Teams that are given no guidance have to consider every possible option. So guidance allows us to streamline and reduce the friction required to make those choices.

Now, what does guidance look like? There are a few things we'll show you on that, but I'll talk first about some of the principles. We think the guidance should be accessible: available to anyone to contribute, not a specific leader, but anyone at any level. We think it should be transparent. We hope that anyone can view it. It's not a great situation if the engineers aren't able to find out what are the approved patterns or what are the best tools that they should be using. It needs to be able to change. We know that this stuff changes very quickly in our industry: tools, languages, frameworks, et cetera. And finally, cultural. Just like DevOps, our guidance model needs to be cultural. The people that are providing the guidance need to also support and promote other leaders to step up and help guide.

In many cases, our Dojo is not the expert, but we maybe have an awareness of what people are doing, what patterns are being used. We'll bring in those teams to actually help the teams that are considering a specific technology or framework.

I'll also mention the Dojo. You may have heard about it. It's a big part of our culture. It helps us, again, empower teams, support them to take smart risks. I want to announce that we've recently had over 350 engagements in our Dojo; 250 of those are 30-day challenges. We've had over 180 tours of our Dojo from many of your companies, as well as internal teams. Again, trying to help guide people towards positive change through empowerment.

I'm going to pass the baton to Dan, where he's going to give you an example of how we crowdsource our technology guidance. Thank you.

Dan Cundiff

Thanks, Levi. I'm Dan Cundiff. I'm a principal engineer at Target Corporation. Today what I want to show you is actually how we take what Levi talked about, this idea of empowering engineers and the idea of going from governance to guidance. I want to show you a real-world example of that in action.

But first, to set the stage for what we created, I have to tell you how we did things before. Maybe this is actually a familiar process at your company, so you may be able to empathize with this. Before, we had a centralized group that would choose technologies for us. We called it an architectural review board, an ARB. What's difficult about these centralized groups is that they have to make decisions for many different product teams. The larger the company you have, the more likelihood that they're probably not going to be able to pick the technology that works for everyone or be able to take into account all the different product teams' unique things they need. That becomes an issue.

The other one is that if you are someone who wants to be able to influence that ARB or that centralized group that's doing that governance, it's probably going to be pretty hard to do, because you have to go through their process. They probably meet on a regular basis. They have some form you've got to fill out and things like that. So these things are problems.

What we did about four years ago, when engineers were starting to feel empowered, was a couple of us got together and said, "Hey, this can be done better. Let's try and disrupt this. Can we do this in a different way?" We just created a repo in GitHub that was a simple list of technology choices, and we wanted to use that as a stage to make a couple of new disruptive choices and see if this thing could take off.

So we did. We created this repo. It used to be just a single file in here; it has been broken up into different categories. What you see here in this list is a different set of technology categories: collaboration tools, application frameworks, caching, data stores, et cetera. You can go into any one of these files here. Let's take a look at the data stores one. There is typically some front matter on the file, just like an introduction, some motivations for why the people who maintain this file make the choices that they do. But if you're a developer and you're coming here saying, "Hey, what choices can I make, or how can I influence this document?" you usually just scroll down a little bit further and you'll see different sections.

Again, this is in the data stores one. You'll actually see what the technologies are and what their status is. So let's say you're looking for a relational database store and you're wanting to see what your options are. Postgres is a choice here. The other fields to the right, the use case and the other stuff, that's more like metadata. But the real important one is the status. There is recommended, limited use, and do not use. You're going to be tempted if you try to do this yourself to want all kinds of other statuses, but keep it simple. Recommended means you can use it. If it's limited use, there's probably some reason why: maybe licensing constraints, cost, or maybe it doesn't implement well in certain environments inside of Target. The last one is do not use. Don't use this. You shouldn't use that here at Target. In fact, we've put things on this list that maybe we've never used at Target, and we just want to put it here and make sure someone doesn't think they can go choose it. And we'll explain why. We'll say why it's not a good fit.

Let's talk about the motivations for why we did this. There is a theme here, I think, when using GitHub or Bitbucket or GitLab. Your developers are already day-to-day in GitHub. They're writing code. They're doing social coding. This is where they're at already. So if you want to do something like this and you want it to empower your engineers, put it somewhere where they're already working. Put it in version control. There are a lot of benefits to that.

For people who are familiar with the pull request workflow, this is: I have a change, and I am requesting that you pull in my work. Anybody at Target, whether you are a junior engineer or a more senior person, can open up a pull request to any of these technology categories and suggest a change. We've had junior engineers do this who maybe have learned something new and want to bring it forward and say, "Hey, can we start using this? I think this is a gap in our technology set." Some people have come forward and had to make very large changes, like we need to change an operating system or a much bigger technology choice that requires a pretty big discussion. So you can do this through pull requests, and it's a perfect workflow for this. Everyone can comment and have a discussion on it, and when it gets merged, then that's it. The technology choice is final. I'll show an example of that here in just a minute.

Another nice benefit is if you're putting it in Git, you can go to any one of these Markdown files and run Git blame on it and see why it changed. This example up here is for Kafka Streams. I can see the actual commit that was done. I can go back and see the pull requests and any associated issues with it. For posterity, I have everything captured to understand, say, a year later, why did we make this choice? It's all just right there in history in Git.

Here's an example pull request. This is a person; his name's Dan Moss. He is actually suggesting a new database technology. You can see his justification up there. He's made a case for it, a sort of pitch, and it started a discussion. This was only a couple of days old, and it already had a lot of comments on it. So this is exactly how it takes place.

Also take note that there will be people, and there is a whole spectrum of people, who will chime in. There are some who will be very motivated to make comments. There are also some people who just want to listen, but they'll react with emojis. You can see that there are three thumbs up. There are some comments in some of these pull requests where a lot of people will add all kinds of input, whether it's comments or different emoji to show their support or which way they're leaning. Take note of the label up there, the CIO forum inputs. I'll talk about that here in a minute.

Because it's in Git, then we can do all kinds of other things, too, right? Just like when you are working on a project and you're putting code in there, it can trigger workflows and stuff like that. It's good to point out that there are other things that you can do. As having an interface into a process like this, you can have the alerts sent to Slack. You can have some sort of job kick off which maybe renders these in PDFs, which maybe is more digestible for certain people. The good thing is that you can stitch all kinds of alerts and nice things to workflows.

This is an important thing I want to bring up: having seen this exist for four years now, resist complexity. There is going to be a real temptation to say, "Hey, we could add more status categories. We could do this process to this. We could really require all these additional steps and things like that." Don't do that, because at the end of the day, it is developers who are coming here and recommending these changes. If you make it too difficult to do, they're not going to come here and do this. Absolutely resist it.

I think the only changes we did: we used to have that single large file. It was nice to just do Control-F on, but it got pretty cumbersome to vertically scroll so much, so we did split it up. There have been some other helpful things that people have added, like decorating some of these pages with motivation and the reasons why we've made these choices and just more clarity on things. But other than that, it's always stayed pretty simple. It's just a Git repo. You can open up a pull request. You make your pitch, and if it gets merged, it goes.

In the earliest days, it was volunteer-led. There is a set of influencers probably in all companies, and there certainly was here when we started this. It's volunteer-led, but make sure over time you keep it fresh and keep new people coming into it. You don't want someone to have a real dominant voice. If someone's going to suggest a change to this, you don't want to feel like they're coming into a conversation with bias or anything like that. So always keep a fresh set of people coming into this and helping groom it and stuff like that. Those people will probably emerge and come forward, and they'll start doing it anyway, and you'll just embrace them and say, "Hey, thanks for helping us maintain the vitality of this community and this process we have."

There are some changes that have a very steep cost associated with them, perhaps. Maybe the switching cost of moving from one database to another is high. Maybe there is just a big disagreement that can't be resolved in the pull request commentary. There is another part of this process where we can actually take it all the way up to the CIO. Our CIO has a near-quarterly setting where if you've made a pull request on something and it requires CIO-level input, you can actually take your pitch, if you're a junior engineer or whoever you are, to this setting. The CIO, his DRs, principal engineers, and other influencers will be in this room, and you can make your pitch for that change. In most cases, you're kind of rehashing probably existing conversation.

There is a set of unbiased criteria that is typically followed in that setting, and you're just making your pitch, just like if you were doing a startup or something like that. You're making your pitch for why we should make this change. The nice thing is not only can people come to that setting and get that type of attention if it's an important thing, it's nice to know also that the trust goes the other way: our leaders at the top entrust our empowered engineers to let this guidance-based process actually work. They trust this process, and they already take into consideration all the input that was already had with the pull request conversation beforehand. So there's a lot of trust built into what dialogue has happened so far, and they are taking this engineer's input into this CIO-level setting.

These next two slides here are just: on the repo, you can see how many watchers there are, getting alerts for things, starred, and forked. This is also another nice thing that GitHub shows you to give you an understanding of whether the community is healthy, how many people are actually making changes, and things like that. What I really want to underscore here, though, is that just because it's not centralized with governance, that it's guidance, doesn't mean you shouldn't measure it. You should measure to make sure everything is healthy and that things are on track, and that there's some good vitality in the community. So you should still do that.

Next, Luke is going to talk about everything from a product owner's view.

Luke Rettig

All right. I'm Luke Rettig. I'm a principal product owner. I own a couple products at Target. I have our enterprise GraphQL platform that I own, one of our enterprise caching platforms that I own, and then I have our core item location data service that we provide out to the enterprise.

I'm going to talk to you guys today about just a view from my lens on why DevOps, and particularly this theme of engineering empowerment and then also product team empowerment, some of the things Levi touched on, really hit home and really drive awesome products.

Before we get started on that, I'm going to do a little survey here. How many people in the room, show of hands, have been on a team or are currently on a team where there is a separate technology and product/business roadmap? Yeah. How many people in here have built something that wasn't used or adopted? Yeah, I have done that a lot.

I think we all know this: building the wrong thing, in that quote that Levi had up there, actually hits home, too. I hadn't really read the entire thing until I was sitting here. But building the wrong thing is a nightmare. It's a nightmare from the you-lose-time, you-have-unhappy-users, you-waste-a-ton-of-money-in-engineering-cycles doing this, and more times than not, you end up rebuilding it.

I think that building the wrong thing is actually stemmed from two different things. This is an extract of code from my brain. But I think that if your engineers or your product development team is constrained, you're dead in the water. And if you aren't, you actually have a chance. I call this my square peg, round hole example. I've worked on so many products where it was either a package implementation where I had a feature, a particular experience, that we couldn't deliver how we wanted to deliver it, and we ended up having to do a process workaround, or it's 10 clicks to get through this workflow or whatever it looks like. I think that having a development team that is empowered to make their choices about their technology is going to ultimately get you to a better result, or at least give you a fighting chance.

I think the other symptom here is actually back to some of the roots of the DevOps movement to begin with. I've worked on many teams that have this siloed approach. If you have separate technology and product roadmaps or objective key results or whatever framework you use, it's going to drive this difference in silos. Your engineering team's going to want to do something that their organizational leadership is driving on them. Your product team's doing something else. Your design team's doing something else. Another characteristic there is if you use language like "the business" or "not our business" or "our users," those are key tellings that you have some silos. The other one that is near and dear to a lot of people is: my product owner only prioritizes features and functions. They don't care about debt or defects or any of those sorts of things.

Ultimately those lead to bad things. Where we need to go to get to awesome products is you have to have shared goals. You have to have an empowered team. You have to be on the same page. You have to break down some of those silos.

I'm going to talk about what I think the role of a product manager is, and then I'm going to talk about what I think the role of engineering is in getting to this siloed breakdown. Product managers or product people really have two things that they are accountable for. One aspect is your traditional textbook stuff: you're the voice of your consumer, you do a lot of product marketing. Some of the quotes are, "You're the CEO of your product." I think that is absolutely important. You have to prioritize the backlog. You have to make sure you're meeting the business objectives that your product entails.

But the other side of this that goes under the radar a lot is you have to spend just as much time ensuring that your development team knows what you're building, why you're building it, and who you're building it for, so that they can come and build the right solutions.

This becomes even harder when you aren't necessarily building for your core customer. At Target, I don't know what our official thing is. I always think of it as suburban mom with a family. My products, GraphQL and caching and data services, don't service suburban mom with a family. They service engineers that are using my data or using my platform. It becomes really hard to do that, but I think it's so important to get grounded in those core product things and say, "Here's who our customers are. Here's who our competition is. Here's what they care about," regardless of who those people are.

Really important is product people need to let your development team and engineers do what they're really good at: problem-solving, creativity. Let them own and drive how you're going to achieve these objectives. Don't get so caught up in the detailed features and functions and what really good looks like. Let some of that go. Step up, get higher level. Here's what we're building. Here's why we're building it. Here's how I'm sequencing my things, why I'm sequencing it this way, and have that healthy dialogue and trust and empowerment to let your development teams build what they need to build.

I'm not an engineer, but I would give this advice to engineers that have skeptical product counterparts. Do whatever you can do to get grounded in that what and why and who. Ask your product person or your end users, "Why are we doing this?" Really get grounded, really understand that.

For the debt, stability, technical stories, I cringe when I hear all those terms because I just view this stuff as things we need to do for our product. But I really came out of my shell when I had a lead engineer that would come and say, "Hey, here's what your objectives are. Here's why we need to do this particular thing, pay down a debt, switch whatever, because it's sucking up our velocity and it's going to stop us from being able to deliver what we need to next and next that aligns to what your vision is." If you have creative ways to articulate why doing something aligns to the overall goals and your end users and benefit, the better off you will be.

So what does awesome product look like? I've built this up, and Dan and Levi did this too, but it really looks like you have an empowered team. Target went on this engineering empowerment journey and talked about some of that. Levi alluded to the Dojo and a lot of the other tools that Target is really getting into around how we center around business value and product value. I think that's the key here to get to awesome product.

Really, some of the detailed things: the product team drives your backlog. It's not your product owner writing every story in your backlog and tagging it in feature versus debt. It's your team all contributes, all says, "Here's what we're trying to do this iteration," or, "Why this is important right now because of what our users have," and they are articulating what that is. I actually think that things not being labeled as debt or stability or feature, you might want to do that for reporting and measurement purposes, but your backlog is just a list of stuff you guys need to do to enable the value that you're driving.

The obstacles that still remain for us, this is my reflection on this: when we started at Target on our DevOps journey and breaking down silos within our IT organization and driving to empowerment of engineers, I actually think those same challenges are still right here with us. It's just more of that bigger, broader community. It's your product team and your business teams doing those things, breaking down the silos, empowering those teams to deliver the right things. I would say that you're never really done. Software is never done; it's only ready to be shipped.

That's what we had. Thank you all for listening to us. Thank you.