Host Intro (Gene Kim)

Thank you, Amanda and Nathan.

Another one of my favorite talks from DevOps Enterprise Summit a couple of weeks ago was a talk from George Kraniotis, Director of Software Engineering at Discover Financial Services. I loved it because it was so startling, and yet it is something I suspect everyone shifting from decades of annual project-funding models to product-oriented value streams will face.

The problem he described is the puzzle that one faces with all the applications and services that do not fit neatly into a product or business value stream. What exactly do you do with all of them? These services are mission-critical, because when you attempt to turn them off, scores of services would actually stop working. But who should run them? What should be their long-term strategy? Who should manage them?

When I mentioned this talk to Dr. Mik Kersten, the author of Project to Product, he said something that blew my mind: the leftovers are actually probably the most important products because they are the internal platforms that all the other products use. Here is George to talk about how they discovered this puzzle, what they did to try to solve it, and the problems that still remained.

George Kraniotis

Thanks, Gene.

It was May 2021 when my organization rolled out the new product-team structure. We had spent the previous five months working from the customer backwards with both technology and business partners to figure out what this new organization would look like. Along the way, we had these functions that did not cleanly align to a specific product team.

My name is George Kraniotis. I am a Director of Software Engineering at Discover, and I am going to tell you a story about why I volunteered to take on this function and how we approached all of these things that did not really fit.

A bit about Discover: we are a leading digital banking and payments partner, and our mission, if you boil it down, is to help people achieve a brighter financial future. Helping do all that are 20,000 employees. All of our call centers are in the U.S., something that we are proud of. We have 1,400 employee engineers working diligently to drive value for our customers, and in doing so we are trending toward having 1,000 deployments to production per month.

Discover itself, and those engineers, pride themselves on having small autonomous teams working on things that make an impact for our customers, being able to focus on learning and innovation, and having access to leadership at all levels.

Before I jump into the story, a bit about the area that I am in. It is called Card Portfolio. In a nutshell, anything a Discover Card member can do, from viewing their statement, to adding their Discover card to an Apple or Android device, to doing a balance transfer and so much more, is what we support.

Where we started, coming into January when we started talking about the new organization and what it might look like, was that in the past teams moved to the work, very much a feature factory. There were over 300 developers, 200 applications across three different platforms, and of course we have mainframe. We were using scaled agile, where we would bring together 100 or 200 people once a quarter, spending a day, sometimes two, all together planning what we would do for the next quarter, hoping things would not change. Of course, they always did.

One thing we were surprised by as we went down this path of figuring out how to shift to product was that 50% of our applications were managed by technology, meaning the teams that were being paged if something went boom in the middle of the night. We managed them, but there was no business owner. What I mean by that is no one owned the customer experience. No one owned what the business logic would be. That shocked us at the time, but looking back it did not, because again, we were a feature factory. That is how we operated.

Things started when Amir Arooni, our new CIO, came in. He had this vision of the runway, where we would shift from project to product using smaller autonomous teams to deliver new products and enhancements while at the same time simplifying how we work.

Whether you are thinking about going down this path or you are already in the midst of it, one thing I can assure you is that it is not a technology journey, nor is it a business journey. It really is a company-wide one. I attribute the success we have had so far to the fact that we have been aligned at the senior vice president level, the vice president level, down to my peers, all the way down to the staff, the engineers, and the product owners doing the work. Everyone is aligned with why we are doing this.

If you want to learn a little more about how Discover is putting craftsmanship at the center of our transformation, I will reference DOES 2021, where Angel Diaz and Sheila Lodia gave a talk about this. You can scan the QR code or visit the IT Revolution site and search for it. I wanted to put it out here because I am not going into the weeds around how engineers are working on their craft.

As we endeavored into this new landscape and asked how we get to product teams, one of the things to do was define what in the world product was. For us, we boiled it down to two things. It was not easy getting here, but I want to share our secret and how we came about it.

The first is a customer journey: something a customer interacts with, whether through the web, our mobile app, or even calling in, because people still do that. That customer journey itself is a product. On the other hand, it can be a platform capability, which we look at as supporting more than one customer journey and being able to deliver that end-to-end experience.

With that out of the way, we knew that at the core of this new landscape we had to have dedicated and persistent teams. There had to be end-to-end ownership as much as possible. The most important thing, I think, is continuous planning, where each team owned its backlog. They owned the pace at which they delivered and what made sense to them. That was definitely a big difference from the old approach, where we used program increments and quarterly planning.

When we went through this, one of the first things that came out was product families, which is a fancy way of saying a logical group of products. This was the first one we went out with. The things in orange were the customer journeys. In blue at the bottom was our platform, Portfolio Enablement. We worked on this from the customer back. The first thing a Discover Card member does when they join is set up their account. Then they start using their card, and maybe at some point they slow down or stop using their card, and we want to re-engage with them. It was very much centered around the customer. Each box had teams associated with it. Some had two; some had up to 10 or more. The intent was that there was a logical grouping of how we thought about the Card Portfolio organization.

I have not even gotten to why I called this The Leftovers. I am not referencing the show. I am certainly not referencing the takeout from the weekend before that you have not eaten yet and is in your fridge. I am not referencing people who maybe are not good enough and are on the sideline instead of on the field.

In fact, one of the things we uncovered as we went down this journey around what was in and what did not clearly align to a specific product team was that we had components that allowed you to log in, others responsible for sending notifications, or allowing you to change your card. I want to double click on this and share more about what is behind each one.

Start with being able to log in. What we came to realize was that monolith data aggregators were in play. When they were built well over a decade ago, there was legitimate value and a legitimate goal: reduce mainframe spend and increase resiliency. Everything from a digital perspective, web or mobile, hit our mainframe many years ago. We wanted to put in place a level of cache using a database on the distributed side, where we could go to the mainframe, get data that is not really volatile and does not change, get it once, have it for the duration of a session, store it on a distributed database, and serve it to our applications through a middleware layer. It met our need at the time, but as we shifted into this product operating model, things became more complex.

Another example is batch-triggered notifications. We have mainframe batch, and the goal was to connect mainframe batch to a distributed system to send notifications. How did we do that? We created 60 jobs to enable it. Again, in this new product operating model, that is not necessarily the best way to have it be, but it served a purpose.

The last example is card device management: centralize all card-management functions. Think about us in our feature-factory way of working before. We had eight customer journeys. We look at them now as customer journeys, but back then it was, how can I quickly add this new journey, and where do I put it? Throw it in this monolith application that leverages MVC Java-based templates and middleware to get it out to customers. It served a purpose and worked, but now eight customer journeys were embedded in a single monolith. Not ideal.

That first attempt left us with 130 people and seven teams without a product family. I touched on a couple of the monoliths that were part of that. We did not know what to do. We also had contractor teams that performed on behalf of functions, one being lightweight front-end development and another legacy production-system support. We had to figure out what to do, and we were not quite sure at the time.

Then we had this hypothesis: create a new temporary product family to hold all of those functions, the applications, components, and on-behalf-of services, and push those all back to the product teams, because product teams need to own everything. That was a hypothesis.

Let us dig into what we did with the leftovers. We did not call it leftovers internally. We have people here that we are talking about, and that is not a nice thing to say. But for purposes of this talk, let us continue to call it the leftovers.

Our approach was two things. First, we wanted to centralize and then shift. Centralize meant pulling it together into one temporary product family, supported with strong leadership. I had an engineering manager overseeing the group because we knew we had to get buy-in from stakeholders, and there were going to be difficult conversations in expanding their span of control.

For my engineering manager and myself, many of these functions were new to us. We knew of them, but we had no idea what they did. We spent the first 90 days just learning: what do you do? Who do you support? How do you measure success? We knew we had to put together preliminary objectives and key results in order to think about where to start.

Starting means starting the shift. Moving to a product-team model is the right thing to do, and while it may be good for product teams, it is not as if they were excited to take these new things on. The approach we used was one of three things. We would move that function and its hierarchy to a logical area that made sense. We would federate it, maybe to many teams. Or we would simplify it, like breaking a monolith down into more domain-driven APIs.

We knew we wanted to start small. We could not boil the ocean with everything in scope for the leftovers product family, and we wanted to link it to broader business efforts. For us, we started with the on-behalf-of team of contractors that did lightweight front-end development. It made sense because they were making changes in the same GitHub repositories that product teams owned. We also knew we had historical issues that impacted customers, and we could lean on those as leverage to get buy-in.

We wanted to be transparent between the business and technology stakeholders, so we created a simple one-page business case. By no means is it perfect, and you do not have to use it exactly as-is. The important thing was that we wanted to answer: so what? Why is it important? What if we did not do anything? We were trying to be our own devil's advocate around what would happen if we did not do anything and how to help people understand why this is important. We put these together, not just one of them. As we approached different functions, we pulled a new one out, filled it in, and had conversations with leaders: vice presidents, my peers, ensuring that everyone in the organization up and down knew why this was important.

The results so far: by no means are we done, but I want to share what we learned and achieved. First, a reduction in product-family size by 59%. We were able to reallocate capacity to other areas that needed it where it made sense. Second, living into this product model means you have teams being paged out in Illinois that before maybe were some offshore first-level team that just acknowledged false positives very quickly. When we took it on, they were able to reduce false positives for one app. The thing that surprised me the most was uncovering deficient architecture. In a way, I understood why we were in that situation, because we were a feature factory before. Teams knew: why invest in addressing technical debt or refactoring when next quarter they are going to be working on something completely different?

With that said, we still have problems. By no means is this easy. Simplification is easier said than done. We know we have to reverse engineer legacy code and then get buy-in. Fortunately, a team came to us and said they were interested in pulling that stuff off our monolith. We said, okay, great, let us start. We are using that as an example, almost a white paper, to showcase and get buy-in from other groups that may want to do the same thing.

The other question is what is good enough? The last 20% is going to be the most difficult, and we have learned that. If we get through 80% of a function and federate it, and that last 20% is all with one product team, we are going to have a conversation with that product team and ask: is this on your roadmap? If not, let us have a conversation, because it may need to shift to you. These are things we are figuring out and thinking about how to approach.

The takeaways: when shifting to a product operating model, you are going to have areas that do not fit. The leftovers are these bits that everyone cares about but no one wants to own. The way to prove this out is to say, okay, we will shut this down and see how critical those components are to your area or to the organization. When we had those discussions and said, if no one is willing to own it, we will take it down, people raised their hands and said, hang on a second, and we got their attention.

For us, we defined a hypothesis: what needs to be true to shift these functions and move them to product teams? I will admit that the initial hypothesis was that the whole product family would go away. We are learning that some functions can go to product teams, but there are other things that, as we dig in and learn more, do not necessarily align to a specific product team that we can federate out and say, okay, perfect, this thing no longer exists. It has to live, and it has to have care and feeding from a product-team perspective. We are having discussions for some components about where they live in a shared platform or shared capability model. For us, we called it Portfolio Enablement. What does it look like there? How do we help it be successful in that space? We had a hypothesis. We wanted to start something. It may have been extreme initially, but we wanted to make sure the vision was set.

If you go on this journey, what was helpful for us was to centralize everything, put strong leadership in play, and then start small with the shift. These are critical functions. They were for us, and I imagine they are for you too if you are embarking on this. Do not take it lightly. Start small, get buy-in, and get to know those who may be impacted and affected by the different decisions that are going to be made.

Thank you. This is my QR code for LinkedIn. My last name is sometimes difficult to spell, so feel free to scan it. I would love to connect with you and hear what questions you may have.