Host Intro (Gene Kim)

We are coming up on our last talk, and I was so delighted that the next guest is Clare Hawthorne. She is Senior Director of Engineering and Product Operations at Oscar Health. Oscar Health is a health insurance company focused on making healthcare more accessible and affordable.

I just so much love this presentation because it told the story about mechanisms that aren't widely talked about in this community, specifically around product operations. Clare gave a non-plenary presentation, so we thought it'd be a great idea for her to give a reprise of her presentation. I have some questions I've queued up for her that I've been dying to ask ever since we finally met in August. Clare, thank you so much for being here.

Clare Hawthorne

Thank you, Gene, and hopefully my audio is going okay. Thank you so much for the warm introduction. Hopefully everyone can see my screen. I do have an abbreviated version of this talk. There's some bonus content that Gene asked me to add, and then I'll be watching the clock to make sure there's time for the Q&A also.

I'm really thrilled to share how I took some inspiration from the DevOps community to build our product operations function.

First, Gene teed this up really nicely, but Oscar is a health insurer founded on the idea that healthcare in the U.S. is really broken. I know we have a lot of international folks on the call, and you probably can't even begin to understand how complicated the U.S. healthcare system is. Oscar aims to make healthcare accessible and affordable for all. We do that through the federal and state ACA exchanges, which launched about a decade ago and you may know as Obamacare.

Our member-centric approach has helped us attract 1.6 million members in about 12 years, and we have an industry-leading NPS of 66. Gene chuckled at the fact that I said our competitors usually hover around zero. The fact that we have not only a positive NPS but one that rivals some of the great consumer-facing companies is something we're really proud of.

We see our technology as a key differentiator in delivering that vision. To give a little context, our tech team is about 500 people and we're organized into about 45 scrum teams, or pods. I joined Oscar in 2021 to build out product operations, and about a year ago my remit expanded to include engineering operations, which we now jointly call Tech Ops. We really think that we create leverage across the whole tech organization by applying this DevOps mindset.

Why did we build out product operations in the first place? Oscar had a problem a lot of companies do. We were so focused on our members and our growth strategy; we were adapting, experimenting, and customizing, which also led to lots of tech and operational debt. When I joined, our processes were not really designed for scale. They were optimized for speed. We also kept setting incredibly ambitious goals. We tackled big tech-wide initiatives, and really our engineers, designers, and product managers were the ones bearing that coordination tax. Finally, leadership had encouraged pods to optimize their processes locally, but this led to a lack of visibility when we needed to aggregate information for our CTO or CPO.

How did we start tackling these problems? In my first 60 days, I surveyed the landscape and realized that while product was in the name of our function, we were really there to help the pods be more effective. You may have thought of product operations as being junior PMs. That's not what we do at Oscar. My vision, even in those early product ops days, was always to unlock Oscar's ability to ship more, better, and faster.

As I was getting my bearings at Oscar, my mom, who I have many things to thank for in my career, insisted that I read The Phoenix Project and The Unicorn Project when I was setting up the team. I'm so grateful to her because I had these aha moments: I was trying to do at Oscar what Bill and Maxine and the Parts Unlimited crew were trying to do in those books.

We're organized around three primary pillars. We're about four years old now and have matured into this model. We focus on pod enablement and product delivery, tech-wide processes and tools, and onboarding communications. We believe that each of these pillars contributes to our vision of shipping more, better, and faster.

Structurally, we're organized into five different sub-functions to drive more efficient, effective, and compliant software delivery. Pod enablement, which internally we refer to as product operations, was the first area where we had our beachhead. Those ICs embed within two or three pods in a single domain. Their goal is to make the pod as efficient and effective as possible. This is by far our largest piece of headcount, representing about 50% of our staff.

In contrast, our Tech PMO team is focused not on the pods but on initiatives. For rolling out big company priorities, that's where we deploy the Tech PMO resources. Tech processes has a tricky balancing act. They need to create just enough process to allow teams to go faster. We've all been in situations with bureaucratic process that slows everybody down, but we've also probably been in situations where there's no process, and that slows you down in a different way. We're trying to find that sweet spot. Currently the processes we focus on are quarterly planning, incident management, and onboarding.

Tech governance is an embedded GRC function within Oscar Tech. We oversee Oscar's adherence to our IT general controls. This includes monitoring and liaising with auditors, but like other parts of Tech Ops, it's not just a compliance function to slow people down. We see tech governance as an enabler, and we're constantly evaluating ways to simplify our controls or better manage to our key risks. Finally, we have a finance operations function in tech that works with budget owners to make sure we're maximizing the impact of our budget without exceeding our spend.

Another thing Gene asked me to do in preparation for today was try to put this all in an org chart. This is my best attempt. It's not a great one. We roll up into our Chief Product Officer, but I really have our leaders in engineering and product as my key stakeholders. Each sub-function works with slightly different sets of stakeholders. Many of them are in tech, most of them are in tech, but we do have that boundary-crossing role, like Jason was mentioning earlier, where we help serve as connective tissue in different ways in different functions.

I'll drill into pod enablement in a case study. Their primary partners are the pod leads, usually a product manager and engineering manager. They also work with design and operational stakeholders. One thing that's been really fun is pod enablement and Tech PMO end up having a shadow partnership, where our program managers can lean on the folks embedded in those domains to understand what's going on on the ground without going to each one of those 45 pods. There's a symbiotic relationship there that's been really lovely.

I promised a case study. Oscar has gone through many consecutive years of expansion and growth. When a user had an issue, they'd just raise a support ticket with the pod. At first it felt scrappy. The product manager was working with users, and it worked great, except that it didn't scale well. Product managers were being asked to manage intake, engineers had to increase the size of on-call rotations, and everyone was feeling the load of increasing backlogs.

We started with Broker Experience as one of the areas with the highest density of these issues. Brokers were a really important part of our sales cycle and ecosystem, but we hadn't prioritized those internal tools. As more and more brokers were onboarded, our support team was feeling more and more pain.

We embedded product operations in this area. The first chart is the number of open tickets, and you can see the queue got out of control. When we embedded product ops there in February 2022, they first focused on stopping the bleeding: how could they make sure intake was under control? The second chart shows the average age of open tickets. This is where we started working down the backlog. Once the queue had stabilized, we had a target of getting all of this under control by September, which is when open enrollment season really starts for brokers.

Because we were embedded, we were able to take those learnings and make suggestions for the product roadmap. We advocated for tooling and made sure people were building this into the internal tools we had for the broker support teams.

Now you're probably thinking: Clare, you said you read The Phoenix Project. Did you just create a branch? Did you put in a bottleneck who is managing a support queue? One of the things I'm really proud of is that we were able to ramp down product ops staffing to zero in this domain and didn't see a noticeable increase in tickets. The ultimate impact was that we were able to reduce engineering and PM staffing by 35% on this pod and redeploy them to other strategic initiatives.

From there, we tackled a tech-wide cleanup effort. Our key results are that we saw a 35% decrease in tickets open during the peak volume months of December through February. Over a 12-month period, we worked down that backlog and saw a 70% reduction in overall support tickets.

What did this do for the pods? This quote from Bill, one of my favorite stakeholders, says the broker backlog was notorious and was a huge constraint for PMs and engineers. It was critical to have product ops focus on optimizing run state to address it. The effort to cull through the aging backlog alleviated pressure heading into the busiest season. One thing I love about this quote is that Bill calls attention to something important: even engineers not actively on the queue, simply knowing there was this giant backlog, felt a huge mental tax. It really took them out of flow and distracted them from the work we thought was highest value.

One thing I know we're big believers in here in the IT Revolution community is: what's next, and what are the challenges that lie ahead? In the interest of time, I wanted to focus on one thing Tech Ops is grappling with right now. In The Phoenix Project, Eric Reid says improving daily work is even more important than doing daily work. Eric is right, but it's hard. We're actively working to find more room for process improvement, especially because by design Tech Ops roles are run-work heavy. It's a constant struggle to find enough space to implement those process improvements while keeping the lights on.

One thing we started to do is simply measuring it. You can't manage what you don't measure. We looked at the pod enablement group earlier this year and found that about 75% of the time is spent on run work, leaving only 25% for process improvement. We've set an aspirational goal internally to bring that process improvement piece up to 40%, because we think that's how we can really deliver an outsized impact for Oscar.

One way we're generating this capacity is using AI and large language models, which ties together this whole day. We had a great use case recently where someone on the team had the inspiration to take work that otherwise would have been 20 hours a week, once a quarter, times three or four, and use our internal sandbox program to bring that down to about half a day of work and transition it to an operational team. Maybe, Gene, this will be the subject of my talk next year.

I hope you've enjoyed hearing a little about our ecosystem and how we're unexpected DevOps champions. Before we transition to Q&A, Gene, I'm going to anticipate your last question. I know you always ask where you need help, so here's my pitch to all of you. When I was starting and telling my team about this talk, I wanted a definition of DevOps. I went to everyone's favorite vendor, Atlassian, and their article on DevOps starts really well: DevOps is more than just development and operations teams working together. It's more than tools and practices. It's a mindset, a cultural shift where teams adopt new ways of working. I love this. It's inclusive and emphasizes the holistic need to work together.

But you scroll down: womp. A DevOps engineer is an IT generalist with knowledge of software development, infrastructure management, and software system administration. True confessions: I have never spun up a Kubernetes cluster. I do not feel like this definition speaks to me. I felt like I wasn't allowed in the club.

But becoming part of the IT Revolution community, being with you all virtually today, it feels like I've made it to the cool kids table. As tech leaders, I want to encourage all of you to think expansively about who you can invite to the cool kids table, because they might not realize they're invited to join in on the fun.

Q&A

Gene Kim: Thank you so much, Clare. My first question is that you didn't mention in your talk something I found so interesting: you're actually a trained auditor, thanks to some advice from your mom. I don't want this talk to be about her, however illustrious she is, Dr. Baldwin. Can you share how that's been helpful to you and your team?

Clare Hawthorne: Again, huge thanks to my mother. I think she saw it as a way for me to understand financial statements, which is totally true. I joined Ernst & Young in 2007, right on the heels of Sarbanes-Oxley, and being a junior auditor on a public client was the best crash course I could possibly have wanted in process design. One thing that has stuck with me throughout my career is the importance of monitoring. Just because you send out a communication or roll out a new process, it doesn't mean it's actually implemented. It's part of your responsibility to monitor and make sure that process is working as designed, and correct it if it's not.

When I took over engineering operations, I now run the team we call Tech Governance. It was this happy accident for me and my boss. I don't think either of us thought three years ago when I joined Oscar that I would be using that part of my background. But in June 2023, we were in the midst of remediating three material weaknesses in access management, change management, and program development. It was June, so there wasn't a lot of runway to fix this problem.

The biggest short-term value I brought to the role was trust. I was rusty, but I was able to understand the context. My partner on the accounting side and our chief accounting officer immediately saw me as an ally as opposed to a novice. We talked a lot about novices today, but there had also been people who were viewed as antagonists or adversaries.

The other thing I realized very quickly was that we didn't have quite the right people around the table to solve this really big problem of how to remediate this material weakness. As an example, we had our head of SRE implementing our job monitoring controls, but he hadn't ever been framed as the owner of those controls. Reframing our partnership and having him feel empowered to create changes, evolve the processes, and make sure we were actually mitigating the right risks, as opposed to just following the process someone else had designed for him, has been a transformational effort for the team. It's been amazing to work with our infrastructure engineering team as they've stepped into that metaphorical driver's seat. We're making better controls. We're investing in our CI/CD pipeline and automated testing in a way that's win-win for everybody. It makes the developer experience better and it makes our auditors happier. It's been really nice to bring those pieces together.

Gene Kim: That's fantastic. When you run the tech governance role, to what extent are you taking the role of the first line of defense, second line of defense, or whatever? What's the benefit for the program?

Clare Hawthorne: The three lines of defense: I'm at the New York watch party today and I brought my whole team, so Kevin Chang, who's in the audience in New York, is our Director of Tech Governance. One thing we talked about a lot when he first started was, which line of defense are we? I describe us as one and a half. We don't see ourselves as an internal audit function that tries to stay arms-length. We see ourselves as enablers, and in some cases we're doers and we perform actual controls. But our goal is to transition some of that control ownership back to the people who are best suited to do the work, in infrastructure engineering, SRE, and our DevOps team, while we play the translator role and make sure they understand what they need to do to create the best possible control and process.

Gene Kim: I have to imagine that with your counterpart, whether internal or external to the company, as a publicly traded $6 billion a year revenue company, your background in audit must have gone some way in terms of building a mutually trusted working relationship. To what extent is that true?

Clare Hawthorne: I think it did. Even as simple as knowing the lingo. For those of you who've worked with auditors, things like completeness and accuracy or timeliness: they didn't have to explain to me why those things were important. I just got it right off the bat. They'd been working with engineers and other folks who really didn't understand why timeliness was something they cared about. Being able to use that experience, dusting off some of those cobwebs but bringing that into Oscar, has been really helpful. I can play that translator role to help explain to engineers what auditors are looking for and what the requirement is.

Gene Kim: That's fantastic. For those of you who don't know, I was a proud card-carrying member of the Institute of Internal Auditors for I think eight years. I totally agree. There's a precision in how auditors talk that is really fantastic. You talked about tech governance. There's another part of your organization, pod enablement. Stereotypes are bad, but you had a wonderful observation that sometimes product teams are focused on launch day. How do you know when a pod is having a problem? How do you recognize those groups?

Clare Hawthorne: It's funny you mention launch day. I've worked with many product managers and engineers over my career. I dabbled in product management, and a lot of really great product managers, as soon as you get to launch day, they're onto the next thing. That's wonderful. I love that about them. It's been this wonderful partnership where I always say our Tech Operations superpower is run state. We think about launch day, but we don't stop there. We think about launch day plus one, launch day plus 30, launch day plus 365. I think it's a slightly different human that wakes up and wants to jump out of bed and fix those problems. It's created a really great balance between the two.

In terms of how we know where the problems are, sometimes it's obvious. You see the run work start climbing up. You see those on-call rotations getting really big. But what we've found to be the best way to know what problems to solve is to embed deeply in those pods. We have a leverage model: usually one product ops or pod enablement person for two or three pods. Then it's up to them to work with those folks and the pod leadership to figure out whether this is a documentation problem, a launch readiness problem, or a support queue problem. We really empower them to understand that local context, usually by rolling up their sleeves and doing the work, and then advocating for where they want to spend their time improving those processes.

Gene Kim: Super cool. I'm so glad that you and Jason Yip from Grainger were able to share an experience report on what's mostly an AI-focused day. I was stunned to see that OpenAI actually did a case study with Oscar Health. Can you tell us a little bit about that?

Clare Hawthorne: Happy to. Our co-founder and CTO, Mario Schlosser, has been an AI evangelist and an early adopter and has really championed its use in the healthcare and health insurance space, but also internally at Oscar. We were hearing earlier today about companies where it's hidden in the shadows or happening on the fringes. AI at Oscar has been front and center for the last year and a half. We've built tooling internally, we've had hackathons, and we've built it into our product.

One thing we're really proud of is that we have an actual medical practice within Oscar Health Insurance that sees virtual patients. One thing we've been experimenting with there is using AI to document clinical interactions. We've seen the time reduction is about 40% to transcribe and document those conversations, which allows doctors to see more patients and be more thoughtful with the patients they have, instead of spending their time taking notes after the fact.

We also see it as a great way to democratize care. It sounds obvious, but sicker patients have longer medical records and charts, and so they're the hardest to query, analyze, and summarize. We really see AI as a way to level out that playing field and equip doctors with the best information to treat patients in the most need of care. We're really excited. We've been partnering with OpenAI a lot. I know there are lots of different models out there, but it's been a really rewarding experience. My team had a leads onsite yesterday, and we're working on an AI workshop for Tech Ops to think about how Tech Ops can either recommend AI use cases or use it ourselves to empower the folks we work with.

Gene Kim: Fantastic. Congratulations on all your achievements. Your experience report was one of my favorites of the conference, so I'm sure there are, like everybody else, so many more fun adventures ahead. Again, congratulations and thank you so much for being with us today.

Clare Hawthorne: Thank you, Gene.