Clare Hawthorne

Hello everyone, my name is Clare Hawthorne and I'm here to talk to you a little bit about Product Operations and how we are an unexpected DevOps champion. My pronouns are she/her, and we'll get into it right now.

01Oscar Health context

Oscar is a health insurance company founded on the idea that healthcare in the US is pretty broken. We pay more for worse outcomes than many other rich countries. We aim to make a healthier life accessible and affordable for all. We focused on refactoring the individual insurance landscape through the federal and state ACA exchanges, which you might know as Obamacare.

Our member-centric approach has helped us attract 1.6 million members with an industry-leading NPS of 66. Most insurers hover right around zero. In 2023 we generated almost $6 billion in revenue.

[Slide: Oscar Health, Inc. (NYSE: OSCR). "America spends twice the portion of its GDP on healthcare compared to any other rich country in the world for results that are no better — and in many instances worse." Oscar founded in 2012. 1.6 million members, NPS 66 as of June 2024. $5.9 billion revenue in 2023. Members enroll in Individual & Family Plans; +Oscar is the B2B technology platform that powers others in the healthcare space.]

02Oscar's Tech Team

We see our technology as a key differentiator. Our tech team is about 500 people and it's organized into about 45 Scrum teams or pods.

I joined Oscar in 2021 to build our Product Ops function. About a year ago my remit expanded to include Engineering Operations, which we now jointly call Tech Ops. We create leverage across the entire tech organization by applying a DevOps mindset.

[Slide: Oscar's Tech Team — highly cross-functional org structure. ~500 people total. ~45 pods aligned to a particular tech surface area or user journey. Cross-functional partners: PMs, Security, Design, Data, ENG, IT, Tech Ops. Clare leads Engineering & Product Operations (aka Tech Ops).]

03Oscar has always been hyper-responsive — but at scale, processes needed to evolve

Why did Oscar build out product operations in the first place? Oscar has always been hyper-focused on being responsive to our members and the overall market. We grow quickly — adapting, experimenting, customizing — yeah, accumulating lots of tech debt. We also had operational debt.

[Slide: Three problems at scale — 1. Pod processes were optimized for speed, not scale — Oscar grew at a torrid pace; the processes that had worked well for pods a few years prior struggled to keep up with increased tech surface area and membership. 2. Increasingly ambitious goals required more coordination — goals became increasingly ambitious, and tech-wide initiatives relied on Product & Eng Managers to manage the rollout. We realized we were placing a "coordination tax" on our builders. 3. Leadership visibility — Pods created local processes that worked well for them, but when leadership needed to understand what was happening locally, we often relied on manual aggregation.]

04How did we start tackling these problems?

In my first 60 days, I surveyed the landscape. It became obvious to me that while my new product operations team had "product" in its name, we were really there to help the pods be more productive from the get-go. We established that we weren't there to be junior PMs.

My vision, even in the product ops days, was that we were always there to unlock Oscar's ability to ship more, better, and faster.

[Slide: Our Vision — Tech Ops unlocks Oscar's ability to ship more, better, and faster.]

05The Product Ops journey — Tuckman model

A quick audience participation moment. How many of you are familiar with the Tuckman stages of group development? It was mentioned earlier in the Vanguard talk by Michael. The Tuckman theory is that all groups go through these four stages: Forming, Storming, Norming, and Performing.

I joined in the spring of 2021 — and boy did we have a rollercoaster ahead. When I first joined, Product Ops included two or three product specialists who floated around the org, often working as junior product managers. I staffed up to a team of six people and we had to find our product-market fit of our function. It was a very tough first summer. We experimented with different operating models until we landed on what we call our embedded model, which we scaled in 2022 — more than doubling the team size.

[Slide: The Product Ops journey — 2020 (Forming: Business case, Early wins) → 2021 (Storming: Experimenting, Product/Market fit, Unrealistic Expectations, Lack of empowerment) → 2022 (Norming: Embedding with pods, Scaling the team, Honing our value proposition, Coming processes) → 2023+ (Performing: Repeatability, Measurement, Company alignment). ⭐ Clare reads Phoenix Project & Unicorn Project → "aha" moment that Product Ops shared DNA with DevOps.]

Most folks on the team are individual contributors aligned to two or three pods. Being embedded allows the team to become subject-matter experts. They serve as that first line of defense for pods and are experts on the pods' ongoing processes.

Another thing happened in the summer of 2021. It was the first time I read The Phoenix Project and The Unicorn Project. I had an aha moment — what I was trying to do with Product Ops at Oscar was what Bill and Maxine were trying to do at Parts Unlimited.

06Tech Ops Pillars for Oscar

[Slide: Three pillars — 1. Pod Enablement & Product Delivery: Pod processes (issue triage, feedback loops, release management); Initiative / Program Management (coordination across pods, dependency tracking, risk identification). 2. Tech-wide Processes & Tools: Processes (annual/quarterly planning, CapEx, incident management); Tools (administration of common tech tools — Jira, PagerDuty, Confluence); Enterprise ITGCs (governance & controls, including SDLC, access management). 3. Communications & Training: External & internal tech comms (monthly release deck, stakeholder updates); Onboarding; Product training; Demo day; Hackathon.]

[Slide: Tech Ops sub-functions (6) — - Product Operations: Pod Enablement (HC 14) — Make pods more efficient & effective. Align ProdOps to pods with high run work; either augment existing PM capacity or reduce the need for PM staffing for pods in maintenance mode. - Product Ops: Processes & Tools (HC 3) — Create standards and processes that increase consistency while minimizing friction; create scalable processes that aggregate key information across 50+ pods. - Tech PMO (HC 3) — Make key Oscar initiatives more efficient & effective. Align Program Managers to key company initiatives. - Tech Governance (HC 5) — Ensure Oscar's technology / IT controls meet standards set by our regulators, especially SOX/SOC. Work with control owners to design controls that address key risks without unnecessary friction. - Tech Finance Ops (HC 1) — Ensure we maximize the impact of our budget without exceeding our target. - Tech Business Management (HC 1) — Enable a strong, healthy team culture across Tech (MY/EOY reviews, Glint, Department meetings, Onboarding).]

[Slide: What does Pod Enablement include? Product Mgmt (purple), Engineering (green), ProdOps* (yellow) across the lifecycle — STRATEGY (Business case development, Problem definition) → DEFINE GOALS (Product roadmap, Define OKRs) → SCOPE (Build vs buy assessment, Annual/quarterly priorities, Vendor evaluation) → DESIGN (Solutioning with stakeholders, Synthesize 0-to-1 requirements, Write product specs, Pod process design, High-level sizing, Technical design) → BUILD (Problem solve edge cases, Sprint management, Coding, Code quality, Tech dependency management, Non-tech dependency management, Communicate / report status, Manage vendor implementation) → LAUNCH (Test plan sign-off, Technical documentation, Define launch strategy, QA X-functional coordination & execution, Publish / document launch features, Educate internal X-functional stakeholders) → RUN (Monitor product KPIs, Ongoing alerting & monitoring, Bug triage & insights / trends, Vendor management, Maintain taxonomies, Knowledge management) → REFINE (Issue resolution & comms, Bug resolution, Gather "1+" requirements, Product feedback & feature request intake, Retrospectives & process improvements). ProdOps activities (yellow) concentrate in BUILD, LAUNCH, RUN, and REFINE — "run state" is the Product Ops superpower.]

07The Product Ops superpower is "run state"

I often joke that the product operations superpower is run state. Most of the product and engineering folks I know are much more interested in launch day — getting something out into the world. They don't really care as much about launch day +1, or +30, or +365. And that's where Product Operations comes and brings our differentiated skillset.

When I compare notes with other product operations leaders, embedding within pods is the most polarizing part of my org design. It's expensive, it's taxing to have people that are embedded in these pods. A lot of people instead have a coaching model where they give advice but don't go in as deeply. But I believe it's key to our DevOps mission. Embedding allows us to understand the local context and to tailor our work within the specific constraints of the pod. (Amy in the John Deere presentation mentioned this earlier.)

08Processes & Tools team — partnering with Governance

The other key element of our product operations team is our Processes and Tools team, led by my colleague Lily, who's in the audience. Their goal is to create standards and processes that increase consistency while minimizing friction. These processes are eclectic — running the gamut from our quarterly roadmapping process to incident management. Similar to the pod enablement function, this team insources processes that are highly manual and streamlines their ongoing operation.

In the last year we've had tech processes very closely work with our Tech GRC team (which we call Tech Governance). They've developed a knowledge and understanding of our IT general controls, and we are able to partner with our Senior Director of Infrastructure and SRE as he simplifies some of Oscar's job-monitoring controls for SOX.

[Slide: Quote from David Winder, Senior Director, Infrastructure Engineering & SRE — "Product Ops regularly demonstrates the ability to understand complex existing processes with a high amount of friction. They can then use a combination of network/relationships, research, and experience to produce process improvements that net significant time saved. Recently I proposed simplifications to our job monitoring controls. I partnered with Product Ops, whose understanding of the overall tech surface area allowed us to connect the dots with our change management process. Without this expertise, the simplifications I proposed would have caused downstream issues with our SOX population — risking a control deviation."]

09Case Study 1 — Pod Enablement: Finding flow by minimizing support requests

Oscar's gone through many consecutive years of expansion and member growth. Our product and engineering teams have been focused on developing a greater product with minimum viable process. Our internal users would raise a support ticket, and pods would work on them. It was a good thing at first — it allowed us to keep our scrappy startup mentality. But as we grew, the support request process became non-viable.

Pods didn't have enough bandwidth to intake and triage the requests raised by stakeholders, and some of those requests were critical to our operations. So as our pods were starting to feel the increasing demands of run work, they had trouble keeping up with user needs. This impacted everyone — Product Managers were being asked to do triage, Engineering had to increase on-call rotations, and everyone felt the growing mental load of a big backlog. It pulled the pod out of the flow of roadmap work.

Sounds familiar, right? I saw flashes of Oscar in The Unicorn Project. Maxine's team achieves flow when they eliminate distractions, reduce work in progress, and create an environment where team members can immerse themselves in their work. Product Operations similarly tackled Oscar's increasing support request queue to help pods focus on their most meaningful work and find their flow.

10Spotlight on Broker Experience pod

[Slide: Spotlight on Broker Experience pod — Product Ops embedded with Broker Experience (aka BrokerX) in Feb 2022. Open Tickets chart peaks ~Jan 2022, declines sharply by Jan 2023, stays low through 2024. Avg Age of Open Tickets (# days) peaks ~Sep 2022, declines after.]

Brokers are a critical part of our member ecosystem and our growth strategy. Over the years we developed industry-leading tech for our brokers, but we hadn't prioritized our internal tooling. As we onboarded more and more brokers, we were getting more and more issues flagged to us by our internal Broker Operations and Support teams.

The queue had gotten out of control. We embedded Product Operations into the domain in February of 2022: 1. Initial focus on understanding and triaging new tickets. Partnered with Broker Support team and upskilled team members to defray volume. 2. After stabilization, works down backlog queue going into peak Broker enrollment season (~September & October). Also engages with Broker Ops to improve intake and escalation processes. 3. Advocated for roadmap investments to increase self-service tooling and decrease manual interventions performed by Eng on call. 4. Ramped down Product Ops staffing from 1 FTE → 0.5 FTE → 0 FTE.

Outcome: Reduced Eng & PM staffing on BrokerX by ~35%.

Now you may be thinking, Clare, did you create a new constraint, a new bottleneck of expertise? Now you have product operations in the mix. Actually, no. About a year ago, we were able to ramp down Product Operations staffing from a full FTE all the way down to zero without seeing a dramatic increase in new support requests.

11Tech-wide cleanup effort

We didn't stop there. Product Operations saw the success we were having in the Broker domain and we decided to tackle the support request queue holistically across all pods.

In August of 2022 we set about tackling the Oscar-wide support request queue, learning from the initial push in BrokerX: - Identified pod ownership and created a JIRA Dashboard for each Domain/Pod with ProdOps support. Set reduction targets for each ProdOps team member. - Updated mapping of issues ↔ owners to streamline ticket assignment, transfer, and tracking. Created a process to add new issues and audit/remap existing issues.

Key Results: 1. 35% reduction in tickets created during our peak volume months (Dec–Feb). 2. 70% reduction in support requests over a 12-month period from July 2022 to 2023.

[Slide: Quote from Bill Williams, Senior Director, Product Management — "The BrokerX service request backlog was notorious-hovered and was a huge constraint on Product Management and Eng time. The immediate attention that Prod Ops provided to issue triage and management was critical to ultimately having the space and time to permanently reduce domain run cost. The cherry on top (among many other things!) was the effort to cull through the aging backlog one-by-one and clean up the entire thing. This alleviates a lot of pressure from our teams heading into our busiest season — the overhang of this backlog was just… mentally taxing!"]

12Case Study 2 — Tech-wide Processes: Making work visible with aggregated roadmaps

When I joined Oscar, I asked to see the roadmap. My jaw dropped — someone sent me the file. It was over 100 lines long with no ability to roll it up into categories. This is how you communicate with internal stakeholders? I mean, I guess it did the job, but it was challenging to see how the various components connected to our company priorities.

Then I noticed something else. The roadmap didn't include any resourcing required to support the annual insurance enrollment cycle — the most critical activity of our business, to enroll new members every year. When I asked why, people looked at me quizzically — it was considered run work, so it wasn't captured on the roadmap. I felt like I was Bill in The Phoenix Project when he realizes no one has been accounting for the internal IT projects. What else were we missing?

It became my mission to ensure that Oscar's tech team had a well-understood roadmap.

13Inspiration from Phoenix Project: "Make work visible" — Improved understanding of strategic initiatives

What we did: Implemented a Parent/Child Taxonomy aligned with the company goals. - Organizing our roadmap into ~15 strategic priorities allowed leadership to better understand work. - Allowed for "local initiatives" that were unmapped — these could be isolated and pressure-tested to ensure high ROI.

Unlocked: Began capitalizing more software.

The first pass was messy, especially to retrofit it, but ultimately we landed on about 15 strategic priorities. No more 100-row spreadsheets.

This improved understanding of our roadmap had a surprising benefit — it actually allowed us to capitalize more software. For years we had been expensing infrastructure investments without realizing that we could capitalize them.

14Demystified "non-initiative" work

What we did: Accounted for 100% of Eng capacity, including overhead & PTO.

Unlocked: - Short term → better planning & capacity allocation. - Long term → identifying areas to reduce technical debt and improve developer productivity.

From the beginning I was adamant that our roadmaps had to add up to 100% of our capacity, including overhead and PTO. We surfaced all the run work that teams were doing.

The first thing it did was unlock our ability to plan and allocate capacity more realistically. We realized we were understating our run costs by about 10–15%. At first it was simply a big bucket of things we called "maintenance and overhead." More recently, we've gotten a lot more nuanced — in the past few quarters we've started to categorize our run work more precisely, and we've used that to find areas where we want to invest to reduce tech debt and improve developer productivity.

[Slide: Quote from Jesse Horowitz, Chief Product Officer — "Product Ops plays a core role in enabling the entire tech organization to run more efficiently. We had our most effective annual planning process in years due to their organization and leadership. With the continued efforts of the Product Ops team, I'm optimistic that we'll be able to plan, predict, report and execute with more precision and confidence."]

Anecdotally, each year we've had fewer reprioritization fire drills — you know, the kind where everyone has to get in a room and lay out on paper all of the different things they're doing and try to figure out what you can cut. This year we were even able to accommodate new priorities mid-year. Of course, we still needed to move some resources around and make trade-offs, but we avoided the thrash we'd had in prior cycles where it felt like a big gnarly effort.

15What's next?

You've heard a lot about where Product Ops started and some of our successes. Every presentation is supposed to have a section on "challenges that lay ahead." My internal comms team didn't want "challenges that lay ahead" on a slide. So I have four things that are top of mind for me as I continue to build on these successes and level up my team.

161. Expanded "Tech Ops" remit

It's been a year, but I'm still adapting to the expanded Tech Ops remit. Going back to the Tuckman model, when taking on the expanded scope last summer, I had to restart at forming and storming. My initial focus was simply to get the team to feel cohesive — which took about six months. Now the team has come together, but I'm still working to hone the specific value proposition of all of those different sub-functions and socialize them with our partners. We're getting there, but I'd still put us in the norming phase.

[Slide: The Product Ops journey (extended) — PRODUCT OPS 2020-2023: Forming/Storming/Norming/Performing curve. ⭐ Clare takes on Tech Ops in 2023. TECH OPS 2024-2025+: A second Forming/Storming/Norming/Performing curve starts in orange.]

172. Measuring our impact: metrics vs. "vibes"

A product ops colleague at another company joked recently that the best way to measure success of product operations is vibes. And let me tell you, we have great vibes at Oscar. You've seen throughout the presentation — our reputation is stellar both with product and engineering.

[Slide: Two quotes — Sahil Mehta, Director, Product Engineering: "Product Ops has been instrumental in increasing our chances of successful launches by meticulously managing the entire process. Their close collaboration with stakeholders and end users has significantly accelerated the adoption of new features. Furthermore, their dedication to running daily operational production processes and supporting our end users ensures a seamless experience and higher user satisfaction scores." | Erin Landau, Director, Product Management: "Product Ops exhibits a deep understanding of our Campaign Builder product and processes. Their comprehensive knowledge of our systems and services has allowed them to execute several process improvements. In the last 6 months, our Product Ops Manager streamlined our SDLC process, improved our launch checklist, and implemented several new Slack workflows in our team's help channels."]

But as great as these are, I still want to just establish clear outcomes and metrics that are driven by Product Operations to more clearly articulate our value to the organization.

183. Finding room for process improvement within pods

We need to find more room for process improvement. Or as Eric Ries would say — we need to improve daily work, because improving daily work is even more important than doing daily work.

Eric is right. But boy is it hard. By design, our Product Ops roles are run-work-heavy and staffed by relatively junior people. It's a constant struggle to find enough space to implement process improvements versus just keeping the lights on.

[Slide: Product Operations: Q2 2024 Time Expenditure — Pod Enablement: Research & Refinement 23.5% | Pod Enablement: Product Health & Improvement 39.3% | Product Delivery: Launch Readiness 23.7% | Other (Miscellaneous 11.8% + Recruiting 1.7%) 13.5% — totals: Process Improvement 27% / Run 73%.]

In a recent time survey we found that approximately 75% of the team's time is spent on run work. We've set an aspirational target to bring that to 60%, so that we can spend 40% of our time on process improvement.

One lever we're experimenting with is gen AI. We've had a few early successes, but we need to figure out the best way to scale that across other pods.

194. Streamlining our quarterly roadmap processes

And finally, I'm looking to streamline our quarterly roadmap process. Yes, it was a process I used as a case study, and I do believe it's had a positive impact on the team — but I would be the first to admit it has more overhead than I would like. We're actively reviewing the process and hopefully we can get improvements to the team soon.

20Close — Who can you invite to your DevOps journey?

I hope you've enjoyed learning about my experience establishing a DevOps-inspired Product Operations (and now Tech Operations) team at Oscar Health. I hope it also inspires you to think about your own ecosystem. Maybe you have systems thinkers that also want to drive company outcomes. Find them, and use them as your own unexpected DevOps champions.

But before I close — since I've got the stage for a little bit longer — I also want to make a pitch to all of you.

I want to challenge the prevailing wisdom that DevOps are engineers who manage cloud infrastructure and SRE. Gene mentioned this in his opening remarks this morning, and I know I'm preaching to the choir — everyone here knows DevOps is more than that. But look at a recent search I did for "DevOps jobs":

[Slide: Google search and LinkedIn search for "DevOps jobs" both return engineering roles focused on AWS, cloud, and SRE.]

So then I went to everyone's favorite Atlassian page. Starts out great, to quote: "DevOps is more than just a development operations team working together — it's more than tools and practices. DevOps is a mindset, a cultural shift where teams adopt new ways of working." I love this. It's inclusive and emphasizes the holistic need to work together. Scroll further — you see "DevOps culture," very inclusive.

But right after that: "A DevOps engineer is an IT generalist with knowledge of software development, infrastructure management, and systems administration."

I felt like I wasn't allowed in the club.

[Slide: Atlassian "What is DevOps?" page — DevOps culture description (YES! arrow), DevOps engineer description (SIGH arrow). DevOps culture: "DevOps is a cultural shift where teams embrace a software engineering culture, workflow, and toolset that elevates operational requirements to the same level of importance as architecture, design, and development. When developers who build software also run it, they have a greater understanding of user requirements and needs. The values of a DevOps culture include increased transparency, communication, and collaboration across teams." DevOps engineer: "A DevOps engineer is an IT generalist with a wide range of knowledge around day-to-day software development, cloud infrastructure management, system administration, and automation."]

So coming to this conference and being with all of you — it feels like I've made it to the "cool kids" table. I'm not going to lie, I've been so excited for this.

As tech leaders, I want to encourage you to think expansively about who you can invite to your cool kids table. They might not realize that they're allowed to join in on the fun.

Thank you.