Host Intro (Gene Kim)

I met the next speakers earlier this year at the LaunchDarkly conference, and I'd love to talk because they embodied one of the goals of this community — they showed how technology and business leaders can work together to achieve a common goal, and that those outcomes must be co-created by both business and technology. So the next speakers are Tracy Behler, Senior Director of Online Experience at Hawaiian Airlines, and Kiyoshi Kusachi, Senior Director of IT at Hawaiian Airlines. Unbelievably, the airline was founded 95 years ago.

Continuing on the theme this morning from Lauren Woods of Southwest Airlines, we'll talk about how critical technology is to so many aspects of running an airline — both in times of crisis such as the COVID pandemic, as well as enabling the organization to make good decisions that enable them to win in the marketplace. Here is Tracy and Kiyoshi.

Kiyoshi Kusachi

Aloha. Good afternoon. Really excited to be here with all of you technology leaders from around the world. Today we'll be giving you a glimpse into Hawaiian Airlines' technology journey through the years.

I'm Kiyoshi Kusachi. I lead the IT digital portfolio for Hawaiian Airlines. My team supports the application development, the architecture, and the operational support for our guest-facing channels. This includes our website (hawaiianairlines.com), our native mobile apps, our loyalty HawaiianMiles frequent-flyer platform, and our API services supporting those channels. I'm fortunate enough to have my business colleague here with me today — Tracy.

Tracy Behler

Hello everybody. I'm Tracy Behler, Online Experience. Before we dive right in, we wanted to give you just a little background on our airline. We are, as Gene said, in our 95th year of business. We started in 1929 as Inter-Island Airways, and we provided essential travel between the Hawaiian Islands. Today we operate over 200 flights per day to 30 domestic and international destinations. We carry over 10 million guests per year, and we have over 7,000 employees — the majority based in Hawaii.

This is our network. We fly between the Hawaiian Islands and the US mainland, primarily up and down the west coast to the Islands, as well as some long-haul flights to New York, Austin, Boston, and Salt Lake City. The international markets we serve in Asia are Japan and Korea. Then we fly to Australia, New Zealand, and South Pacific, which include Tahiti, Samoa, and the Cook Islands.

The guest-facing channels that we are responsible for — Kiyoshi mentioned — our web and mobile websites, hawaiianairlines.com, our iPhone and Android apps, and our loyalty program. These channels are responsible for the majority of sales for the company, generating over a billion in revenue per year.

Kiyoshi Kusachi

The technology across our IT organization recently went through a large transformation. Today we'll be giving you a glimpse into the slice of that journey specific to our web channels. Five years ago we had large, significant feature rollouts. These rollouts required large in-depth analysis. Over time, we had a lot of web pages and user interface logic bundled into a large website application, and our services were a big set of endpoints into what we call our internal API application. Over time those all became very tightly coupled, and projects were a large undertaking. We were not able to realize the full potential until everything was complete. Releases took months.

To stay competitive at our scale, integrations with our partners is a core competency. Years ago when we started, our business logic and our capabilities were built server-side and required knife-edge cutovers once all dev was complete. An example of this was a cutover to a pricing service that enriched our offerings to our guests. It was always an orchestrated effort when we had to juggle production in the middle of our development process. Ultimately, we were able to get the functionality out to our guests, and we followed this for several implementations for our multi-month projects.

Tracy Behler

But 2020 came around, and we ran into some major turbulence. And then this happened, and everything changed almost overnight. This photo still takes my breath away because we'd never seen anything like it before — almost our entire fleet was grounded and parked on the runway in Honolulu airport. We had taken down 90% of our schedule in a matter of days. It was a really, really scary time for the business, and especially for our guests who had travel plans — their flights were canceled, and they didn't know when we were going to resume flying.

Unfortunately, we did not have the capabilities on our website and our app to cancel flights or allow people to rebook. So the high number of impacted guests quickly overwhelmed our call centers. They were absolutely crushed with calls — it took people waiting for hours to be able to get through to talk to somebody, and this went on for weeks.

The months that followed the shutdown were extremely challenging, with crazy and constantly changing travel restrictions and entry requirements for each of the destinations that we served. Hawaii did not want visitors, so the state imposed a 14-day quarantine on visitors coming to Hawaii. Then each of the individual islands layered on their own unique travel requirements. So it was just a really confusing time trying to keep up with the constantly changing requirements and then trying to communicate them to our guests.

We quickly worked to add new online capabilities that we didn't have — like the ability to cancel a flight, or look up a travel credit and apply it. These were things — and flexibility — that we really didn't need before, but we needed it now.

Kiyoshi Kusachi

So we only had days, not months. To optimize our development time and time to market, we created a new space in our architecture for purpose-driven API microservices. We introduced a new capability-flags concept to deliver dynamic connections to our guests, and there were certain conditions we could target based off of: itinerary (where you're flying), travel date, your member type, if you're logged in or not logged in, if you're a platinum-type guest, and the state in the customer journey.

This pattern worked really well for us, and we used it for several integrations. We saw a lot of potential, but we felt like we could do better. We ultimately wanted to cover all the use cases to support our guests and make it a complete package — but we were constrained due to some critical areas we needed to support that were buried deep into the history of our architecture and were challenging to uncover. Our website consisted of multiple layers of previous initiatives. This caused complexity with various versions of technology, programming languages, and multiple point-to-point integrations with our partners.

So in 2021 we continued building upon our accelerated solution that we found in 2020 and embarked on a web modernization journey while supporting our guests. We reimagined the entire stack — from the edge user interfaces to our API services, our system integrations, and our monitoring/analytics approach.

We chose to take advantage of our partners' enhancements over time. Concepts like mainframe-native commands or old SOAP APIs were refreshed to the latest current versions of RESTful APIs. To centralize our business logic and our system integrations, we created an API ecosystem and catalog of lightweight reusable microservices we call our "hub."

As soon as a service was ready, we immediately wanted to take advantage of it by directing traffic to the guests, using the capability flags that our systems could support. Our former stack and infrastructure became our legacy stack. Zooming in — in addition to the application redirects at the root level, we were able to redirect within our flows to the new refreshed capabilities. We set up rules to allow specific use cases and flows to our new architecture by turning up our distribution percentages and slowly kicking up the tires on the system and monitoring our infrastructure.

Tracy Behler

This worked really well, and we felt like we reached a cruising altitude for about six months — but then we ran into some unexpected headwinds that we had to fly up against.

Just months into our modernization project, what felt like a major bomb was dropped on us. Our company made the decision to replace our Passenger Service System (PSS) — from our current platform called Sabre, which we'd been on for over 25 years — to a new one called Amadeus. And you don't understand how massive an undertaking this is. I heard Lauren from Southwest this morning liken it to a heart-and-lung transplant, and I 100% concur with that metaphor, because it was really big.

The PSS powers a large set of critical capabilities for our airline to operate. It's connected seriously to like every airline system — not just the bits that Kiyoshi and I manage, like the shopping and booking and ticketing day-of-travel stuff, but our accounting systems, flight operations, crew scheduling, revenue management — all key systems the PSS interacts with. So it was one of the largest and most important technology projects that we had taken on in decades.

At the same time, we had just taken on this other very large and important project of modernizing our web services — something we would never choose to do at the same time. But in order to do both these two initiatives at the same time, we had to make some really tough trade-offs as it related to features and functionality that would be available on our website and apps when we did this cut-over to a new system. And it definitely had an impact on our user experience. We did not have the time to rebuild all the features and capabilities that we had on our legacy services. So it was literally a race against time the whole year to build out our new SPAs, learn about this new PSS, and integrate it.

Essentially, we had to go backwards before we could move forward. This is a slide that gives you a sense of the features that we could deliver in the time that we had. Really, it's like table-stakes — core airline functionality to book and manage a trip — and a lot of features and revenue-generating ancillary products that we had on our site, as well as just features that our guests loved and used all the time, like the ability to search via price calendar, hold a fare and purchase it later, or get an online receipt. All of those things were not things we could deliver by the time we had to cut over. Our guests were not happy — they were super pissed. We got tons of complaints. Our satisfaction levels dropped to the lowest I'd ever seen. It was really, really a tough time. So we had to work really hard following that cutover to rebuild and deploy a lot of these features.

Kiyoshi Kusachi

In order to make our delivery date, continuous delivery was not a nice-to-have. It was required for us to be successful. We had to continue and operate to our guests with our existing platform while working through a new platform at the same time. Our new application infrastructure and configurations had to be evaluated as early as possible. So dry runs helped us divert our traffic to specific areas of the stack ahead — for performance of our infrastructure, and data mapping and migration. We switched on and off technology as a group combined to validate data and mark our work as done.

Despite this core system migration landing right in the middle of our web modernization initiative, we felt ready for the test, as we were already equipped with the patterns and technology needed to be successful. We already understood server-side integration and understood how to create interfaces to target one system to another. Now that we had capability flags, we used them heavily to turn on and off new platform capabilities to test it — to do side-by-side comparisons in our test environment. From the user-interface level we had routing, and we used that as a global switch for our large cutover. Having these existing foundational patterns allowed us to spend time learning on and taking advantage of new concepts of our partners' offerings, such as real-time data streaming or operational data stores for new event-driven patterns for our guests.

With this new system in place, we were able to launch on time and were well-prepared for discovering new ways of finding functionality for our guests.

Tracy Behler

This is a chart of our customer satisfaction survey scores. The gray line is 2022, before we had begun our modernization and our PSS cutover, and the pink is last year. You can see the significant dip that we saw in our scores when we did the transition. Thankfully we've been able to dig out in the months that follow back to where we were, and even exceeding it. So the teams have done just an amazing job getting us back to where we need to be.

Kiyoshi Kusachi

We now have a reference architecture across all of our channels in the IT enterprise. Our hub is this set of services and integration standard that is leveraged and reused by multiple teams, so we can deliver on a more frequent basis to our guest expectations. Core functionality like flight status, pricing, payments, seat map, ticketing, reservation management, check-in, or guest notifications are all optimized reusable components that our channels embraced as they were ready.

Tracy Behler

So there are many lessons learned, and here are a few we wanted to highlight.

Focus on the minimum viable product (MVP). Make the time and effort to understand your capability set and your partners' offerings, and create a blueprint of where you want to head, where that first step is going to be, and how it impacts your organization. Simplify your business rules and technology solutions, while resisting the temptation to custom-code or work around features or components that are not readily available.

With cross-functional team communication, keep in mind that other teams are on their own journey within your organization, and they may not be ready to take on that complex messaging or data model that you're expecting. So remember to adjust. While you have those discussions, keep the ultimate end user and the customer experience in mind, because ultimately that's what we're doing at the end of the day.

Reusable patterns — from product documentation, business functionality, external connectors — prioritizing the components that you have the most reuse should be the top of your list.

Kiyoshi Kusachi

We talked about during development using capability flags to cut over functionality, but we are also able to reuse those and gain advantages in production. After cutover we had some seat-map issues for certain types of markets. Instead of having the guests wait in and get an error, we were able to gracefully display messages so they can get the answer earlier and timely. We also had check-in routes coming back — checking in for certain itineraries had challenges, and being able to tell them and target that messaging was really helpful, and guests appreciated that.

Tracy Behler

So we were asked what we can get help for and what we're looking for. As you may know, it was announced that Hawaiian Airlines is in the process of being acquired by Alaska. If you've been part of an acquisition — either acquirer or being acquired — we'd love to hear any stories or talk about lessons learned, things you recommend or would not recommend.

Finally — we're really proud and honored to be working with an amazing group of talented individuals at Hawaiian Airlines. Despite some challenging circumstances, they were able to produce quality work and deliver results. And while they're doing that with the aloha spirit within our teams and with all of our partnerships we've built throughout the years — Mahalo.