John Roe

John Roe: Good afternoon. Welcome to our talk entitled Doing Taxes at the Speed of DevOps.

My name is John Roe, vice president of technology at H&R Block, in charge of our modern tax platform systems. Joining me is Tony Ogden, director of technology for modern tax platforms.

Today, we will be talking to you about our journey from developing a brand-new omni-channel tax engine. It is a journey started about two and a half, three years ago, and one that, when we started the journey, we knew that we were going to have to approach the problem differently. And with that approach, we started to utilize DevOps.

Before we go into the details, let us talk a little bit about who H&R Block is. H&R Block was founded in 1955 by Henry and Richard Block, as an accounting business that grew to do taxes. Believe it or not, prior to the start of H&R Block, when you needed your taxes done, you actually just sent your information to the IRS, and they actually did the tax calculations for you. Henry and Richard were at the right place at the right time, and the rest is history with the company being founded.

Today we have grown to over 80,000 tax professionals across the US, Canada, and Australia. We have over 11,000 locations, and we serve 23 million prepared returns and about eight million clients do it themselves. This equates to about one out of every seven tax returns in the US is done by using H&R Block software or an H&R Block professional.

Now I am going to turn it over to Tony Ogden to talk to you about our seasonality.

Tony Ogden

Tony Ogden: Thanks, John. Good day, everyone, and thank you for your interest in H&R Block's DevOps journey in the modern tax platform space.

H&R Block is going to continue to invest in products and services that bring year-round value to our clients. However, it is virtually impossible, almost impossible, to remove the seasonality from our core tax business. I am going to address several of those unique challenges that we have to plan for year in and year out.

First and foremost, we have to align with the IRS as it relates to the number of days in our tax season. We typically start in the mid-January timeframe, and then we go through mid-April. That is approximately about 112 days of the tax season. But we are at the mercy of the state departments of revenue and the IRS to certify their forms and schedules with us in preparation for the season. So if that certification process extends past the first part of the season, that 112 days actually becomes much less than that.

We do have peaks and valleys that we have to deal with. At the very beginning of the season, from mid-January through mid-February, we typically deal with our refund client base. Then we slow down at the end of February all the way through the end of March, and then we see a big ramp-up as we get closer to the end of the tax season. Between the first part of April and mid-April, we typically see our procrastinator clients like me, or those clients that typically have a balance due. Those peaks and valleys, though, we are starting to see be less sharp as it relates to the last few seasons, and I think that is primarily driven as we are seeing client volume actually spread more evenly across the tax season.

Our biggest challenge, though, continues to be that we experience one client visit each tax season. What does that mean for H&R Block? What that means for us is that we have to get it right the first time. We have to provide that wow factor. We have to prove that our experience and our expertise is a differentiator in the marketplace.

I mentioned that we have 112 days, approximately, in the tax season. But on average, it takes 12 days for a client to file their return. 12 days. So what does that 12 days mean for us? It just means limited opportunity. It just means limited opportunity to convert that client or to be able to continue to retain our existing clients that prefer our brand.

John also mentioned that we have around 11,000 active offices each tax season. As we get to the end of tax season, we close the doors on about 75% to 80% of those offices, which just means for us that we have a limited footprint once the tax season ends.

Additionally, we have about a four- to six-month delivery timeframe. And when you throw in the pandemic, that actually reduces that timeframe even more. But I am going to paint the picture of what that means for us typically. As we approach the end of tax season, usually about two to four weeks post-tax season, we are now planning for next year's initiatives. We are finalizing budget, we are finalizing those strategic imperatives, we are locking in on what that scope is, and we are starting to do research and design activities. But as we approach the summer months, that is normally when we kick off our initiatives, and when you extrapolate that out to the beginning of tax season, you will see that we normally have four to six months to get ready for the upcoming tax season.

Now, that is not limited on all of our initiatives, as we do have year-round initiatives, such as modern tax and our investment in the cloud, that will span multiple years, but primary number of our initiatives are limited based on that four- to six-month timeframe.

Last but not least, we have a high volume of seasonal workforce. We have over 80,000 tax pros that we have to onboard for each tax season. That onboarding process typically happens in a four- to six-week timeframe. But we also have to make sure that they are properly trained and certified as we approach each of those tax seasons.

I am now going to turn it over to John, and John is going to address tax complexity at H&R Block.

John Roe

John Roe: Thank you, Tony.

When we think of taxes, one of the issues with taxes is the complexity. That is why a lot of people have their taxes done by the use of a tax preparer or through a software product that actually guides and assists them through the tax preparation process. At the end of the day, though, there is really one answer that comes out: either I owe money or I am getting a refund.

But to get to that answer does require a complex number of fields and forms. In fact, we have over 150,000 regulatory fields, and also 15,000 tax regulatory changes that come in every year. So when you think of the regulatory fields, 15,000 changes, there is a lot of automated testing and a lot of regression testing that has to be done to make sure that if you change something here, you did not break it there.

Now, we have changes that come in from the federal government. We also have changes that come in from the various states and Puerto Rico. On top of that, cities and counties across the US, some, not all, but some actually have their own tax laws that people are required to file taxes. Sometimes they pull from the federal or state returns, sometimes they do not, all depending upon each individual instance.

Now, on top of all that, each individual has their own unique circumstances. In fact, you could say that doing taxes is somewhat like everybody is like a snowflake. Everybody is unique. You might have bought a house this year. You might have gotten married. You might have had a child. You might have adopted a child. All these situations create complexity around your unique situation, and they create complexity on how your taxes are actually calculated. To do that requires a lot of field analysis and scenario testing to make sure that we are doing taxes correctly.

To help understand how that is, let us look at a little bit of an example of how a field inherits and does calculations from other fields. In this situation, what we are seeing is IRS pensions and annuities and how it draws values from other fields, and that field draws values from other fields, and it keeps cascading down and down and down. So as you see, when you get to the end, a change there can have a cascading impact further up the stream you go. You could have that far thing on the left be IRS refund, and if that was true, great, but understanding how those fields move and combine with each other, and sometimes comes close to actually being a circular reference that we have to work through.

Now, to do all this, we had to approach the problem differently. Traditionally what we have done is a lot of brute-force regression testing. We basically had seasonal associates that would come in, we had test scripts, and they would log into the application, and they would manually go through scenarios of basically sample tax returns to look for issues. Manually, they would test everything, whether it printed right, whether the tax calculation was right, whether the screens flowed correctly.

But we knew we needed to start separating our testing out. One thing we have done is we have created a UI that we use when we are building out and actually setting up these fields. What we have allowed to do is we have actually allowed tax analysts, these typically are folks that are not computer programmers, through a UI to actually build unit tests. So we give them an easy way as they build a calculated field, and it could be take field A minus field B, that they could then build in unit tests for each different possible scenario.

Now, in doing so, every time we do a build, these unit tests get run. So if we make a change in the very far right of that previous slide, we will unit test the far left, and we can see if we actually broke anything or not.

Along with unit tests, we have automated test execution. So again, when those kick off, we run through various state tests, and we run through various unit tests, and we will go through scenario tests as well. Now, scenario test is simply we take sample tax returns at the highest level. So when we file with IRS, when we file with the states, before that process, we actually have a process where we do certification returns. They give us a set of inputs, and we enter the inputs in, and we then produce a set of outputs. They validate that those outputs work.

We also have our own returns that we know, that we go through, and making sure that as we go through our taxes, that we do not break anything. So we can have over 1,000 of these scenario tests that are run. You can see, we put inputs on the top, we put outputs. We will test whether the calculations were correct. We also have a term we use called diagnostics. So this may mean that the calculations are correct, but something did not look right. It could be that, hey, did you know that you declared more in deductions than you actually had in income? Well, those are logic that we need to put in to make sure that the client is aware or the tax pro is aware that something is not going quite right.

Taxes, with those changes, you can imagine, all those fields, all the various tax scenarios across federal and the various states require lots of testing. Traditionally, like I said, that was done manually. What we are doing now is automating that process. And through automation, we are able to run through tens of thousands, hundreds of thousands of tests in a matter of minutes, what would have taken us days, if not weeks, prior to get through.

Now I am going to turn it over to Tony, who is going to talk to us about our journey from client server to the cloud.

Tony Ogden

Tony Ogden: Thanks, John. And before I review the tactical ways that we are modernizing both our tax platform and our culture, I wanted to go back and visit the decision on why we made this investment in the first place. If you look at two to three years ago, a couple of things that we really wanted to solve for. One, we wanted to be able to bring products and service to the marketplace sooner and reduce that delivery timeframe. The second reason is that we want to be able to support our clients where they want to be supported, how they want to be supported, and when they want to be supported as it relates to their tax preparation services.

So now I am going to review several tactical ways in which we are modernizing not only our technology stack, but also our infrastructure and our culture.

We have built and maintained large monolithic tax applications over the last several years. This has resulted in a fear of change and also slower response times. I am sure that is a familiar story with many of you. I want to hit on a quick example of what that means for us. Typically or historically, when we have made a change to our tax application, even if it was on the home page or if that was a tax entity that we had to get out and ready for the season, regardless of where that change was made, we would have to test, build, package, and deploy that application, and sometimes that would take days, maybe even a week to get that out to the field. So you can see that our response times were very slow.

Where have we landed? We have landed on a microservices-based technology. We have allowed our teams to build independent capabilities and work more autonomously so that we can deliver things in parallel. We have separated out our core tax application from our non-tax products and services. We are changing the engineering culture. So when you think about engineering with monolithic applications, we would just build stories sprint after sprint after sprint. Now what we are seeing is engineers have a cultural mind shift where they are starting to break down work into much smaller components, more manageable components. They are also making sure that their WIP bin, or their work-in-process bin, is only a couple of specific items each time. Additionally, they are making sure that their tasks are less than eight hours in duration.

We also are migrating over to a platform as a service. We are consolidating our development and our testing methodologies, our frameworks, our tooling, our database models, and also our regulatory components.

I just want to pause here for a quick moment. I want you to think about three tax products that we typically build and maintain each and every year. We have our DIY tax product, which is our online tax service. We have our software tax service, and we also have our brick-and-mortar, our assisted tax product and application. When you think about all of those applications, right now we have three very large teams that support those in a very siloed manner. That means that we have three separate teams building those capabilities in parallel, similar capabilities. We have struggled with being able to integrate and deliver cross-functional initiatives in a year-over-year fashion, and we did not have a lot of synergies. But now we are at a point where we are building microservices-based capabilities, capabilities that are built centrally and are easily extended to all of our three tax products.

So I want to take a quick look at our infrastructure. When you think about the approximately 11,000 offices that we have open on an annual basis, that is a very sizable footprint. When you think about the monolithic application that I addressed previously, that is a client server-based application that needs to be installed locally. What that requires is an office server and several workstations to be able to serve our clients in the office. We have to do that investment every three to four years in terms of refreshing that infrastructure.

When you think about our VMs and you take a look at the VMs that we have to scale up as we get ready for the tax season, especially for our DIY product, we have to build those VMs in a way that they support our peak transaction volume. We build that and support those VMs in a way that they can support that peak transaction at any point during the tax season. But those VMs are up and operational for that entire tax season, and we do not tear them down until the end of the tax season. What that means for us is that we are not managing our environments in a very thorough and efficient way.

So where are we now? We have made significant progress with our transformation over into the Azure infrastructure. We have implemented push-button automation in our LNP environment where we can easily scale up and scale down as needed, saving us thousands of dollars each month because our LNP environment is not up on a 24-hour basis. We have also turned on auto-scaling with ACE, and now that has allowed us to be able to deploy outside of our normal maintenance windows. We have typically deployed anywhere from 11:00 PM to 4:00 AM, but now we are seeing a much more flexible environment and infrastructure that supports our ability to deploy outside of that normal timeframe.

We are starting to invest and look at opportunities to create infrastructure as code, and we are leveraging Terraform Enterprise to be able to do that. With our assisted product that we are building natively in the cloud, we are also using AKS to deploy and manage our containerized applications. We have also introduced Istio service mesh that is helping us manage our microservices, and we are migrating over to Cosmos DB for a more flexible, scalable NoSQL database service.

So let us take a quick look at testing and what it means to where we have migrated from in terms of manual testing to more of a reliance on automation testing. I will paint a quick picture on where we are coming from as it relates to the manual side of this.

Typically, we have two major releases as we march towards the beginning of tax season. We will have a major release in the November timeframe and a December timeframe. When we look at the November release, that typically means that we have about four to six months of change that is being introduced into that release. What does that mean for us? That means that our development, about the mid-October timeframe, significantly reduces in terms of the number of enhancements they are churning out on a day-in and day-out basis. Our focus then shifts over to code stability and application stability. What we do is we typically have a 5-4-3-1 regression test model that we execute against, very manually intensive. Then our developers are primarily just focused on defect resolution. Then we do a rinse and repeat, and we start that process over again in the December timeframe.

Then as we get out of the December timeframe and we look at some of our smaller releases, now we are going back to what I addressed early in the presentation, where we do not take as much risk, and we are making sure that each and every enhancement and defect that we review does not destabilize our tax products and services.

So when you look at where we have matured to in terms of the automation side of it, and with our testing, the quality of our unit testing has gone up significantly. I talked about the cultural mind shift that our engineers have gone through and how we are breaking things down into much more manageable, smaller chunks. We are also seeing with the improved quality of unit testing and also the increased coverage, we are seeing lower defects, much, much lower defects, especially when you think about the repeat defects or reopened defects. Our smoke testing is starting to run on every pull request, specifically to our omni-channel engine. We are also investing in integrating automated UI and API testing frameworks. We are building these, leveraging Selenium, Cucumber, and other technologies.

We also have the plan to then integrate those testing frameworks into our CI/CD pipeline so that we can run those automation into our pipelines and ensure the quality at the very beginning step of the process. We have thousands of test scripts that are being executed on every single change before they are committed to the master branch. So that you can see our automation is playing a critical role in our ability to not only move faster, but ensure quality at the early stages of our project delivery.

So let us take a quick look at our teams and applications and our org structure and how we are migrating from there. Right now we have application-based teams. Cross-functional teams and cross-functional integration has been a challenge or a struggle for us in the past. We have to align our delivery timelines and our priorities, and it has been a struggle for us.

Typically, what you will see is one team will start the delivery on a particular initiative, and we will have a dependency on another team, but those dependencies are not delivered in parallel and prioritized at the same time. So what you will see is that our testing teams actually are impacted, and they are not starting testing at the same timeframe. And so the delivery of that initiative actually ends up taking several sprints rather than one sprint if we were to plan accordingly.

We are now organizing our feature teams around centralized capabilities. When you think about these centralized capabilities, they are things like printing, e-signature, data import and data export. We are building out these central capabilities in a way that we can easily extend them, as I alluded to earlier, to the other tax products that serve other people in the organization that can easily use those because they are built in a centralized manner.

These feature teams are being built with the skill sets they need to remove all their dependencies, all those handoffs that I referred to earlier, and let them move in a much more free way and an autonomous way so they can deliver without the oversight from our top-down leadership.

We are very early in the process and doing very small trials with feature teams at the moment. Again, very much centered around our modern tax platform space. But as we see the maturation of these feature teams, we are going to then continue to look for opportunities to extend these feature teams further out into the IT enterprise.

So now I am going to turn it back over to John, and John is going to address the larger IT or enterprise impact of our DevOps journey.

John Roe

John Roe: Thanks, Tony.

When we started this journey on modern tax platform a couple of years ago, at the time, there was a handful of people that were doing CI/CD. We had some teams, a lot of teams actually, doing variations of agile, scattering of unit tests across applications, but nothing really consistently applied, very ad hoc-ish in terms of when it got applied. And so we knew that as we got excited about what we were doing, the larger impact is what could we do with the enterprise? Shortly after getting our feet underneath us, we formed an internal DevOps user group.

What we looked to do was we brought engineers, we brought analysts, architects in, and we had them start doing demonstrations and demos for our teams. But more importantly, we increased the span to include all of IT. And so that quickly started to catch on through the use of video recordings. We recorded the sessions, and that way people who may not be able to make the session could actually go back on their own time and watch what was presented. This really started a good grassroots movement from the bottom level in terms of what we could do. We had demos on unit testing. We had demos on feature flag. We had demos around CI/CD and demos around infrastructure as code. Heavy engineering focus, but nonetheless, we started to get a lot of folks within the IT departments interested in some of these concepts that we were embracing within our teams.

After the DevOps user group started to get its feet underneath it, we started to notice that everybody got real excited and everybody wanted to go do things, but a lot of them did not really understand where could I go to turn to for help. Now, the user group was one avenue that they go to, but we then also formed up centers of excellence, and we actually started off with three of them.

One center of excellence was really more around agile, project management, product management in terms of how do we get information from our customers and organize that to get ready for build as soon as possible. Then we had more of an engineering one, and the engineering was focused really around what changes could we make in terms of how we manage code. We looked and we implemented code tools to do static code analysis, whether it was for code quality or security, to eliminate steps and gates from already before we go to production where we have got to do all these security scans. Let us bring that up front so we can make it do much faster and learn those issues quicker. They also started to try to come up with best practices and standards in terms of tool choice and tool technologies.

Also, what we uncovered is there were a lot of things that nobody had actually spent any time looking into yet. So what the centers of excellence would do is somebody would take it on. If it was early on and it was feature flag usage, it was great. Teams took it. They would go away, and maybe they would meet with a software provider, or maybe they would do it on their own, but they would come back to the center of excellence and provide an input or an update on how they progressed and seek any guidance or feedback they might have. After those, and it was engineering, we also did it with testing.

We knew we had something going, and so the IT leadership determined that the best course of action to help, but we wanted to see this proceed across all of IT, was we actually created a position, Director of DevOps Transformation. This role's primary purpose was really twofold. One was to increase the amount of automated testing that we had across the enterprise. When you look at the test automation pyramid, earlier attempts at test automation really focused primarily on just automating the manual user tests that we have always done. Those were flaky, they broke, and typically they wind up getting abandoned. So we needed to make sure that we were taking a new approach when we approached test automation.

The second was helping to become that DevOps advocate for the IT, as well as now moving on into more of the H&R Block enterprise as a whole. There were a lot of groups, whether it was engineers or project managers, that are kind of used to doing things the way they have done them for 10, 15, 20 years. That change management aspect was needed. We wanted to make sure that we brought folks along on the journey, that we could explain the benefits of why we were going after DevOps, and what part they may have in helping us achieve our goals.

Tony mentioned a little bit organizational design. How can we organize ourselves better in terms of who does the work? Typically, you are very siloed. You were an engineer, you did engineering. Somebody else tested your stuff, and you did not necessarily do the analysis. Somebody else did the analysis. So how do we come up with these multifaceted individuals that can do more than just a single skill set?

When we look at it from a bottoms-up, tops-down, we uncovered that a lot of things we tried was to do as much bottoms-up as we could. We wanted people to feel empowered and feel that they could make decisions. At the same time, we could not have anarchy. We knew that there were certain controls we needed to place, and so it was trying to define what were standards and what became best practices.

Lastly, and this is something that we are looking at as an enterprise overall, is how do we embolden our teams? If teams for many years had gone through where they had basically received a set of requirements and a delivery date expectation and said, now go do this, now we are changing it around and we are asking teams to come up themselves with what should we do? What should we work on? How should we solve that problem? Bringing the UX engineers, the UX designers, bringing the product managers in, one conversation to come up with a solution and setting our target goals is unique and different.

Secondly, we do taxes. Making sure that people can still be bold when it comes to how we do taxes. Now, we cannot be bold necessarily in how we do calculations. At the end of the day, A plus B has to equal C. But how we set the experience for the customer, what data can we gain that is necessarily outside what they provide us? Where can we gather data from other sources to pull in to make that tax interview as seamless and painless as possible for our customers? At the end of the day, we want to wow them, and like Tony had mentioned, we get one chance, and potentially that is over a 12-day window. Some days it is a one-day window that you may get, and then after that, you do not see them again. Hopefully, you see them next year. Majority of the time we do, but we just need to make sure we get that chance and they come back.

Lastly is empowering teams. We are looking to make sure that we have set up our way so teams feel empowered. If they want to take a risk, take a risk. And we are doing that in many ways. One is rewarding people who have taken risks, making sure that if they fail, they still can get recognized for failing because they took a risk. They took a gamble, and that is what we are looking for.

Secondly, on decisions. Too many times in our culture, I think a lot of people's cultures, all decisions get run up a chain of command to where somebody makes the call, and reality is they probably really do not know the details of what the call should or should not have been. We are really trying to push those changes down into the managers, into the individual team's hands to where they can make the call. And as long as they can logically explain what decisions were made, it is the right decision, the decision we will move forward with.

Now, thank you all this afternoon for listening to Tony and I. We really appreciate presenting at the conference. We represent everything that Gene Kim and others have done that have helped us on our DevOps journey. And now Tony and I should be online to answer any questions you may have. Thank you.

Tony Ogden: Thank you.