Rob Cummings

Hi, I'm Rob Cummings, a senior director in our platform engineering team here at Slalom Build in Seattle.

A little bit about me. I've been in the technology field for over 25 years now, most of that in enterprise organizations, usually on the systems engineering, technical operations side of things.

Over the last 10 years, I've been most interested in the intersection of technology and people: how do we bring these together to achieve meaningful outcomes, both in terms of what we're building and the people doing the building?

This brings me to my most recent gig: building software for the U.S. government. Four years ago this is not something I would have thought I would be putting on a slide, but here we are. Enter USCIS.

The U.S. Citizenship and Immigration Services, USCIS, verification systems process millions of transactions every year and are used by hundreds of thousands of people across the U.S.

A big part of the program's mission is focused on the people who have newly arrived in the United States. So for those immigrants who are eligible to receive benefits, improving the speed and accuracy of USCIS systems ultimately means less time in lines and more time spent building a new life and a future they love in the U.S.

To give a sense of scale for these products that our teams are supporting, we have a team of over 150 working across eight different geographic locations in the United States while also collaborating with multiple other USCIS contract teams.

And personally, this is the first set of products that I've been a part of that were operationally impacted by pandemics, government shutdowns, refugee crises, wars, and where that's going to require an act of Congress isn't used as a metaphor.

So it's been a ride, to say the least. But speaking of rides, in the middle of all this I decided to execute on my very long-planned midlife crisis. I ordered a 2021 Ford Mustang EcoBoost with the high-performance and handling packages back in April 2021. It arrived later that year in October.

Once it got here, I promptly signed up for a high-performance driving education event, usually referred to as an HPDE session, at my local track, Pacific Raceways, with ProFormance Racing School.

Why? Well, I've always wanted to learn how to really drive a car closer to its limits and not just operate it like we all do when driving down the street.

I also know that driving is extremely dangerous. In 2021, U.S. traffic deaths hit 42,915 for the year. That averages out to over 117 deaths per day due to traffic accidents or traffic crashes.

I want to stack the odds in my favor by knowing how to best react in any given situation and have actually practiced some of this on the racetrack.

So that began my adventure of very uncomfortably unlearning nearly 30 years of driving habits.

So while telling myself that embracing being uncomfortable while learning something new was going to be good for my brain, I inadvertently started comparing some aspects of this very educational experience to what I experienced working with teams building, delivering, and operating complex software systems, especially over the last number of years here with the federal government, kind of as a way to help me along on the learning process.

So enter the Three Ways: flow, feedback, and learning.

It's okay if you aren't already familiar with these. This talk will hopefully inspire you to learn a little bit more about them while also having a bit of fun applying these to the art of high-performance driving.

Disclaimer, two things. I am not a professional driver or instructor. I've only been at this for a year. I have a lot to learn. I am also not talking about racing or competition driving. HPDE is really about learning how to safely drive your car closer to its and your own performance limits.

Second, the pictures of Slalom folks in this presentation are all from internal projects. I'm trying to avoid getting in trouble from the federal government.

All right, that out of the way. Let's talk about the first way: flow.

You need intense focus when on the track. You absolutely cannot be thinking about anything else but the task at hand. What is that task at hand? Well, Don Kitch, the chief instructor at ProFormance Racing School, would probably say your primary job is being a weight manager.

What does that mean? You're working to be smooth in all the things. So imagine having a full cup of water on the dash. You don't want to spill any out the front, the back, or the sides of the cup, but you're doing this while driving over 100 miles an hour, inches from a concrete Jersey barrier, in traffic with other drivers on the track.

And you're doing this potentially, at least in my case, in a car that you want to be able to drive home.

So you can think of this as a very complex system, right? There are impacts from weather: is it sunny, is it cloudy, is it raining? What is the track pavement like? What are the cars in front of you doing? Are you ready to react if the car in front of you spins out? Do you remember where that one dip is that will ruin your day if you're braking while going into it? All of these variables need to be taken into consideration and constantly processed.

The consequences are bad if you start multitasking and thinking about other things. There's enough going on on the track that you really, really can't be thinking about problems at work, projects left undone at home, or anything else. Really, you need full focus to be in a flow state.

And guess what? This is a lot like making sure your engineers have dedicated and focused maker time. Complex systems are complex. Even if the task is relatively simple, if you aren't allowing your engineers the time to get into a state of flow where they can be focused on the mental model of what they're building and shipping, you're in for some negative consequences, particularly when you ship to production.

Maximize blocks of building time. Be intentional when you are interrupting flow time with things like meetings.

To summarize: multitasking bad. Make sure you are focused and not multitasking, whether you're driving or building software.

Sadly, being focused alone is not enough to enter a state of flow, either on the track or when delivering software. You need to be focused on the right thing. One thing you'll get drilled into you over and over and over again, and something I'm still working on, is look up when you're driving.

And again, it's as Don Kitch says: eyes up, look ahead, not in your rear-view mirror. What is behind you is of no consequence in life or on the racetrack.

So what does he mean? It's natural to want to focus on what's right in front of you, right? It's natural to focus on what you want to avoid hitting. But your brain is a bit funny. You will instinctively steer the car towards what you are looking at.

If there's an obstacle you're trying to avoid, you want to instead look at where you want the car to go, usually to the side. If you're looking at the obstacle, you're very likely going to hit the obstacle. On the street, this might be a pothole or a deer. Look where you want the car to go: to the side of the pothole or to the side of the deer, not at the deer.

And I can tell you this as someone who has hit a deer. It is not natural. It takes practice to make this looking where you want to go, not at what you're trying to avoid. It takes practice to make that instinctual.

It's also natural to focus on that turn you just bungled or whatever you just did that wasn't right. Maybe you got on the brakes too hard coming into a turn and you're kicking yourself a bit. Well, this is also bad because you are focusing on what just happened. You're looking in the rear-view mirror. You aren't looking up at what's ahead, and on the track that next thing coming ahead is coming fast.

You've got to have a plan and be reacting well before the turn or obstacle arrives in front of you. Look at that orange cone in the picture. At this point, you should be able to completely ignore it and have already worked out what the next turn looks like and how are you going to safely get through that.

This helps you drive smoothly and not react with jerky steering wheel inputs. So smooth is fast. Smooth is fast. There is time to retro what happened on the track when you're done, not while you're driving around the track, though.

When building software, something similar applies, right? Engineers are usually happiest and at their best when solving complex problems in front of them. Just make sure you have a few folks that are looking up and ahead, anticipating those future turns.

And this is where architects and other technology leaders should be contributing to the mission's success. Be aware of where the team is at, but mostly be looking up and ahead, focused on defining the next two turns and helping engineers draw the smooth line between where the system is today and where the system is headed.

When you're moving with velocity, looking up ahead lets you be smooth, and smooth is fast. Reactively jerking the team left and right, or between turns or initiatives, is going to slow you down and keep you out of a flow state.

That brings us to our second way: feedback.

When you're learning on track for the first time, you have a coach in the car right next to you, talking to you through an intercom because it's loud and you have helmets on, giving you immediate feedback about what you're doing wrong. And it is humbling. It is also powerful.

I can't stress enough how awesome it is to have that coach in the car and getting feedback on everything you're doing wrong, and occasionally, in my case, what I do right.

I end up with a bunch of takeaways that I then spend time working on solo until I feel like I've made enough progress, or I'm stuck enough, that I'm ready for my next coaching session.

When we're building complex systems, we have something similar. We found we got feedback from new joiners that, regardless of their experience, their background, they found onboarding and getting up to speed in our program to be daunting. It was large, a lot of moving pieces, very complex.

Well, much like having a professional coach that first time on the track, we became very intentional about our onboarding process and ensuring new joiners had an experienced person alongside to get oriented and to get feedback from.

Every new joiner got led through the overall program mission, the various products we were building and operating, our approach to on-call and operations, along with a bit of who's who in the program. The session was held every week and was facilitated by one of our senior leaders.

We learned how to have a standard set of Jira stories that went on the board for any team that was getting a new joiner. These stories included things like getting their equipment set up, access to appropriate systems, set up in our on-call tooling, and more.

And then we had a weekly program-level meeting to get feedback on where the new joiner was at in that onboarding process, and if there were any issues or blockers with the stories, we could then take that feedback and dig into those problem areas of our process.

So create feedback loops with both experts and process to bring new joiners up to speed. Getting those feedback loops established at the very beginning is crucial.

Another feedback loop when driving is feeling. You can actually feel what the car is doing, and it will kind of telegraph what it's about to do. So this is primarily right through your hands on the steering wheel and then through your back and your butt in the car seat.

To maximize the feedback, along with maximizing control of your car, seating and hand position is key. There is one proper way to be seated in a car, it turns out, or so I'm told. You need to be able to break your wrists over the steering wheel while you are fully sitting back in the seat, as I'm demonstrating here.

This is unlikely to be your normal driving position, and it's going to feel super awkward. Your back will feel too straight. You'll feel too close to the wheel. This is one of those trust me, give it a go things. It's crucial for controlling the car and receiving all the feedback that the car is giving you.

Once you have the seat position correctly, it's time to put your hands on the wheel correctly, and it turns out this is the correct way: nine and three. That's it. There's no other proper position. Again, if this is not how you're used to driving, I know that back in my driver's ed day I was taught something very different.

It's very awkward when you first start driving, but once you start doing this for a bit and turn it into a habit, you quickly realize that yeah, this is truly the way you have the most control over the car, and you're maximizing the amount of feedback you're able to get from the steering and the front suspension in pretty much any situation.

What's the equivalent in software delivery? I think it's well-structured log messages.

It took some learning the hard way, mostly at two o'clock in the morning, but our teams quickly realized that we had to do better with the feedback being given by our systems. We were not positioned in a way to understand what the systems were telling us clearly without doing a whole bunch more digging, and so we got crisp on our error messages and made sure they were human-readable in addition to being able to be parsed by log management.

This ended up being huge and made it a lot easier for our folks to manage our systems. So remember, at some point the human will be interpreting the feedback your system is giving. Set them up for success.

All right. Now the Third Way: learning.

This is a favorite quote of, again, Don Kitch, the chief instructor at ProFormance Racing School: insanity is doing the same thing over and over and expecting different results.

What does this mean? Well, if we don't experiment, if we don't try adjusting how we're doing things, we're not going to see a different result and learn something new.

The last time I was out at the track, it was raining hard. Don suggested intentionally getting the car a little loose on one of the slower turns. The key here is intentionality. You are causing the car to break free a bit rather than having it happen to you.

I practiced putting the car into a slight oversteer around one of the slower turns. It was a lot of fun, and I got practice on recovering from an oversteer.

Back in the world of building software, something we love to do on large programs is this idea of an innovation sprint. This is typically a week and happens around quarterly planning efforts. Used as a way to give the team a bit of a break, something different and interesting to work on, while at the same time injecting experimentation and innovation into the program. It's a way for us to avoid doing that same thing over and over again.

Team members submit ideas for the sprint, typically proof of concepts for something around that will make it easier to build or deliver or operate our software. They then self-form small teams to complete that work and demo it to the rest of our team and stakeholders. It ends up being a fun week, and some of that work is so valuable it gets prioritized in future sprints.

All right, back to driving. The logbook. Once you're qualified to drive solo on the track without an instructor, you're given a logbook. And as you're going through each of your practice, when you get off the track, this is where you log your observations: what worked, what didn't work, what you want to focus on next time. It's an indispensable tool for learning.

In much the same way, we use transparent post-incident review discussions for production outages, or even sometimes for near misses. We keep these learning-focused. What was confusing at the time? What are the assumptions? What did we learn about our systems? What did we learn about our process? What was surprising? Where did we get lucky?

We don't assign blame, but at the same time we ensure our teams feel accountable and empowered for improving the system and making on-call life better for the next rotation.

And over the life of the program, we found that these are sometimes even referenced during troubleshooting future incidents, if we think they might provide a hint around some of the system behavior we're seeing.

So whether you've invested in time on the track or you had an outage of one of your software products, it's important to get all of the learning you can out of that investment.

One last learning bit here. Technology has gotten us to a point where you can get a pretty realistic experience driving on your favorite track via something called sim racing, which is essentially a video game with a fancy steering wheel and pedals. And it lets you practice and experiment without worrying about getting sent to the hospital or destroying your car that you're hoping to drive back home.

Sky has become the limit in terms of spending for realism here. You can start with a very basic setup, like I have here, or spend your way into something far fancier and immersive, including full virtual reality.

Well, in software, we have something similar: game days. And our teams have gotten quite a bit of value out of this. We develop a scenario, often based on a real-world outage that happened on another product, and create that outage in a lower environment. The team then gets to work together to troubleshoot and bring that environment back up.

We want to make these both challenging and a bit fun. We'll then do a retro that looks like a post-incident review, and there's typically a lot of runbook updates, monitors, and better log messages that get generated out of these.

So practice frequently in a low-consequence environment.

We are out of time. So to keep two key takeaways. Driving is likely one of the most dangerous things we do daily. Hopefully I inspired a couple of you to do an HPDE event and learn to drive with intention.

Second, agencies within the U.S. federal government are operating using the Three Ways. You can and should too.

Thank you.