Transformed! A Musical DevOps Journey with Forrest Brazeal
Transformed! A Musical DevOps Journey with Forrest Brazeal
Chapters
Full transcript
The complete talk, organized by section.
Host Intro (Gene Kim)
I know exactly the moment in 2021 when Richard Seroter, whose work I've admired for many years, sent me something on Twitter. It was about this amazing song that the brilliant and talented Forrest Brazeal had composed and performed.
I was just leaving the parking lot of a grocery store with all three of my kids. I pulled over, played it for them, and they loved it as much as I did.
Obviously, I then tried to find everything I could that Forrest Brazeal had ever created, and in my conversations with him I discovered that we both were interested in Hamilton. We're obsessed with Lin-Manuel Miranda, and I must admit, in my grandest and wildest dreams, if there ever were to be a famous project musical, I'm hoping it would be as amazing as what Forrest has created.
The other thing I learned is that this is only just one sliver of the amazing things that Forrest has done. In particular, I continue to be blown away by the success stories from an initiative he created called the Cloud Resume Challenge.
So early last year I asked Forrest if he would be willing to talk about his amazing work and share some of his amazing creations with us, and I'm so delighted that he said yes.
And for me, one of the most amazing moments of DevOps Enterprise Summit in Las Vegas two months ago was seeing Forrest performing these amazing numbers on stage on this baby grand piano. We wanted to share that moment with you. So here's Forrest.
Forrest Brazeal
Good evening.
It's 5:41 p.m. Do you know where your DevOps transformation is?
We know where it started, don't we? Always starts the same way: starts with high hopes. Starts with big dreams. Starts with a vision.
[Singing]
Picture us, glorious, rising up at last to be reborn. With cloud, big data, and AI, we'll bid our legacy ways goodbye. We're caterpillars now, but we're gonna fly. Yeah, we're gonna be transformed digitally.
Never done it before, but how hard could it be? All these slide decks look so great. Set the deadline. Won't be long to wait till we declare victory digitally. How hard could it be?
Years pass. Oh yes, fighting still the fires we fought before. Our POCs looked great, so why are we not seeing ROI? And all the time our spend is climbing high. Oh, will we ever be transformed digitally? Why didn't anyone warn us how hard it would be?
Airport billboards had it wrong. Transformation's messy, and it's long, and we're failing miserably.
But wait. Maybe transformation isn't just about technology. Maybe it starts with us, the leaders.
When we address the questions that have always plagued our teams, like: when we ship code to production, what is causing all those screams? Have our results been feeble because we only pretend to train our people? And do we know our systems are secure? Are we sure?
And are we helping solve the problems that face our world today? If it's hybrid work or climate change, how must we change to lead the way?
If you want to turn into a butterfly, you've got to break down into goo and die. It's painful and it's gross, but if we try, well, we just might be transformed digitally.
Doesn't happen at once, but continuously. May not feel it happening, but gradually we'll grow and flex our wings. We'll achieve maturity digitally. Oh, this time you'll see: we'll be transformed, probably.
[End singing]
Thank you very much. My name is Forrest Brazeal. I work for Google Cloud, and I'm their head of developer media, which basically means I get to make all kinds of extremely unusual content that developers will enjoy. But before I worked at Google, I did cloud and DevOps kinds of things for a whole bunch of different companies, and this is what would happen to us: we would go on these multi-year DevOps transformation odysseys, and at the end of that time it sometimes felt like all we would have accomplished would have been to have refactored ourselves into slightly different silos.
But can I be real with you? Can I confess something to you tonight? Okay, good, good. Yeah, you're gonna have to give back tonight. I can't do this all by myself.
Listen, I kind of like silos. I do. I know. Listen, hear me out. I like them. They're safe. They're comfortable. They're great at optimizing communication within functional groups. Sometimes they're full of corn. There's really no downside to a silo until it's time to change the status quo, right? Time to change the way we do things, and then they become missile silos.
And I had seen this so much at different places over the course of my career. Not Google. I'm not talking about Google. Don't come after me, Google. But I'd seen this at so many places over the course of my career that I felt like I had to write a song about it, and the song is called "The Reorg Rag." Would you like to hear it? Yes? All right.
So just like most reorgs, this song is a little bit too difficult for me to pull off accurately in real time. So good luck to us all.
[Singing]
Well, our company used to have these big silos. We had Dev and Ops in two different rows, and they'd rarely communicate except to fight.
So we said, "We're gonna build ourselves a dedicated DevOps team, because everybody loves a go-between. We're gonna break these silos down. We're gonna do it right. We're gonna do a little reorg. Yes siree, this time we're gonna fix this company. We're gonna have this DevOps thing in the bag after one more round of the reorg rag."
Well, the DevOps team started hot as heck, but they soon became kind of a bottleneck, and instead of two silos, now we had three.
So we said, "We're gonna make Dev do their own Ops. Now it's all in the cloud. They can figure out how, and we believe the term for this is SRE. We're gonna do another reorg. Yes siree, this time we're gonna fix this company. We're gonna storm the silos and capture the flag after one more round of the reorg rag."
Well, it turns out Ops is incredibly tough, and our Devs soon said they'd had enough. So we brought back Ops, but now they needed more.
They said, "You're gonna need a platform team to guard and guide," and the DBAs wouldn't come along for the ride. We had silos everywhere, and they were all at war.
So here's where we're at. I'm on the tooling team, which is owned by Dev, but functionally it rolls up under this new Cloud CoE, which was recently spun out of Corp IT with a dotted line to Security, and some of those folks still report to me. I think technically I'm my own VP.
We're gonna need a bigger reorg. Yes siree, somebody put us out of our misery, because my optimism may start to sag if we play another round of the, play another round of the, play another, play another, play another, play another, play another, play another round of the reorg rag.
[End singing]
Thank you so much. But this is what we do, isn't it? And I think no matter what organizational structures or strictures we adopt, we're always at risk of falling back into these kind of adversarial org patterns because fundamentally we're human. We're a tribal species. We like to in-group around things, whether it's an organizational thing, whether it's a technology fad. We do that for sure.
And then once we've formed this new group, what do we do? We dig a deep moat around ourselves, and we take up pitchforks and torches to ward off the bad, ignorant people who are still using the paradigm from like four minutes ago. And we do that to distract ourselves from how small and insecure we really feel.
Speaking of things that are small and insecure, who here is using serverless functions in production? Anybody? Yeah, some of you. Great. How about Kubernetes? Who's using Kubernetes in production today? Yeah, some of you are. Wonderful. Who thinks it's all just a bunch of hype at this point?
Pathetic fence sitters, I see where you are. Well, listen, we're gonna settle this tonight once and for all. We're going to do it in the time-honored fashion. We're going to do it with a serverless versus containers rap battle. Are you in? All right. Let's do it.
Okay, now listen, as you've noticed, I'm one person, so I cannot do this by myself. I'm going to need your help. So this whole side of the room, everybody down here, whoever's up in the balcony, right down the middle, you're gonna be my serverless fans for tonight. Sorry if that hurts you. You're gonna be serverless fans for tonight. And when I have my serverless hat on, can you see that? When I have my serverless hat on, I need you to make some noise and show some love. Let me hear all my serverless people. All right. Good. Okay, all right. We've got a robust fan club.
Now on the other hand, when I have my containers hat on, I need this whole half of the room to show me some love. So when containers gives out a sick burn, right, I need you all to go, "Oh!" Let me hear y'all do that. Oh! All right. Okay. I don't know if any of us can win. It'll be all right. We'll get through it.
Okay. Wonderful. Well, in that case, I think we're ready to go. Are we ready? Okay, drop the beat.
[Singing / rap battle]
Yeah, you can call me crazy, but it's time. My mama raised me. Servers make me nervous, but serverless never faze me. Maybe I'm just lazy, but why deploy a box when abstraction brings the action for a fraction of the cost? My code is in a zip file, requirements in a pip file. Cram it in a function and bam, all I do is ship while standing proud in the cloud on the shoulders of the giants. What you need a server for? The value's in the clients.
Yeah, but I wish there was something more portable to wrap my app in. Oh wait, there is. It's called containers. They make crap happen. Check it out: I can build the way I'm used to. Open source is dope, of course. Run it where I choose to. Automate and orchestrate with Kubernetes at the helm. Hand on my tiller, I'm a killer in the service realm. Plus all these hello world tutorials are slick, so someone call the doctor, 'cause the stack is looking sick.
See, this is what you do. It's insane. You overcomplicate the name Kubernetes so arcane, you spell it with an eight. I'm no cloud economist, but I'm sure you don't want to miss the savings you could find if you'd put your mind to simplifying this. Are you building Pixar? Is your name John Lasseter? Then why you need a service mesh? Why you need Ambassador, Istio? Miss me, yo. This kingdom isn't magic. You throw it in production and the outage gonna be tragic.
Hey, take a breath, man. You ran kind of long there. Functions have a timeout. You've got to save your song there. You've got limitations. I run applications. Every enterprise has got their eyes on containerizations. Why? Long jobs, real time, doesn't matter. I eat hard problems for mealtimes, so pass the platter. So many industries left to disrupt. I know your cold start makes it hard, but try to keep up.
Check it: one, two, three. He's getting warmer, wait and see. The latency's no worse than your complacency. Face it, see, you're basically chasing a place you don't want to be. My services improve all by themselves. They get better. Meanwhile, you're out of luck, stuck chucking out the cheddar. Hey, remember Spectre and Meltdown? You were up all night. Me? I slept. That's right. The cloud provider kept it tight. You can patch your runtimes. I love happy fun times, delivering value while you fight the same old fight.
That's great. But wait, let's use our brains here. Yo, I've got constraints here. I'm running Java 8 here. Digging in the brownfield, moving the ball downfield. Can't re-architect it all until we look respectable. I just want to build more. That's what I get billed for. Functions are a blast our past selves would have killed for.
I know. I'm just saying we're in a different state of being. Yeah, but functions are amazing. Wait, are we agreeing?
Obviously, both of us have the same destination: get rid of heavy lifting without differentiation. So whether your abstraction is a function or a node, you can get a lot of traction on the old cloud road. If your app goes down at 3 a.m., and it will, you got to own that. It's your problem still. There's no silver bullet hocus-pocus magic guarantee, but when business is your focus, you'll be where you want to be.
[End singing]
Thank you very much.
All right. Well, I have one more song to play for you tonight, but before I get there, I want to introduce you to a fellow named Daniel Singletary. Because let's be real: serverless isn't the problem. Containers aren't the problem, right? There's no magic reorg that's going to deliver value to our customers. Even the right vision isn't enough. We can attend all these talks. We can underline all the right parts of The Phoenix Project. None of it means anything without what? Without the people who have to execute these things. You can't transform your organization unless you empower the people in it to transform themselves.
So I want to introduce you to Daniel Singletary. This is Daniel. I met Daniel a couple years ago right at the beginning of the pandemic. At the time, he was a residential and commercial plumber working in metro Atlanta. Doing all the things you would expect a plumber to do, on his feet a lot, long days.
Shortly before I met him, he got called out to a shopping mall there in Atlanta because the owners were reporting a strange and horrible smell pervading the entire building. Maybe something you would expect a plumber to deal with. So Daniel heads out there right away. The smell hits him in the face. He can detect it.
So he heads into one of the bathrooms. First troubleshooting step: takes one of the toilets off the floor. Instantly, he's hit with a blast of noxious air. Now, I'm not a plumber, but Daniel assures me that not only is this not something that should happen, it shouldn't even be possible. You shouldn't have currents of air blowing through your sewer system. So how do you break down a problem like that?
Well, in Daniel's case, he came back to the business with a colleague, and they started at opposite ends of the shopping mall, and they started working their way systematically toward each other. Over the next couple of days, they systematically unscrewed, tested, and re-screwed every pipe fitting in the building, trying to figure out where this horrible smell could be escaping. They couldn't find it.
Finally they got down to two businesses in the center, a restaurant and a hair salon, and they had to try something else. So this is where they tried a smoke test. Who here has done a smoke test in the context of software engineering? Yeah, a lot of you have. Did you know that's a real term with a real smoke-related meaning? I didn't until I met Daniel.
Guess what a smoke test is in the context of plumbing. You get up on the roof, you find a vent, and you drop a smoke bomb down it. That's a smoke test in plumbing. And the reason they do that is, you know anywhere that this gas could be getting out, smoke can get out too. But the difference is we can see smoke with a flashlight.
So that eventually clears up the problem. Turns out that someone has tied the vent hood for the restaurant's commercial stove into the sewer system, and that's what's causing this. It's always something like that, isn't it? That's what's causing this error.
It was around that time, I believe, that Daniel decided that he was ready to look for a new career. Maybe one where he could use what he'd smelled from time to time. And that is how Daniel and I met, because his roommate worked in IT, and his roommate pointed him to something that I had created called the Cloud Resume Challenge.
If you haven't seen the Cloud Resume Challenge, you can check it out. It's at cloudresumechallenge.dev, and it looks like this. It looks like a wall of text. It's basically a spec that tries to get people coming from non-traditional backgrounds, so people without a four-year college degree in computer science or whatever, people who haven't worked with cloud before, it tries to get them hands-on and learning and building like an engineer.
So it's not a video course you watch. It's not a certification you take. Although we do encourage you to get a certification, as you can see. But basically, it's giving you a spec with some instructions to put your personal website in the cloud. But there are specific things you have to do. You've got to use, well, here I'll show you an example.
This is a recent challenger's architecture diagram. And you can see they're having to use source control. There's some infrastructure as code built in here. They're using Terraform. They've got a cloud storage bucket going on. They built a small serverless API. And if you can complete all this, which is not a given, many people try this and can't get through it, you've actually built something pretty impressive. You've built something not only that a lot of people can't do coming out of a university computer science program, you've built something that a lot of professional cloud engineers haven't done.
In fact, every day we have people working on the Cloud Resume Challenge who are three, five years into their career, and they say, "Oh yeah, I've never strung all those skills together at once. I would definitely like to try that."
So put yourself in Daniel's shoes and imagine how he felt looking at these kinds of things for the first time, trying to decide if he wanted a career in technology. What do you think his reaction would have been? I'll tell you exactly what his reaction was. He said, "That looks familiar."
So he went out and he bought a whiteboard. This is his real whiteboard. He sent me a picture of it. And this is what he drew to try to make sense of the project he was about to undertake. Daniel called what he was drawing an engineered print, because that's what they call it in the trades. He did not realize he was drawing his very first cloud architecture diagram.
So over the next few weeks, Daniel spent part of his time working on sewer pipelines and the rest of the time working on CI/CD pipelines. Again, a real picture. Daniel, let's not ask what's on his shirt.
So he's working in the back of his work truck. He's getting home from these 11-hour days, and he's crunching Python and JavaScript, learning about YAML for the first time. Can we just pour one out for this man?
And so eventually, he thinks about giving up multiple times, but he finally completes this thing. He has it live. He has his website in the cloud. He's built all these full-stack front-end and back-end things to make it work. You can check this out. It is totally live. It is at dsresume.com. It's maybe not the most beautiful HTML-formatted thing you've ever seen in your life, but it doesn't matter. I call it the greatest resume I've ever seen, because just look at the credentials that Daniel has on his resume. Check this out. This man is a licensed journeyman plumber. He's certified in backflow prevention and cross-connection control inspection, and he's got four certifications on AWS. How awesome is that? I love that. I love that story.
So Daniel writes a blog post about his experiences. Actually, the first requirement for, or I should say the last requirement for, any one of these projects is you have to write a blog post about it and share it. He called his blog "A Plumber's Guide to Cloud." It ended up going viral on LinkedIn. It was viewed more than 200,000 times, and just a month or two later Daniel started his new career as a DevOps engineer at a large enterprise.
And as well he should, right? Because when I talked to the hiring team that had brought Daniel on, they told me they liked that he had done some technical things. I mean, that was cool. But let's be real: he's still very green. He's got a lot to learn. But they really liked that story of the leaf-blower sewer.
Think about what's being demonstrated by that story. Daniel is able to take a gigantic, kind of diffuse problem, this entire building stinks, he's able to break it down into achievable steps. He's able to work with a colleague, basically to pair program. Break the problem down, eliminate possibilities step by step until they arrive at the only solution. He gets tracing. He knows how to use tracing techniques. That's what that smoke testing thing was.
And most importantly, this man has a great grasp of business continuity. Right? He knew that he couldn't just kick out everybody in the restaurant while he meticulously took apart their entire plumbing system. That wasn't going to work. He was going to have to figure out how to troubleshoot this live. And so for that reason, and many others, he was a fantastic fit on his team and continues to kill it in his career today.
In fact, that team at that enterprise has actually gone on to hire, I think, six more non-traditional career changers because of the success that they had with Daniel. And Daniel's not the only person who's got a success story out of the Cloud Resume Challenge. There's Jacob, who was an infectious disease researcher. He's got some great COVID pandemic stories that are applied to tech. There's Adiam, who was in HR. There's Brooke, who worked at T-Mobile. We've had loggers go through. We've had professional poker players go through it. You can read a lot of their stories at cloudresumechallenge.dev. Many of these people are now my friends and colleagues in the industry.
But I wish I had even more of those stories to tell you, because let's be real: you know, it's tough to get a job in this industry at the best of times. It's tough to break in. I think a lot of us came up in an era where you could kind of just say, well, I scraped and scrabbled around for a few years and eventually I turned out okay, so the next generation will turn out okay too. I don't think that was ever true. It's definitely not true now.
Google Cloud is saying that we're going to need something like 40 million more cloud builders in the industry by the end of this decade. AWS has other but broadly similar numbers out there. Those people are not all going to come through that traditional four-year golden path where they get a computer science degree and take advantage of some of the green bean and new grad programs that you may have at your companies. They're not just going to be floating around on the open job market. We have to think bigger. We have to get more creative with how we source and find this talent.
And we have to be more creative and more intentional about how we onboard and create experiences for those people. It's interesting. This company that brought Daniel on, I told you they brought on several more people from the Cloud Resume Challenge, and what they ended up doing was kind of backing their way into creating an apprenticeship program at their company, as opposed to a traditional developer onboarding process.
And I think that's interesting because guess who had lots of experience going through an apprenticeship program? It was Daniel, right? That's how he got into plumbing. Very familiar with that, and some of the things that he had to do in the plumbing apprenticeship world, like going out and working under different plumbers. I don't know if you know this, but in the trades you don't just pin on to one practitioner. You get rotated around to different people through your local trade board. Some of those people were probably better to work with than others, but he had to do that.
And in the same way, when Daniel came on to his team in tech, they had to find senior engineers who were willing to basically pair with him, and they had to set expectations with those people. And it wasn't all their senior engineers. They found some people that were willing to do this, and then they kind of grew from there.
In the same way, Daniel got paid in his plumbing apprenticeship. He didn't do it pro bono. He didn't get paid much, but he did get paid for every hour that he worked. So this company that brought Daniel on had to figure out a way to make sure that he got paid a living wage. They had to set that expectation with their leadership team that, hey, even though this person isn't going to be contributing at full capacity for a little while, we think that there is substantial value in investing in someone like this. We think there's value in making sure that they have what they need to succeed.
And then finally, Daniel, in his plumbing apprentice world, when he completed a year or two, he was able to go on and get licensed. He was able to start out as a journeyman plumber on his own. So there had to be some clear progression and goal set as he was going through that process. It wasn't just nebulous and open-ended, and that's what ended up having to happen in tech as well. There had to be this clear understanding in the organization of: this person is an apprentice. You have to have different expectations for them right now, but they will get to a point where they're able to help. And here's what that's going to look like. I want to talk to you much more about that later tonight.
So my ask to you, everybody's been asking for help with different things, my ask is for you to help me help you help others. I want to talk to you after this about maybe apprenticeship programs you might be running at your company. I want to learn from you. I want to share what I've learned. We've put thousands of people through the Cloud Resume Challenge now over the past two and a half years. They all have unique stories. They have unique stories of where they've landed and how they've gotten brought up in the industry, and we need to be sharing those practices more broadly.
So please come find me. You can find me. See cloudresumechallenge.dev. There, see my Twitter, see my email address. Please reach out. I really want to hear from you. This is not part of my regular job. It's just something I do because it needs to be done, and I really want your help in doing it.
Let me leave you with this bottom line. Wait, let me leave you with this bottom line. Every one of us here in this industry, every one of us sitting in this room tonight, got here because at some point in the past somebody took a chance on you. Somebody looked at you when you had zero experience and said, "That person has the aptitude and motivation to succeed in this field, and I think I can teach them the rest." And you proved them right. So how are you paying that forward?
I have one more song to play for you tonight. I actually wrote this song a few months ago for my daughter, who was born in May. I wrote this song for my daughter, who was born in May, so like her, this song is new. You're the first public audience actually to ever hear it.
And I hope that as you go back and work with your teams and mentor them, create cultures of belonging there, this will resonate with you. The song is called "You Belong."
[Singing]
You won't always feel like a rock star. Sometimes you'll struggle just to carry the tune. In the mirror you'll see an impostor, and you'll wonder if the whole world's smarter than you, and stronger, and faster, and you're a disaster. But baby, you're wrong.
Because you belong, even when you don't believe it. You belong, I've got help here when you need it. Just hold on. Yeah, the road ahead looks steep, so take my hand. We'll make the leap together. Oh, oh, you belong.
Sometimes you'll feel kind of burned out. You'll work with folks who seem diffuse and unfair. You'll have a dream, and it turns out that no one else around you seems to care. And when you see what you're up against, it's fine to fight, to feel incensed, or just to move on.
Because you belong in a place where you're supported. You belong where you're challenged and rewarded. Just hold on. As you feel your power growing, you can choose the way you're going, and it's oh, oh, where you belong.
One of these days you're gonna reach the mountaintop. Yeah, it's not if, but when. And when you do, I hope you'll take a beat to stop and celebrate how far you've come. Remember where you've been, and then reach back to those behind you. Shine a light so they can find you. Help the ones you can.
Then sing to them again. Tell them you belong when you're new and nervous. Then you still belong. I know it 'cause I've been you. Just hold on. I know the road ahead is steep, so take my hand. We'll make the leap together. Oh, you belong.
Oh, come on. Sing it with me now. Oh, you belong. Oh, oh, that was good. Let's try it again. Oh, oh, where you belong. Oh.
[End singing]
Thank you very much. Enjoy the rest of DevOps Enterprise Summit.