Log in to watch

Log in or create a free account to watch this video.

Log in
San Francisco 2014
Share
Download slides

No Whiteboards Allowed

In this LA story, we find a successful advertising startup confronted with moving forward after a failed software development project. The team sets out on a life-or-death adventure to become the darling of the industry. Along the way they find themselves hamstrung by the traditions of their culture and tools.


Come hear how going from low-ops to devops helped them mitigate these challenges.

Chapters

Full transcript

The complete talk, organized by section.

Jessica DeVita

Hi, everyone.

Thanks so much for having me here. It's really a pleasure to see all of you. What a great conference this has been. Thanks for the introduction, Dominica.

My name is Jessica DeVita. I'm a technical evangelist at Microsoft, which is a really fun, exciting job. As an evangelist, I get to bring the good news of DevOps and cloud transformation for IT professionals. It's a lot of fun.

Before joining Microsoft, I had a consulting firm and had the privilege of working with some of the most complex entertainment and music firms down in Los Angeles. It was really incredible to see some of the transformations that were happening in these companies.

Every time I start a new project, I wanted to bring a little DevOps to every project. You can't always call it DevOps, because sometimes that word is maybe not serving you that well. It's got a lot of negative connotations at some companies.

The title of my presentation is "No Whiteboards Allowed," and I wanted to give you a little context of that. I'd worked with this company, and one story kind of stood out for me particularly. The CEO did not allow whiteboards in the office.

So how do you do DevOps when you can't have a whiteboard? It kind of doesn't work that well. So you do something I call a little stealth DevOps.

But the most important thing that I want you to take away from the presentation today is that, even in a low-ops or low-trust environment like the one I'm about to tell you about, if you're willing to be brave, to look at all the dysfunctions that might be happening in your organization, you can get some successes.

It's not going to be perfect. It's going to be messy. The earlier speaker was talking about the messiness of these kind of projects. So I'm going to share some of that stuck-in-the-mud story with you.

Because at the end of the day, for me, when I think about DevOps, I'm reminded of Mitch Hill, the former CEO of Chef, who sadly passed away last year. But he, in his ChefConf keynote, said, "Get it right and make it repeatable."

For me, that really captures the essence of what DevOps means to me and what does configuration management mean. For me, that just really kind of solidifies that and just makes it so very clear. I love that comment that he made. Sad I didn't get to meet him.

So a little bit of history. Down in Los Angeles, obviously home of massive entertainment companies, many of whom I've worked with for quite a number of years before coming to Microsoft. But this company, the CEO is a guy I knew really well. I'd worked with him on quite a few projects over about six or seven years. So I really knew him well and felt that I could help him, because I remember the day he called me and told me about this new company.

A little backstory is that this gentleman started a company back in the '80s, and it was a very successful advertising company. By the time he sold it in 2005, I think their revenue was over a half a billion dollars annually. So he was doing pretty well for himself.

When he called to tell me about his new project, I got excited. He really had the vision for what, to me, was going to be a very successful project. So I was excited, and I wanted to help him out. He saw this opportunity to kind of recycle the prior business model, and I saw it too, and I thought, "What could go wrong here? No problem. We don't need any whiteboards, right?"

I think from all outside viewpoints and metrics, this company was very successful. Every month, over 100 million downloads of their content. Small from an employee perspective, with 40 employees, but their sales last year were nothing to sneeze at. And in 2015, they're projected to do 22 million. I don't have that number up here.

So from that, looks like a pretty good company, right? They know what they're doing. They're obviously selling things, and people are buying things. So that's good.

Internally, they're kind of a hot mess. It's funny what you see with these numbers. But yeah, inside, which we're going to unwrap here a little bit, it was pretty ugly.

With an org structure like this, I think you can understand some of the challenges that I had trying to help this company. It was just awesome to see. I love this org chart. And this is really what I was dealing with: a bunch of really smart businessmen sitting around in a room arguing over artificial types of constraints.

For me, when I saw all of that happening, it was frustrating because I wanted them to get to what is the real constraint, and what are the real problems that we need to solve with this company?

So this is what I was working with. Again, I want you to get into the mindset of the CEO a little bit. I spent enough time there. For example, his assistant gave me a copy of this 50-page report that they would produce for him weekly, and he never even looked at it.

It was beautiful. It was something to marvel at. This was spiral-bound, graphs and charts and downloads and all that really important data, but it was too much data. Leadership does like to see their dashboards, but we need to condense it down and make it into something consumable for them.

A little bit more of the mindset is that this founder is still on AOL. They had not yet realized that they need to be a technology company first, that maybe just then happens to be in the advertising vertical. They just didn't understand.

We've heard that throughout this conference, the importance of being a technology company first. And this company hadn't sorted that out yet, so it was definitely hindering them.

How many of you have heard of Brent from The Phoenix Project? Yeah. We all have a Brent. Are you the Brent? Have you been the Brent? Do you have a Brent?

I have a special soft spot in my heart for Brent because I've been Brent in my earlier years as a sysadmin. Because what happens is sometimes companies think that we technologists can do anything. We talk about someone being a full-stack engineer. We can't do everything. We are not magic, and apparently they think we can do that.

In every organization, there's this comic so funny. He's like, "There's always somebody that knows what's going on, and we're going to fire him." And that's what was happening in this company. They didn't recognize how critical Brent, and I'm just going to call him Brent, had grown steadily.

He was originally hired to manage email accounts and printers and help people with the PowerPoints and stuff like that. But over the year, he had grown steadily more and more critical to the business.

He was the only person in the company who understood the complex vendor relationships that were involved in the handoff of the product from idea to conception through the value stream.

This is anonymized and quite boring, but if I look at the value stream and try to... Because without a whiteboard, we kind of have to do a mind map, which is another amazing tool in your DevOps toolkit. He doesn't have to see it, but I still get to process what's going on.

As I started to look at the value stream here, a couple of things became pretty clear to me. These critical dependencies would happen from the time the content was created, and there were four separate, distinct manual processes for these four different vendors to add in their little bits to make it a product that could eventually be sold and delivered.

The metrics in this industry are actually still being standardized. It's not standardized yet. How do you measure reach? How do you measure impressions? Because in advertising, you need to be able to sell that and say, "This is what I'm going to be able to deliver to you. I want your ad to be shown to 18-to-49 demographic, females, and who live on the East Coast."

These are the types of metrics that advertising is sort of built on. But the way you measure that has changed in this particular industry. So if you've got your dependencies that are, by the way, all undocumented except for Brent, Brent's the only guy who knows how it all works, you have some trust issues.

You can't trust Brent. You can't trust the vendors who are doing the handoff because you don't know at which point it's going to break down and how that's going to impact you. You don't even have a way to make an understanding of how the risk is going to be.

So as I looked at that process, I think about the cycle time. Dominica spoke about cycle time and lead time and how critical that is. So I like Marc Andreessen's quote about how important cycle time is to giving you a competitive advantage.

Some of the risks that were going on too, and maybe this is common knowledge, or this is more common than I want to believe, but one of the things I found is that they didn't even own their own domain name. And for me, I just sort of got a little sick when I saw that. Because the domain is valued at over $200,000, and this was sort of the elephant in the room. Nobody wants to talk about it.

And I'm just seeing how if vendor A, who, by the way, was holding that domain sort of hostage, if that relationship broke down, and it was breaking down because only Brent understood how that piece of it worked, I just saw this as just another glaring risk.

Then it comes from trust, or lack of trust, between the vendor and the CEO and how that all worked out. So that was frustrating to see that.

When I first heard about Conway's Law, I thought, surely they were talking about this company. They had this in mind.

One of the first projects that we were tasked with is this kind of go, no-go. Are we going to continue on with the development of this custom application that was commissioned? Which turned out to be, by the way, a really crappy Access database, but had been sold as this full-fledged business management product for this company.

Yeah, when I saw that, it was, "No, we're not going to do this." So a decision was made to kill that project, and ultimately, after that, they chose a software product that had been in use in the prior company.

So a couple of good things and bad things about that. Number one is they knew the product, so they trusted it, and very little customization was required for them to be able to put that into production. But like any kind of 20-year-old piece of software, it's really monolithic, pretty intense: three million lines of code in a Pervasive SQL database.

But I was really happy that they had picked something, and that they liked each other, and it was all going to go forward. So we went ahead and built this.

Some of the key, I guess, DevOps things that we did is a cloud-first approach. There's no data center happening at this company. They don't have a server guy. They've got Brent. So just putting it in a virtual machine in Server 2012 and putting that up in Azure and having that kind of auto-scale on demand.

So I considered that a success, as best as it could. And the funny story is, after this was all put into production and everything's live and running smoothly, the vendor of this crazy large application took me out to lunch and said, "How did you put our software in the cloud?"

He was absolutely baffled, like I had done some kind of magic or something. And I just explained it really wasn't rocket science. It's just like bare metal, except it's on somebody else's bare metal. I mean, there's more to it like that, but it was really funny to me that they didn't understand how I did that. They were pretty impressed with that.

So I'm thinking back too, in a retrospective of all of this pile of failure, and I'm reminded of Semantic Will's quote on Twitter: "Planning is really great for removing the fear of uncertainty, but really terrible at actually removing the uncertainty."

Because I think back about all these planning meetings that we sat in, these useful, should-be-very-productive stand-up meetings. We're getting stuff done. No, they were all sitting around pointing guns at each other.

It's like command and control, fear, absolutely scare your people into mediocrity. Yes, let's do this. It doesn't work. It's not good. Don't do this.

Yesterday, I saw Paul Reed's tweet, and I think he's left. He's gone to France now. But he was hearing a lot of employees having some fears about being replaced.

So I'm thinking back to Brent, and to me, it just explains so much of his behavior because, again, as he got more and more put onto his plate, he became really nasty. Very brittle, very angry, and more and more verbally sort of expressing his unhappiness, which I understand.

I'm trying to put myself in his shoes, and I think he wasn't valued. He wasn't staffed. He didn't have enough technical staff. It was just him. He was under-equipped to do the job that they were tasking him to do. And on the other side, they didn't understand what they were asking him to do and what he was really doing for their organization.

So by the time that this was all kind of coming to a very dramatic close, meetings and meetings and meetings of him, just everybody bitching about Brent, like, "How can we quantify the risk he presents to our organization?" Because he knew everything, and he didn't document it.

By the way, documentation solves all the things if you actually write it. That's a quote from Jennifer on Twitter. I'm trying to think of her Twitter handle, but I love that one.

So that's how we end up, in my opinion, where Brent just lost it. He just really became this very unhappy person, and I think that's how we end up with BOFH, if you know what that stands for. So I think that's to be expected in this kind of situation.

So I'm thinking back to when I first started learning about DevOps and a quote from Gene Kim came to mind. He's just really talking about this situation where leadership is just absolutely locked in this chronic strife and unable to get anything done. But all of these situations, I think, are really preventable.

I look back and I just think, could we have done maybe some requirements analysis a little bit at the front? I'm trying to think of all we could have done to make this any better, but I don't know that it could have changed.

There was a lot of cultural issues and a lot of ideas of, "I have this amazing company, and every one of you is all here to serve me and build this great company." But people don't work like that. You can't treat people like resources, to be plugged into a spreadsheet. We need to value our Brents, and hopefully, a lot less of these kinds of situations will occur.

The day that he stopped showing up to work was absolutely the best and the worst thing that could've happened. At that point, we'd done everything we could to mitigate the risks that he presented.

At some point, you just have to deal with what happens, and he just didn't show up to work one day. A lot of the operations schedule for that day was disrupted, and it represented significant monetary damage to the company.

We were very fortunate that he didn't do anything malicious, because there was definitely a concern of that. I obviously wanted to believe that his heart was in the right place. But he was a terrible communicator, and I don't think that served him well. So he didn't have all the skills he needed to advocate for himself, and I think he just reached a place where he couldn't deal with it anymore.

So the best and the worst thing happened. What was so great about it, actually in hindsight, is that all the tension in the room was just gone. Every day prior to that was like walking into... I don't know, you could feel the hatred and the tension in the room.

Getting anything done, having planning meetings, any of that stuff, it was really unpleasant. There was a lot of yelling, there was a lot of shouting going on, a lot of unreasonable demands. He just wasn't able to deal with that, and I don't blame the guy.

So this has been a little bit of a failure story in DevOps. A very much stuck-in-the-mud kind of story. Hopefully next year I'll have some success stories for you.

Do you have any questions?

Q&A

Q: Since you were a consultant doing this in lots of different places, I'm sure you saw some repeated stories. I'm curious, when you ran into places where there was this intention to kind of move ahead, and there was a good-faith effort that people wanted to become more Agile or use DevOps, but for whatever reason, could not let go of the way it has always been. I know this is kind of general, but strategies to deal with that? Because I think people know that there's a better way to go, but the fear factor is so great and they think, "Oh, the whole thing's going to fall apart."

A: So the question is, in other experiences, what did I see in other companies that were dealing with kind of the similar things?

Fear is an absolute hugest impediment to adopting Agile, to adopting DevOps. For me, the only thing that I can say is that action trumps fear any day. You have to do stealth DevOps. You have to make it work with the tools that you have and know that you're not going to get 100%.

Some companies, I don't know if they don't actually want help or want to get out of their own way. I'm not sure. I don't know if that helps.

Q: So my main question is, this can be very demoralizing.

A: Yeah.

Q: The last speaker said it, too. "I already feel like quitting." How do you keep your morale up?

A: I run every day.

Yeah. No, it is demoralizing, and there were some dark days, both working with that company and several other larger enterprises that I was working with at the same time. There was definitely an element of, "I don't know if I can keep doing this."

Well, now I work at Microsoft, so...

But I still have all those relationships, all of those companies. Those are guys I'd worked with for almost 10 years. So I'm still going to be there for them, moral support, I guess. But it's half technology, half psychology. Right?

Q: I was just going to ask, what about getting into the business that you left? A, did you leave for a reason in terms of not consulting, helping companies kind of become DevOps? And it seems to me there's a huge opportunity for folks at these conferences to go off and become evangelists, so to speak. Maybe not in the companies where they work, but as a career. What's your take on the market and how that goes?

A: To be an evangelist or to do consulting and sort of... Yeah. Well, so I always considered myself an evangelist before I joined Microsoft. I love technology, and I get super excited about it. When I can help some company get a big win and move forward, that's huge. That's very fulfilling for me.

So my advice would be to be the evangelist already. The job title will come, the consulting gigs will come. For me, it's just about personifying what that means to be an evangelist.

Q: So, a question and an observation.

A: Sure.

Q: Starting with the observation. So for all three days, we've heard over and over again that we are to be change agents.

A: Yeah.

Q: Being a change agent is difficult. You used the term evangelist, but remember that many evangelists were stoned to death too. Okay?

A: Yep.

Q: So just really, there's the observation. So the question is, since you said it's part technology and part psychology, for those of us who may be passionate about the tech and not so much on the psychology, what advice would you give us? How do you get smarter in that space?

A: Sure. I think the first piece is recognizing when you're dealing with a technology problem versus a people problem, and asking yourself that, asking the company that.

Because the technology, we know how to do that, right? We've solved that. The people stuff, that's not for you to change. You can model behavior and show them a better way, but it's up to them to want to change.

And so the type of consulting that I was in, literally every single client is really challenged in those ways. And I don't know why, but they like to help me.

Q: Otherwise, why would they call you?

A: Exactly. I don't know. For me, I just felt that the feedback I always got is they felt safe talking to me. They didn't feel like I was judging them.

Because we could look at those issues and see them for what they were and not internalize them as a fault of the person. But it's much easier with a whiteboard because then you're just all looking at the same data. Right?

When we're just having a meeting and griping about stuff, it's really hard to turn that into concrete action. Does that help? Maybe we can talk after.

Other questions? Any other questions? Okay. Thank you all for sharing your time with me. Appreciate it.

Thank you.