Host Intro

Hi there. Thanks for joining us today. We have Tom Limoncelli from Stack Overflow, and he's going to talk to you today about low-context DevOps. Tom, over to you.

Tom Limoncelli

Hi. Welcome to Low-Context DevOps. My name is Tom Limoncelli. I'm the manager of the SRE team here at Stack Overflow. I've been a system administrator for much too long. I blog, I tweet, and I've written a number of books. And if you believe the back cover of the books, I'm an internationally recognized system administrator and DevOps pundit.

I mentioned I work at Stack Overflow. We have over 100 million users in 2019. Demographers say that there are about 30 million software developers in the world, which means for every three developers you meet, 10 of them use our website.

Now, what's less known about our website is Stack Overflow for Teams. This is a product that gives a private Q&A experience for your team, and maybe we'll talk about that in the future.

So my talk has three parts. We're going to talk about the sociology concept of high- and low-context cultures. We're going to talk about low-context DevOps, a phrase that I coined recently, and we're going to end by talking about leadership.

01Part one: high- and low-context cultures

So part one: high- and low-context cultures.

Let's begin with a story. Story number one: a man is traveling in a foreign land, and he enters a village. He goes into the first shop, and the shopkeeper won't talk to him. He goes into another shop, and they won't talk to him either, and he's quite perplexed. So he's sitting on the side of the road wondering what to do, and a little boy comes up to him and says, "Mister, the reason that they're not talking to you is first, when you come to a village, you have to talk to the elders and get their blessing. And then once you have their blessing, everyone will talk to you." Well, he does that and everything goes fine after that. Everyone will talk to him.

Well, that night he's thinking to himself, "How the heck was I supposed to know that?" And amazingly enough, at that same time, the little boy was thinking, "How could anyone not know that?"

Story number two. Story number two is about my first week at Stack Overflow. I've been at Stack Overflow for about seven years, and I still remember my training because there was this point where I asked how to create a virtual machine. Now, we use a product called VMware, and my mentor walked me through the process. It was five very complicated steps, and it wasn't written down. It was just verbally passed on from one sysadmin to the other. And I said, "Wow, this is really complicated. How can anyone memorize this?" And he said, "Well, we kind of expected anyone that would get through our interview process would be able to just know how to do this kind of stuff."

Well, that night I remember thinking, "How could I have been expected to know all of that?" And I wonder if he was sitting at home that night thinking, "Gosh, we've hired this world-recognized system administrator. How could he have not known how to do these things?"

Well, there's a postscript to this story. The next day, I got a phone call from my boss who said we had made a mistake in the process. And that just goes to show that even though one person was very experienced, even he had not memorized the whole process.

So these first two stories are examples of high-context cultures. In a high-context culture, communication is implicit. Things aren't really written down. People just know the collective history. People have to read between the lines to understand what's going on often. Often, what's not said is more important than what is said.

In a high-context culture, we usually rely on long-term relationships. So this might work better for situations where you're going to be there a long time. Some examples include a party with friends or family gatherings. There's just norms and customs that aren't written down; people just know to do them.

Contrast this with a low-context culture. In a low-context culture, communication is explicit. There are rules. You're told what the rules are, and you follow the rules. Knowledge tends to be codified. It's written down. It's public. It's accessible. They make it accessible so that you can follow the rules.

A low-context culture works best for short-duration interpersonal connections. For example, a large airport. You're not going to be at the airport very long. There's a lot of rules, and that's why they're on signs that you can read and learn that way.

Another benefit of low-context culture is that the knowledge is more transferable. Because it's written down, the next person can come along and read it.

Sociologists that created this concept have ranked languages around the world as being high context, low context, or somewhere in between. And while this is a generalization, Eastern languages tend to be known as being more high context. Western languages tend to be more low context.

One of my favorite examples locally of a high-context environment is New York Penn Station. Stack Overflow is headquartered in New York City, and I live in New Jersey, which means I take the train into work every day. And Penn Station is just an incredibly high-context environment. In fact, this is a map of Penn Station. This map only makes sense to people that created the map. This is otherwise pretty much useless. You can't even tell where the second floor and the first floor would overlap onto each other.

But my favorite thing in Penn Station is this sign right here, the sign that says, "Seventh Avenue Subway to the left." This sign only makes sense if you knew that 30 years ago, the Seventh Avenue subway line changed its name to the 123. That is, to me, the epitome of a high-context culture.

Now, recently, I was thinking about this sign, and I realized it's illuminated. There's a light bulb behind it, and light bulbs don't last 30 years, which means for the last 30 years, someone's been changing that light bulb, and that person is not empowered to tell their boss, "Hey, maybe we should update the sign."

02Low-context DevOps

Let's talk about low-context DevOps. I assert that DevOps environments should strive to be low context, and I hope you agree.

You should spend more time working and less time frustrated with roadblocks and information gaps. But most of all, we shouldn't just change the light bulb; we should fix the darn sign.

So let's talk about three ways to lower the context of your environment: carefully constructed defaults, making right easy, and ubiquitous documentation.

03Carefully constructed defaults

Carefully constructed defaults means the defaults should be the way you expect most people to work. For example, when you join a new company as an engineer, you probably need three things to be productive. You need a PC, you need the software required to do your job, and you need access to various systems and the correct permissions to do what you need to do.

Typical employees don't have all three of these things for weeks or sometimes months. One friend of mine was not able to do her job. She didn't have the software she needed to do her job until six months into working there, and this is very typical.

Why doesn't this get fixed? Well, as a new employee that's feeling the pain, you can't fix it. You're the new employee. You're not empowered to fix things. And experienced employees don't feel the pain anymore. They've got their setup. They're all working. So they don't feel the pain, so there's no incentive to fix it.

Plus, even if you do want to fix it, oh boy, to fix these things, you're going to have to work across so many different silos: the IT department, InfoSec, engineering, HR. So many different organizations are involved in the onboarding process. Could you imagine, if you want to fix this problem, just having to bring all these people together to get that done? The problem is that if you don't do it, no one else will.

So there's a new book, The Unicorn Project, by Gene Kim. It's the follow-up to The Phoenix Project, which is an excellent book that kind of crystallized the DevOps movement. And now Gene Kim has this new book, The Unicorn Project, which is about this very problem: how do we make the new environment for employees workable?

And you might be thinking, "Well, yes, Tom, that's good, but I'm going to stay at this company for a long time. So it's a little overhead, but who cares?" Well, you might be at a company for many, many years, but you're probably going to be changing projects all the time. And every time you change projects, it's like being a new employee. And for a company to be really dynamic, you need people to be able to change projects quickly. Plus, hopefully, you're always hiring and growing, and you want those people to be productive as soon as possible.

04Making right easy

The next thing we're going to talk about is making right easy. Or another way of saying it is, the lazy path should guide you to the right way. I have a couple examples of this.

The first is LibreSSL versus OpenSSL. OpenSSL is the technology that most websites around the world use to have a secure connection: HTTPS instead of HTTP. And if you've ever used OpenSSL, it requires a freaking PhD to get it all configured right. If you're going to build a secure application, there's so many knobs to turn and settings to set, and even if you do get them all right, six months later, all of those settings could be completely stale because some hash algorithm is proven to be less secure than we expected, or there's a new encryption scheme, or a new protocol. And so OpenSSL is very difficult to get right and stay right.

The LibreSSL fork is done by a number of people who said, "Let's make right easy." They came up with a new API that is timelessly correct. So if you do the lazy thing and just say, "Listen on port 443," it does everything correct in a timeless way.

So we can learn from this. We can embody this kind of thinking in lots of different parts of our environment. For example, at Stack Overflow, our CI/CD pipeline embodies the recommended practices. If you just go with the defaults, you're going to get all of our recommended practices.

Another example would be the base libraries. A friend of mine at another company explained to me the problem that they were having. He said, "It's difficult enough to hire a good programmer, but it's almost impossible to find a good programmer that also understands how to write code that's operational." By operational, I mean runs well on a 24/7 website: collects telemetry for monitoring, standard flags, all these different things that are important at an operational level.

Well, they embodied all of those things as the default part of their base library. So now, if you're writing a "Hello, World!" web server, you can't not have all those good operational qualities. You could diverge from their standards, but you would have to go out of your way. And we're busy people, so people generally stay with the defaults.

We can embody this thinking in all aspects of our foundational tools in our DevOps world, from our ticket system to our source code control system, to just how we do our OS setup.

05Ubiquitous documentation

Third thing I want to talk about is ubiquitous documentation. I mentioned earlier that knowledge should be codified in a low-context environment. Low-context environments make knowledge accessible. That's what I want to embody in my DevOps world. I want documentation when I need them, where I need them, and that's easy to do with a deep-link URL. This is not like the old days where you had to specify a book and a page number and chapter and all that kind of stuff.

Things are so much easier now that we have the web. So let's use this technology. Let's put our URLs in error messages, in our control panels, in our alert messages, wherever people might want the information when they'll need it. You know who gets this? Apple gets this. I don't know if you've upgraded your Macintosh lately, but they've changed the default shell in the terminal. And if you're using the old shell, you get this message. It tells you, "We've changed the shell. Here's the command so that you could change along with us, and if you need more information, here's the URL that explains exactly why we're doing this."

So how do we create a culture of documentation? Here are my four favorite ways of creating a culture of documentation.

First of all, management has to set expectations. It's not done until it's documented. And I don't mean just big projects. I mean every little project. Every time you close a ticket, you should update the documentation so that the next person that has a similar ticket has an easier time.

Second, file bugs about docs just like software. Documentation should be treated just like software. You file bugs about it. Those bugs become visible. Time is allocated to fixing them, et cetera. This is very important because stale documentation is dangerous, and everyone should feel empowered to update a document. And if you aren't the right person to update the document, you should at least make it visible so someone else can.

That relates to creating a culture of always updating documentation as you work. You should always be documenting. Don't think about documentation as something you do at the end of the project. Document as you do your work.

And lastly, you need to have good responses to excuses to not document. For example, when a developer says, "My code is the documentation," you have to say, "Well, at least there needs to be documentation that points people to that code." Give enough information so that people can get started.

Another way of inspiring people who might be resistant to writing documentation is point out the fact that you'll be able to relax better while you're on vacation. Many engineers haven't taken vacation in a long, long time because they feel like they can't. Well, the better they document, the more relaxed they can be while on vacation. Also, another way of thinking about it is, if my stuff is really documented well, someone else can do my job, and I can go on to more interesting projects.

Now, some people say to me, "Tom, this all sounds great, but docs? What docs? My organization has no documentation at all." And that makes sense because if I asked everyone to raise your hand if you like documentation, I have a feeling that not a lot of people would raise their hand. And those that do, well, maybe they're just raising their hand to make me feel better.

So let's talk about why people avoid documentation, and let's try to come up with some ways to fix that. I think there's three big reasons. First, I know that often when I start to write a document, I get kind of flustered at the scope. Like, what am I trying to write here? Is this the document that is going to have to explain the whole history of things, or just the specific how to do this one task? And I get flustered enough that it becomes a reason to procrastinate, and then I look at my watch and it's lunchtime. I go to lunch, and I never come back to this document.

Similarly, I'm often uncertain what the audience is. Do I have to write this for someone who just joined the company, and I have to explain every little detail, or can I just focus on the task at hand?

And thirdly, there's a really scary thing, and I want to warn you before I turn to this next slide. This is the scariest thing to writers. This is more scary than any Stephen King novel or movie. Okay, I'll show you, but promise me that you're sitting down and ready for a scary slide. The scary thing is the blank screen. Yes, the blank screen is probably the scariest thing to writers. And just to drill down on this a little bit, here is a blank Vim document. For you Microsoft Windows users, here's what a Word document looks like. For you junior people, this is a blank Notepad. For you more sophisticated people, blank Sublime. And of course, because it's trendy, we wouldn't be complete without dark mode blank screen.

Blank screen syndrome is a real thing. It's something that writers talk about all the time. And while not a recognized medical condition, blank screen syndrome is a very real problem.

So how can we work against this? Well, first of all, we have really awesome templates. In my environment, every service in our ecosystem has what we call the service doc, and it's hard to write. Well, it was hard to write until we came up with a template. And now, it's not like you're writing this document, you're just kind of filling out a form. And you could leave certain items blank and come back to them later, or let someone else fill it out, but it gets you started and avoids that blank screen.

Similarly, every alert that we get in our monitoring system, we have a template. This is the stuff that gets documented about each alert. It brings consistency, and also it's very important because when you get paged at 4:00 in the morning, you're not your full self. So being able to just find something in a template, and you know it's always the same place in the template for suggested resolution, you can just start following from there.

Another thing you can do is encourage everyone to write in small batches. I mentioned before: every time you close a ticket, update the documentation so that it's easier for the next person. That's exactly what I mean by writing in small batches. Don't do a project and say, "Oh, I'll do all the documentation at the end." Keep the documentation updating as you're going.

And related to that, include documentation updates in work estimates. So when you tell your boss this is going to take three days plus one day to do the documentation, you're hurting yourself. Instead, just say, "This is a four-day project." Don't think of documentation as this extra thing. Think of it as part of the project itself.

The last thing I'd like to suggest is find where engineers already write and repurpose that. So if someone sends you email with a great explanation of something, reply with a compliment: "Hey, what a great explanation. You should put this in our document repository. Then it would help everyone." They've already done the work. Now you're making it visible all over the place. I do this in email, in chat rooms, instant message. It's a great way to motivate people, and how many times a day do you have an opportunity to compliment a coworker? Well, here's another opportunity.

You know where else engineers write? Even engineers that hate to write, for some reason, they love posting to StackOverflow.com, and I'll tell you why. It solves those problems that we talked about earlier. There's a very specific scope. Someone asks a question, and you know the scope. You have to just answer that question. You know the audience. Based on the question, you kind of guess. You can guess that this is a beginner or an expert, and you can target your response to that audience. And there's also templates and wizards that help you through both the question-asking stage as well as the answering stage.

I mentioned a little foreshadowing. So I don't want to turn this into a big commercial, but I would like to point out that Stack Overflow for Teams does give you that question-and-answer environment private for your team, and it's a great way to leverage that energy that the engineers have.

And it's also nice that new employees can fix or update questions or answers. Experienced employees who are maybe seeing pain of others can take an opportunity to fix things. I met one of our customers who said he maintains this library that's used across his entire company, and he never got feedback about it. But he created a tag in his personal Stack Overflow. The tag was the name of his library, and he started watching for any question tagged with that tag. And every day, he just started seeing this flow of questions, and it became the way that he was able to help people, and he felt so good about being able to help people. And it let people use this great library throughout the company. And we're seeing great success in IT, in engineering, software development, all sorts of areas.

06Leadership

The last thing I want to talk about is leadership. So we've talked about a lot of different things today, from smart defaults to templates, et cetera. Who is going to make these things happen? Well, it's going to be you. It's got to be you, because you're the person that's concerned about these things.

Now, I give a lot of talks at conferences, and people often come up to me afterwards and they say, "Yeah, I liked your ideas, but I'm not a manager. I can't make this happen." And whoa, whoa. Managers and leadership are two very different things, and I want everyone to be a leader. So first, what's the difference?

Management is a thing on the org chart. It's a thing for HR. Management is about setting priorities, providing resources to meet those priorities, and clearing roadblocks along the way. It's something that's in your org chart or your business card.

But leadership is something everyone can do. Leadership is about going first, clearing the way, and making it easy for others to follow. I think this is so well symbolized by the kids' game Follow the Leader. That person in front, they're going first, and they're making it easy for others to follow. So maybe they're walking around and they get to a tree with a low-hanging branch, and it doesn't matter how they're going to get through this branch. They could duck underneath it, or they could lift the branch and walk under it. The decision doesn't particularly matter, but the decision has to be made, and that leader makes that decision, and that makes it easy for everyone else to follow.

So you can make that template and everyone else can follow along. You can build the system, maybe build half of it, but create the scaffolding so that other people can join along and plug in their addition. I also like to think of the stone soup metaphor. You're kind of setting up the kitchen and having everyone else bring their bits.

So we've talked about a lot of things today. I hope you agree that DevOps environments should strive to be low context, and there's a couple ways we can do this. We can focus on smart defaults, we can make right easy, and we can make ubiquitous documentation.

There are a number of impediments to writing documentation. There's the burden of the audience and the scope, the blank screen syndrome, and we can fix these things with templates, templates, templates, repurposing text from where engineers are already comfortable writing, and providing a central source or repository for people to put the documentation.

I hope you enjoyed my talk. Thank you for attending.