Elisabeth Hendrickson

So, this seems like a good time to mention a couple of things.

First, this talk would not have existed without Gene, because despite the fact that this has the same title as last year's talk, it's actually not. You should think of this as the sequel. And this talk ties together the stuff that I've been thinking and writing about for 15 years. And I wouldn't have made those connections, I wouldn't have connected the dots, if Gene hadn't pointed out to me that, "Hey, you've been writing about this stuff for 15 years." I wouldn't have thought about it that way.

The second thing that I should point out is that, yes, my handle is `testobsessed`, and you can find me on Twitter as `testobsessed`, you can find me on GitHub as `testobsessed`. Pretty much anywhere where there's a public handle, I am `testobsessed`. And people think that this is because I am a QA person and that I self-identify as a tester.

I don't.

Instead... well, you'll see, because it's kind of part of the talk.

So this is a talk with digressions. My first digression is that I would like to take you back in time to 1999, Silicon Valley. The first dot-com bubble that we referred to as the dot-com bubble because it was the first time there was a dot-com. So it wasn't the first bubble, but it was the first dot-com-bubble-like thing.

Anybody around in Silicon Valley in that time?

Okay. It was a crazy time. It was an insane time. And during that time, I worked for a small startup, as many of us did. I had worked for startups before, and in the past when I had worked for startups, there was an emphasis on do things cheap.

It was an amazing realization the day that I realized that at my little startup, my leadership would pay almost anything if we could ship one day sooner. So this is a completely different dynamic. I'm no longer optimizing for cost. I'm now optimizing for time.

However, remember, this is 1999. The term DevOps certainly didn't exist, and in fact, when I proposed something remarkably like continuous integration, the VP laughed me out of his office. So this was a completely different time, and cycle times were long and painful.

During this time, I participated in a working group called Steamer. This is where that paper that you saw the front page came from. In fact, Sam Guckenheimer, who's sitting in the front row, I think participated, if not in Steamer, then in LOST, Los Altos Workshop on Software Testing. Two working groups that Cem Kaner had started.

At Steamer 3 in the fall of 2000, we considered the question: what should the ideal ratio between testers and developers be?

This was a hot topic at the time. Microsoft had set the standard at about one-to-one, and so that was a much-discussed ratio. And my leadership in any company that I was with always wanted to know what the ratio should be, because that was how they wanted to do their staffing planning.

Now, at this working group, the format is there's a bunch of people, somewhere between 12 and 17 people, who are all sitting around and telling each other war stories. And then in a particularly stylized, facilitative approach, we pick apart each other's war stories and we really drill into the details.

So in this particular working group, the facilitator said, "All right. I would like to give you all two stickies. And on one sticky, I want you to write down the number, the ratio, for the project that was the very best project you ever worked on, the ratio of testers to developers. And on the second sticky, I want you to write down the ratio for the very worst project that you were ever part of."

And then he gathered up the stickies and we put them on the walls. And on the best, there was no real discernible pattern that was terribly interesting. But on the worst, they were all really high.

The worst projects had the most testers.

Now, we don't want to confuse correlation and causation. That would be the fundamental attribution error. We did not immediately want to assume that the testers caused the failure of these projects, but there was certainly something very interesting there. And frankly, that stuck so hard in my brain that it led to Cem Kaner and Jennifer Smith-Brock and me writing the paper.

By the way, the punchline in the paper is, "Ratios are irrelevant. Don't bother using them. Terrible staffing model. Use critical thinking." That was the punchline of the paper.

Now, time passed, and I was still at this little startup. The little startup, I still have nightmares about that place. It was the kind of place that just leaves you kind of a little bit nuts for a while.

So time passed, pressure increased at the startup, and a funny thing happened. I realized that our quality was getting worse. And this seemed odd because I was originally brought in to help improve quality, and I had done a lot of work to build up the test team there.

And I came to realize something really horrifying. This was the worst day in my professional career to date. I realized that the better I got at my job running a quality group, the worse I did for my company.

That's pretty depressing. Right? That's pretty bad.

So let me explain how that works. This is a system of effects diagram, and if we look over on the top, perceived quality. We started with a problem: perceived quality was too low. Management at that point gets to make a choice, and that's what the little flaggy things mean.

And if you look on the right-hand side, it says QA team test effort. So one choice that management can make, you've got low quality: we'll throw a bunch of QA people at it. That'll fix it.

Remember, this was 2000-ish, 1999, 2000. It was totally the accepted norm. That's how you fix your quality problem: you throw a bunch of testers at it.

So we tested away. We found a bunch of bugs. We were doing our jobs. This is good. We fixed a bunch of bugs. Now, theoretically, that means that we're shipping fewer bugs. We have fewer unknown bugs, and the bugs that we choose to ship are a known quantity. We've made a rational decision about those.

So theoretically, if we follow along with the system effect diagram, our perceived quality should go up. Sounds like a great theory, right?

Anybody at a company currently operating under this theory?

Oh, good. Because if you were, I have really bad news for you.

There's another effect within the system, and this was one that I hadn't even thought about until this moment when I realized that our quality is getting worse and I'm putting that together with the Steamer roundtable from the previous year. And I realized that up until I got there, the developers had felt responsible for doing all of the testing because the test effort had been so weak.

And the more they trusted me, the less testing that they did. So no wonder we were finding more bugs. It was a target-rich environment.

So we thought we were doing great because we're testing, we're finding all these bugs, we're fixing all these bugs. And what we didn't realize was our very existence increased the number of bugs that there were to find.

This is pretty depressing, right?

Okay. The talk gets better from here.

Okay. All right. So that was the flashback. That was the digression.

Now let's talk about feedback, because this talk is about care and feeding of your feedback cycles.

Feedback is actually a super simple concept. You do a thing, and then you observe, and you see the effect of your actions. Right? So feedback is the information that you get by observing the effect of your actions. Makes sense, right? Okay.

We have lots of fancy ways of describing this. We've got the Deming cycle: plan, do, check, act. We've got the OODA loop: observe, orient, decide, act. All of these are simply feedback cycles that boil down to do a thing, see what happened.

Okay?

We have more recently the Lean Startup method of build, measure, learn. They're all still just feedback loops.

There are various types of feedback that we might care about. As a programmer, did I do what I intended to do? When I check it in and CI kicks off, did my changes violate any expectations within the code?

Exploratory testing is looking for unintended consequences. At Pivotal, we do a whole lot of exploratory testing. In fact, we do so much of it that we have a whole position called explorer.

Did I get the feature that I asked for? This would be the acceptance feedback loop. Stakeholder feedback: are we headed in the right direction? And ultimately, the users or the customers: are we producing something of value?

And as we think about all these different types of feedback, they take different amounts of time. And so I think of this in my head kind of like a series of concentric circles.

The very fastest loops are at the developer's workstation, where they're running local tests. And if you're practicing test-driven development, that's a whole feedback loop in itself, because you set an expectation, you set an intention, you express that intention as an expectation in an executable test, one little, itty-bitty, tiny test. You then execute that test, see that it fails, write the code that makes it pass, watch it pass, and then refactor the code, and that's one loop. And it should be relatively tight, just minutes.

I've had, by the way, QA professionals ask me, "How do I insert myself in that cycle?" I said, "Pair with the developer," because that's the only way you insert yourself in that cycle. They're not going to ask you to write a test and then ping-pong it back to them over 50 feet of cube walls.

Okay, so as you think about the feedback cycle times, you'll note that there's this notion of latency.

So let's cast our minds back to that day when I realized that everything that I did was actually making things worse for the company. We were following a pretty traditional model. There was some market analysis. So we had product managers who would write these big MRDs, market requirement documents. Then the developers would go off and design some stuff, and then they would implement some stuff, and then they would hand it over to QA to stabilize.

This was the traditional model at the time. I realize nobody actually does this anymore.

So, as we're analyzing, the thing is, we're speculating. We're speculating that we actually identified the right problem to solve. We're speculating that we got the requirements right, that we understand the actual constraints that we need to solve to.

Then we design some stuff. It looks great on paper. Awesome. PowerPoint architecture always works.

Then we go off and we implement. We're still speculating, because even though we're learning a lot about the extent to which the PowerPoint architecture didn't actually work, what we're not learning is whether or not the implementation matches the expectations.

So then we get into test, and we start to learn a little bit, and ultimately the rubber hits the road when we ship and customers tell us whether or not what we shipped was crap.

The area under that curve is risk.

This is why it didn't work. Worse, when I realized that I was doing my company a disservice, I not only was participating in a system that increased risk, I was also driving requirements sideways through the organization because with every single bug that we filed, we were making an assertion about the actual requirements. Only those assertions were based more on opinion than on extensive market research because we were QA. We didn't go do market research.

So this is why this doesn't work.

I think of this like Schrödinger's cat. So this is a thought experiment in quantum physics that says, all right, hypothetically, you stick a furry critter in a box. The box is sealed. The box is not clear like this. It's opaque. You can't see in the box. You have no idea whether the animal is alive or dead. This is all theoretical.

It was a thought experiment that Schrödinger wrote to Einstein as they were debating effects of quantum mechanics. So then you hook this up to a valve. The valve is controlled by something that's got a 50/50 shot of going off or not, a decaying radioactive atom.

And if it goes off, then the cat's dead because the poison gas is released. And if it doesn't, then the cat's fine. And the punchline to the Schrödinger's cat thing is the idea that the cat is neither alive nor dead until you open the box. And in that moment when you open the box, then the probability wave collapses, and you know whether or not the cat's alive or dead.

This is pretty much what we were doing with releases. So we had the Schrödinger release. We had no idea if it was alive or dead until we actually opened up the box, talked to the customers, and found out how much they hated us.

There was one morning at that startup when I came into work, and I swear I could hear the screams of pain coming from support. It was sad. It was very sad.

All right, so theoretically, Agile is going to eliminate all of this risk because we do this in iterations. And so although we may have a little bit of speculation building up, it never actually gets to build up completely. So theoretically, this is eliminating all of our risk.

That only works if we're actually doing Agile. If we're doing fragile, which is where you have iterations, but you don't actually finish everything in the iteration. You're going to save that performance testing until later. You're going to save that full system testing because it takes a long time. You're going to save that to the end. But everything else is tested, so it'll be fine, right?

That's speculation, which means that we've got speculation build-up, which means that over time... doesn't that curve look awfully familiar?

That's why fragile and waterfall have the same exact characteristic with respect to risk. Only you've actually taken away all the controls that made waterfall kind of work, sort of, maybe. So now you don't have the controls in place at all, and so it's probably actually worse.

But this is not supposed to be a depressing talk, so I'm going to move on.

I really wish that I could have found a way to make this a picture of a squirrel, because then I could make this digression about squirrel. However, instead, I want to make this digression about fruit flies.

Anybody know why you use fruit flies in scientific experiments?

Short lifespan. Itty-bitty, teeny, tiny, short lifespan. Right. You get lots and lots of generations. Super awesome. You can have school kids do genetic experiments. Yay.

So feedback latency. Itty-bitty, teeny, tiny lifecycle. You get more turns of the crank.

Ultimately, that's what I learned out of all of this, is how critical it is that the feedback cycle, that the feedback latency, not be so large. And at that startup back in 1999, our feedback latency was excruciatingly large.

We were proud to get our regression cycle down from six weeks to two weeks. These days, I'm complaining at my development team that it takes hours upon hours to do the regressions, but they still finish in total runtime of something like eight hours. And if we had had an eight-hour fully automated regression cycle in 1999, I would have thought I was amazing and that the team was amazing.

Anyway.

So let's talk about an example of tightening feedback cycles. This is an example from, actually, Cloud Foundry. So I work at Pivotal, and I was Director of Engineering for Cloud Foundry for a couple of years.

And when Pivotal Labs... This was before Pivotal was born. Pivotal is only about two and a half years old, and before Pivotal the company was born, Pivotal Labs got involved with this project at VMware called Cloud Foundry. Now, of course, these are all part of Pivotal the company.

But at the time when Pivotal Labs got involved, Pivotal Labs practices a very strict form of extreme programming, so a lot of focus on tight feedback loops. And we got involved with this project, and what we found was that it took forever to get a full regression cycle.

And in fact, I had developers at the time who told me, "Do you want to know what my week looks like? It looks like I check in code on Monday. It goes through the Gerrit code review system. Nobody bothers to plus-two it, so it sits there. And meanwhile, all of the other more senior team members are getting their commits in, which means that every single day, I spend my entire day doing a Git pull, merging, retesting, resubmitting, and hoping that maybe today, maybe, just maybe, my little, itty-bitty, teeny, tiny code change will get to go in."

That's pretty depressing, too, right?

I realize that what I'm about to say may constitute a religious statement, and I am going to apologize in advance if I offend any of your particular religions, but I took great joy in ripping out the Gerrit code review system.

It was one of the biggest impediments to getting actual, real feedback. I value code reviews, but the way that it was implemented in that organization at that time prevented change from going in. And anything in your process that becomes that kind of bottleneck that prevents change from going in is preventing feedback cycles from being tight.

So we went from: you check in, there's simultaneously some unit tests, some review, then there's this merge feature-branch-like thing, then more unit test, then more system test. And this picture here is actually a little bit simpler than it was. I didn't go into all of the nooks and crannies of the process.

We went from that, and it took days to weeks to get a change in and then get feedback on it. Real honest-to-goodness feedback.

By the way, there is a difference between feedback and opinion in the way that I'm using it. An opinion is merely speculation. And remember what happens when you have speculation build up? Big risk curve thing. Right.

So here, when I'm talking about real feedback, it's empirical evidence, because empirical evidence trumps speculation every single time.

So after we simplified, the way we did code review is because we were doing a very strict form of extreme programming, we paired on all the things, so pair is a living code review all the time. So we're pairing. We're running the local tests before we check in. When we check in, we check in directly to master. We weren't doing feature branches at all, even. And then CI picked it up and ran all of the tests, not just the ones that could run quickly on your local machine.

And the end result was that even when the system grew so that it was at a sufficient level of complexity that the system tests took a couple hours, still, it's two hours instead of a week. So that's tremendously driving down the time in the feedback cycle.

More recently, in another part of Pivotal, we are shifting to tighten up feedback cycles in a way that, frankly, I've been a little surprised by what I've learned. We had very long-lived team branches on one of our other products, by which I mean two months. So a team goes off and develops away for a couple of months. This was their history. This is how they used to work.

And it felt great at the time, right? Like, "We get to ignore all those other people. We're so productive. We're writing code. We control our destiny. We're in our branch."

And then it comes time to merge. You all know what happens then, right? Everything comes to a screeching halt for four weeks while we try to get everything slammed together and make things get back to green.

So this was not actually nearly as effective as it felt at the time, but it feels so good for just those few weeks when you have the illusion of progress.

And so instead, we're moving to short-lived feature branches. And the end result of that is that we're getting these changes in much faster, which means we're able to get real feedback on those changes. So now we've tightened up those feedback cycles.

And ultimately, again, remember the fruit fly. Now we've got much shorter-lived branches. We get more turns of the crank. We get more opportunities to fix things. We get feedback faster.

One thing that I have had to learn about, though, is that fast feedback doesn't do you any good if it's a completely polluted feedback stream. False alarms, false failures cause people to lose faith in the system, and that ultimately can lead to a series of dysfunctions, including the pattern of, "Oh, it's red again. Just kick off the build. I'm sure it's fine. Let's ship anyway."

You ever seen that?

Sad, right? Okay.

Distortions. Remember that feedback is not the same as opinion. My team at that little startup was actually distorting the feedback stream with a fair amount of opinion.

And then finally, entirely broken feedback loops, like if you don't have a solid feedback loop from support. This is something that we're also working on at Pivotal, to make sure that we've got that feedback loop so tight that we get that feedback from our customers, not just when there's an escalation, but also to start to mine the incredible gift of information that is in every single support case. Getting that back into the system so that we can make sure that we're acting on that and steering, using that, making sure that your feedback streams are not polluted: super critical.

Another digression. This time I can't say squirrel, but I can say hamster. Those are little hamster erasers. They're super cute.

Once upon a time, before I joined Pivotal, I ran a little consulting company. I was running Quality Tree Software, Incorporated when I wrote that little paper that Cem saw, or sorry, that Gene showed. And at the time, my job was get on airplanes and go help people.

And I would run this simulation to help them as they were transitioning to Agile, called WordCount. And in it, it becomes a microcosm for an Agile transformation. I set up the room at the beginning of the day with four areas: one for the testers, one for the developers, one for the product managers, and then a funny little area for the computer, which is played by human actors executing written instructions that are written by the developers.

And so this is this little simulation. They are not allowed to talk in the beginning of the simulation. They can only communicate through the interoffice mail courier, who is another person in the simulation, who walks around with little envelopes and passes messages between the tables.

As you can imagine, the effect of running this simulation, it tends to resemble the company that brought me in as a consultant as they run it. And there's much laughter because, in fact, it resembled reality for a lot of those companies, where their developers and their testers and their product managers didn't actually talk except through email.

So as I would run this simulation, in the first round, they have to follow these rules. In the second round, they get to make up their own rules. And then we would do four rounds, and there would be plenty of opportunity to reflect and adapt.

I ran this 150 times.

So one of the adaptations that most groups would adopt very early on is, first of all, to do away with the interoffice mail courier. They were almost always fired in round two. I only had one company that continued to employ their interoffice mail courier throughout the entire four rounds. They had tremendous difficulty making money in this... it's fake money, but in this simulation.

The other thing that would frequently happen is they would realize the power of visibility. So the people in the simulation would start to put all the artifacts up on a board where everybody could see everything all at once. And then you'd start to see the division between the teams melt, and then you'd start to see them really crank and able to turn out value very quickly.

We see something similar. So at Pivotal, in the parts of the organization... Pivotal, you have to understand, is the world's biggest startup. We're 1,700 people. We are only two and a half years old. So imagine forming a company of 1,700 people with only two and a half years of shared history. It's pretty amazing how well we've done.

And one of the things that we did on the Cloud Foundry team, they created this thing called Concourse. And what you're seeing here is Concourse. It's a CI system. It's open source, and it gives you a tremendous view of your entire pipeline because the software we work on, on our products, it's all enterprise software, massively complex pipelines.

So by just being able to visualize our pipelines and see red in a different way than just having a Jenkins build shows it to us is tremendously powerful. We're adopting this elsewhere within the organization.

The other kinds of things that we're getting visibility on are all of our escalations for all of our data products are now in a single dashboard. And by having them all in one place, it's prompting conversations that nobody had ever had before around, "Wait a minute. What exactly does constitute an escalation? What is and is not?"

And so just by making things visible, we're having better, higher quality conversations that are improving our ability to serve our customers.

So we start to get to the takeaways portion of this talk.

One of the things that I've learned is that without extreme effort, you're going to suffer from feedback entropy. So I view this as anything that's causing feedback to break down. One thing is that test suites and build times, therefore, are going to grow unbounded.

As one of our engineers says, "You do realize that test suites are an append-only thing, right? Nobody ever goes back and takes tests out." And that means that the test time, left unchecked, it will just continue to grow unbounded. And you'll end up with thousands and thousands of tests, and nobody actually knows what value those tests have. But dang, they take 12 hours to run every single time, and we've got a policy that says we have to run and have them green before we can ship.

So left unchecked, test suites and build times will therefore grow unbounded.

The feedback will become polluted naturally. It's something that you have to pay a lot of attention to. Opinion will start to creep in. Information will start to get dropped. If you don't take care of those tests and the build and all of the CI infrastructure, you'll have false failures.

And it's too tempting. It's so tempting to just say, "Oh, it's fine. Just run it again. It'll be fine." Because it actually takes real effort and dedication and commitment to go in and say, "Wait, why did this thing fail? Okay, it shouldn't have failed for that reason. I'm going to go fix it right this time so that it stays fixed."

That takes the willingness to set aside whatever else might have felt like a high priority to treat fixing the infrastructure as a high priority. And that's a level of discipline that a lot of organizations don't have. And I can appreciate how difficult it is because every single day, I end up making a decision like that. No matter what else was important from a product-facing standpoint, if we don't have clean CI runs and we don't have clean infrastructure, we can't trust our feedback.

And ultimately, it is so tempting to seek the illusion of speed over real progress. That temptation to say, "I know how we're going to go really fast. We're going to ignore everything else. We're going to ignore everybody else. We're just going to work on our team branch because that way we don't have to worry about these painful merges. We're just going to get our feature done."

It's not actually delivering value until it's in the shipping product. But it's so hard to remember that when it's feeling so good to write code.

So it takes such effort. Fighting that feedback entropy takes an enormous amount of energy and dedication and cannot be underestimated.

One of the other things I've learned is that meetings are easy. By the way, I broke the rules. I don't have five takeaways. I have six.

Meetings are easy and getting real work done is hard. So at Pivotal, if you want to get anything done, it has to end up in a team's backlog. And early on in my tenure there, we would have these meetings about how we were going to tighten the feedback cycles or improve the test suites or whatever. And the meetings felt awesome because I'm working with incredibly smart people who are all incredibly test-obsessed, and they care deeply about the feedback cycles. They care deeply about CI.

And we'd have these fantastic meetings, and we'd whiteboard out all the things, and then nothing would happen. And I came to realize that if you want something to happen, it has to end up in a backlog. And so the action items to come out of that meeting, instead of just being action items, because nobody ever did the action items... We made lists of action items. It's not like we were just talking at each other.

The action item had to be, "All right, who's going to put this item in in which backlog?" Because then ultimately it becomes a discussion with the team and the product manager to figure out how to prioritize that work. But if it doesn't end up in a backlog, it's not going to happen.

So it's so easy to have a great meeting and so hard to make something actually come of that within your organization structure, whatever that is. You have to figure out, how do we make sure that we've made the commitment to actually do the work?

On both Cloud Foundry and on the data teams, we've now built toolsmith teams. And one of the lessons that I've learned is that the tools that these teams are building, they are foundational.

So there was a wonderful blog post about "let a thousand flowers bloom and then rip out 999" from Gigamonkey, who was at Twitter. And I forget his real name, and I'm really sorry. And if you're in the audience, I'm going to be incredibly embarrassed about that.

But oh my goodness, what a wonderful, wonderful blog post about their engineering effectiveness organization and how they thought about engineering effectiveness and the joy of coding.

We had a similar challenge, and we have a similar process for making sure that we have tools that make a developer's life easy. And what we figured out is that the team that makes those tools had better be some of the best developers in the organization.

And it's so tempting to put the new grads on that team, to say, "That's an easy on-ramp into the organization." It's so tempting to say, "It's just a script. It's just Python," right? That's so tempting. But the fact is that these are foundational. This is the foundation on which the rest of your software is built. And if that foundation is shaky, then that means that the software you're building is going to be even shakier.

Visibility turns out to be more important than I ever even imagined. And just getting visibility into things can cause change.

Figuring out where to draw the lines in the system, this requires just a teeny bit of explanation. So if you're writing a class, you have to figure out, is this one method or two? Do I extract this out into a private method that's a helper method to this thing?

Even if you've never written code, you can appreciate how this might be an interesting challenge. How do I draw the lines there?

Then there's, if you're doing architecture at an architectural level: is this one module or two? What should the responsibilities for this module be? Should this module take responsibility for this other stuff? That's at an architectural level.

At an organizational level: should this be one team or two teams? Should we have centralized QA, or should we have all of our QA people be part of our development teams, or should we just not have anybody with that title whatsoever?

These are all decisions in which we decide where in our organization, our system, our code to draw the lines. And it turns out that one of the hardest things about designing anything is figuring out where to draw those lines.

And by experimenting on where should the lines be, you can find new perspectives and also transformational ways to shift your organization. And that's one of the things that we've been experimenting with, is where to draw the lines.

Thus, Cornelia, who's sitting in the second row over here, introduced me to someone today as, "Yes, she's the person who came onto Cloud Foundry as the Director of Quality Engineering and promptly got rid of all of QA." Which pretty much sums it up. There was a little bit more there to the story that I'll skip for now, but that's an example of drawing the lines.

And then finally, I always knew that branching was a little bit painful, but oh, branching hurts productivity far more than I ever thought.

So those are my takeaways. That's the end of what I came to say.

Almost. Sorry, forgot two more things.

How to care for your feedback cycles. One is to make sure that they're tight. Two is to make sure that they're clean, thus the rubber ducky. And then finally, the super-secret sauce is that the Kolb learning cycle is just another example of a feedback cycle.

So as you're improving your ability within your organization to turn the crank faster, to have those better feedback cycles, ultimately, you're becoming a learning organization.

And as Andrew Clay Shafer, who also works for Pivotal and is known as Littleidea on Twitter, as he says, "If you're not becoming a learning organization, your company will be beat by somebody who is."

So there's no failure. There's only learning, and that's what I came to say.