Data blindness

It's a disability that many people find refreshingly comfortable

Most of us are swimming in a sea of data about our products, companies, and teams. Too much of this data is non-actionable. That’s because many of our reports feed us vanity metrics: numbers that make us look good but don’t really help make decisions.

Yet even among those who have access to good actionable metrics, I’ve noticed a phenomenon that prevents taking maximum advantage of data. It’s a condition I call datablindness, and it's a painful affliction.

Imagine you are crossing the street. You constantly assess the situation, looking for hazards and timing your movements carefully to get across safely. Now imagine the herculean task that faces those who are blind. That they can function so well in our inhospitable modern life is impressive. But imagine if a blind person had to navigate the street as follows: whenever they wanted to know about their surroundings, they had to ask for a report. Sometime later, a guide would rattle off useful information, like the density of cars in the immediate vicinity, how that density compares to historical averages, the average mass and velocity of recent cars. Is that a good substitute for vision? I think we can all agree that it wouldn’t get most people across the street.

That’s what most startup decisions are like. Because of the extreme unknowns inherent in startup situations, we are all blind – to the realities of what customers what, market dynamics, and competitive threats. In order to use data effectively, we have to find ways to overcome this blindness. Periodic or on-demand reports are one possibility, but we can do much better. We can achieve a level of insight about our surroundings that is much more like vision. We can learn to see.

I got a powerful taste of data blindness recently, as I’ve started to work with various large companies as partners in setting up events, speeches, and other products to sell around the Lean Startup concept. Yet, whenever I find myself transitioning responsibility for one of these events to these third-parties, I have this sudden sensation of loss. I suddenly lose my ability to judge if our marketing programs are being effective. I start to get very fuzzy on questions like “are we making progress towards our goals?” In other words, I’m experiencing data blindness.

What’s happening? Mostly, I’m no longer being hit over the head with data.

For example, a recent event I held started with a customer validation exercise (actually, this example is fictionalized for clarity). I had it all set up to a jury-rigged SurveyMonkey-PayPal minimum viable product. It was pretty ugly, the marketing and design sucked, and I was embarrassed by it. Yet it had one huge advantage. Whenever someone decided to buy a ticket, I got an email immediately letting me know. So throughout the process of taking deposits and then selling seats, I was getting constant impossible-to-ignore feedback about how I was doing. For example, I quickly learned that when I twittered about the event, more often than not I would make a sale. Yet, when I tried other forms of promotion, I’d have to accept their failures when the emails failed to come. True, this wasn’t nearly as good as a true split-testing environment, but it was powerful nonetheless.

Now that I put on events with official hosts and sponsors, my experience is different. Of course, I can still get access to the data about who’s signing up and when – and a lot more analytics, to boot – but I have to ask. Asking imposes overhead. When I get a response, when someone tells me “hey, we had 3 more signups” I’m never quite sure if those are the same three signups I heard about yesterday, and this person just has somewhat stale information, or if we had three new ones. And of course, if I twitter about the workshop on a Friday afternoon, I won’t know if that had any impact until Monday – unless I want to be a pain and bother someone on their weekend. There are lots of good reasons why I can’t have instantaneous access to this data, and each partner has their own. I wonder if their internal marketing folks are as datablind as I feel. It’s not a pleasant sensation.

Let me give another example (as usual, a lightly fictionalized composite) drawn from my consulting practice. This startup has been busy transforming their culture and process to incorporate split-testing. I remember a period where they were suffering from acute datablindness. The creators of split-tests were disconnected from the results. So the product development team was busy creating lots of split-tests for lots of hypotheses. Each day, the analytics team would share a report with them that had the details of how each test was doing. But for a variety of reasons, nobody was reading these reports. The number of active experiments was constantly growing, individual tests were never getting completed. This had bad effects on the user experience, but much worse was the fact that the company was expending energy measuring but not really learning.

The solution turned out to be surprisingly simple. It required two things. First, we had to revise the way the reports were presented. Instead of a giant table that packed in a lot of information about the ever-growing list of experiments, we gave each experiment it’s own report, complete with basic visualizations of which variation was currently more successful. (This is one of the three A’s of metrics: Accessible). Second, we changed the process for creating a split-test to integrate it in with the team’s story prioritization process. The Product Owner would not mark an experiment-related story as “done” until the team had collected enough data to make a decision about the outcome of the experiment relative to their expectations. Since only a certain number of stories can be in-progress at any one time, these experiment stories threaten to clog up the pipeline and prevent new work from starting. That’s causing the Product Owner and team to spend more time with each other reviewing the results of experiments, which is allowing them to learn and iterate much faster. Within a few weeks, they have already discovered that huge parts of their product, which cause a lot of extra work for the product development team due to their complexity, are not affecting customer behavior at all. They’ve learned to see this waste.

Curing data blindness isn’t easy, because unlike real blindness, data blindness is a disability that many people find refreshingly comfortable. When we only have selective access to data, it’s much easier to be reassured that we’re making progress, or even to fall back on judging progress by how busy our team is. For a lean startup, this lack of discipline is anathema. So how do we reduce data blindness?

1. Have data cause interrupts. We have to invent process mechanisms that force decision makers to regularly confront the results of their decisions. This has to happen with regularity, and without too much time elapsing, or else we might forget what decisions we made. When the incidence rate is small, emails or text messages are a great mechanism. That’s why we have operations alerts trigger a page, but it can also work for other customer events. I’ve often wanted to wire up a bell to sales data, so that when we make a sale, we literally hear the cash register ring.
   
   When the volume is too high for these kinds of tricks, we can still create effective interrupts. Imagine if the creator of a new split-test received a daily email with the results of that test, including the computer’s judgment of which branch was winning. Or imagine an automatic system that caused the creator of a new feature to get daily updates on its usage for the first three weeks of it being live. Certainly our marketing team should be getting real-time alerts about the impact of a new promotion or ad blitz.
   
   
2. Require data to justify decisions. Whenever you see someone making a decision, ask them what data they looked at. Remember that data can come in qualitative as well as quantitative forms. Just the act of asking can have powerful effects. It serves as a regular reminder that it’s possible to make data-based decisions, even if it’s not easy. When you hear someone say that they think it would have been impossible to use data to influence their decision, that might be a signal to investigate via root cause analysis.
   
   My experience is that companies that ask questions about how decisions get made are much more meritocratic than those that don’t. Any human organization is vulnerable to politics and cults of personality. Curing datablindness is not a complete antidote, but it can provide an alternative route for well-intentioned people to advocate for what they think is right.
   
   
3. Use pilot programs. Another variation on this theme is to consistently pilot new initiatives before rolling them out to full-scale release. This is true for split-testing features, but it’s also true for marketing programs or even operations changes. In general, avoid make big all-at-once changes. Insist on showing that the idea works in micro-scale, and then proceed to roll it out on a larger scale. There are a lot of advantages to piloting, but the one that bears on datablindness is this: it’s extremely difficult to argue that your pilot program is a success without referring back to the expectations that got it funded in the first place. At a minimum, the pilot team will have to consult a bunch of data right before their final “success” presentation. As people get more and more used to piloting, they will start to ask themselves “why wait until the last minute?” (See Management Challenges for the 21st Century by Peter Drucker for more on this thesis.)
   

Luckily, data blindness is not an incurable condition. Have stories of how you’ve seen it cured? Share them in the comments, so we can all learn how to eradicate it.

(Image source: scumdoctor.com)