The numbers trap
More and more data and metrics play a significant role in my life as a designer of digital products. You can measure everything about an end-user; how often do they visit your product? What do they look at for and for how long? What do they buy…? The numbers are endless, and yet in my experience they tell only half the story.
A while ago I was working on a project that made this clear. When I met with the metrics team they would confidently tell me that there were four types of users who exhibited different behaviors when they visited our website service. For example, one kind of user would linger on the site, while another kind of person would immediately find the content they needed and make a purchase. To me, this seemed odd—who were these four different kinds of people?
But the metrics told us that there were four types of users and we should customize the experience of our service around these four user types, because the customer is always right, right?
One day a colleague who was curious about this data visited a call center to hear what people did when they called in to buy products instead of buying them directly from the website. While listening to the calls something interesting happened; a person would call in to buy something and just before they were about to buy they would hesitate and rethink their purchase. The call center operator told my colleague to just wait, the customer who had hesitated would call back in a little while.
Sure enough, the customer called back. They had resolved to buy the product but had a few more questions. Once again they got to the point of purchase and then found a reason not to go through with it. Again, the call center operator said the customer would call back. And they did—two more times. On the final call they made their purchase almost immediately, having gotten all the facts and justified to themselves that they really needed the product through the earlier calls.
What was interesting was that this behavior matched the online data, which had been interpreted as identifying four different types of customers. What the data had not told us was that this wasn’t four different types of people, but was in fact one person exhibiting four different types of behavior over time as they made their mind up to make a purchase.
Connecting this qualitative data to the quantitative data revealed a much richer picture of what was going on and how people were making decisions. This changed our outlook on what we were making, and how we should design the product to accommodate these different behaviors and needs.
Too often one-dimensional mass data is used to make decisions. Even worse than our misinterpreted data, many types of data are deeply flawed, don’t measure what they purport to, or are biased, like IQ tests. And many crucial things have no reliable measurement, like how much pain a person is in.
Data and its analysis rules much of our lives, from taxes to healthcare, and yet much of what is important cannot be measured directly and often has no measurement at all. In the upcoming couple of posts, I will explore some of these examples which cover money, the brain, and the body, and show how often what we think are facts are in reality just half the story.
