previous article
Being data-driven is one of the hallmarks of a modern organization. In practice, as opposed to management theory, data always comes with a certain measurement error. It is never 100 percent accurate. Everybody knows this, but only very few people act on it. This blog article invites you to find out what happens when you take measurement error seriously. Let us take the red pill together.
Data-driven decisions: a commendable example
Imagine the following scenario, fictitious, but not unrealistic: A large company routinely uses data on the size of the markets they are operating in. They buy market research data that covers revenues of markets around the globe, broken down into detailed product categories. This data is being used throughout the company to inform key strategic decisions on sales approach, production sites, and other important topics. The most widely used KPI is the percentage growth of revenue compared to the previous year. This number may be looked at for the overall business, or for smaller product and country segments, depending on the decision it is used for. Everybody is very happy to have these hard facts and proud to base their decisions on data, not gut feeling.
Agile decisions, driven by data
When you talk to people in the company, they tell you how much they’ve learnt from this treasure trove of data. You talk to Bob. He’s a country manager, and he tells you: “Before we had this data, I thought I knew my business. The countries I’m responsible for are mature markets, and our industry is a very stable one. So, basically, everything grows at more-or-less the same rate every year, I thought. But then I saw the real market data for the first time, and I was in for a surprise! There is no steadiness in the markets. They’re moving all the time. At first, it was a shock. I realized I had to change the way I manage things. I always tried to keep a steady hand. But in a reality of moving markets, you can’t stand still. I’m taking decisions much more rapidly now. And I am prepared to change them quickly. We have all become a lot more agile around here. Some of us struggle with the loss of predictability and security. But it’s important to move quickly if you want to survive.” Bob looks proud when he says that. He also looks exhausted. The new numbers have arrived last week, and they show a sudden downturn in the largest of the markets Bob is responsible for. Layoffs are discussed, and Bob is not sure whether they can avoid them.
Data-driven or noise-haunted?
When you ask Bob why the numbers are so bad in that country, he shrugs. “Honestly, I haven’t figured that out yet. It’s not the overall economy, you know. People still have enough money in their pockets. Before the data came in, I would have assumed that we’ll see maybe a 5% growth. I assumed 5% for this country most of the time before we had the actual data, you know. It’s a mature market, but also a rich one in which people like to buy new and better stuff often.” And then you do it: You swallow the red pill. It enables you to see not only the data, but also the error contained in it. And you begin to wonder: This is market research data, right? It’s produced from surveys being sent out to the companies in the industry. What’s a reasonable error for this kind of thing? It’s hard to guess without knowing the details of the data acquisition, but 2% is certainly on the optimistic side. What if Bob is actually right, and in reality, this market grows by a steady 5 % each year? What would that look like when you see it through the distorted lens of a 2% measurement error? You open your laptop and do a quick simulation:
import altair as alt
import pandas as pd
import numpy as np
np.random.seed(9876) # you might want to try other values here
def error_simulation(measurement_error: float, growth: float, start_value_abs: int = 100000,
first_year: int = 2004, last_year: int = 2018) -> pd.DataFrame:
'''Simulate growth at a constant rate, with and without measurement error.
:param measurement_error: Measurement error (between 0 and 1, e.g., 0.02 for 2%)
:param growth: Growth rate (usually values like 0.01 for 1%, 0.05 for 5%, etc)
:param start_value_abs: absolut value in the first year
:param first_year: starting year of simulation
:param last_year: last year of simulation
:return: Pandas dataframe with observed and error-free data for absolute and relative simulated values
'''
current_wo_error_abs_value = start_value_abs
growth_factor = 1 + growth
wo_error_abs_value = [] # error-free absolute value
wo_error_rel_value = [np.nan] # error-free relative value (percentage growth against previous year)
observed_abs_value = [] # observed absolute value, including measurement error
observed_rel_value = [np.nan] # observed relative value, including measurement error
for i in range(last_year - first_year + 1):
wo_error_abs_value.append(current_wo_error_abs_value)
rand_error = np.random.uniform(low=-measurement_error, high=measurement_error)
observed_abs_value.append(round(current_wo_error_abs_value + current_wo_error_abs_value * rand_error))
if i > 0:
observed_rel_value.append((observed_abs_value[i] - observed_abs_value[i-1])/observed_abs_value[i-1])
wo_error_rel_value.append((wo_error_abs_value[i] - wo_error_abs_value[i-1])/wo_error_abs_value[i-1])
current_wo_error_abs_value = round(current_wo_error_abs_value * growth_factor)
return pd.DataFrame({'year': range(first_year, last_year + 1), 'observed_abs_value': observed_abs_value,
'observed_rel_value': observed_rel_value, 'wo_error_abs_value': wo_error_abs_value,
'wo_error_rel_value': wo_error_rel_value})
simulation = error_simulation(measurement_error=0.02, growth=0.05)
# define charts without error
wo_error_abs_chart = alt.Chart(simulation).mark_line().encode(
x='year:Q',
y='wo_error_abs_value:Q'
)
wo_error_rel_chart = alt.Chart(simulation).mark_line().encode(
x='year:Q',
y='wo_error_rel_value:Q'
)
# define charts with error
observed_abs_chart = alt.Chart(simulation).mark_line().encode(
x='year:Q',
y='observed_abs_value:Q'
)
observed_rel_chart = alt.Chart(simulation).mark_line().encode(
x='year:Q',
y='observed_rel_value:Q'
)
# put all charts next to and below each other into a single figure
(wo_error_abs_chart|wo_error_rel_chart) & (observed_abs_chart|observed_rel_chart)Fooled by randomness
The simulation results are startling (see figure below). Although we know the real growth rate in our simulation is 5%, and the measurement error is only 2%, the observed growth rates fluctuate wildly, from somewhere near 3% to over 8%. This is not an extreme result caused by an unlucky choice of start values for the random generator. It is quite representative as a normal simulation run, as can be verified by playing with the start value in the notebook. So, what happens here? The measurement error has only a moderate effect on the absolute figures, as you can see in the lower left panel of the figure. But the growth rate perspective magnifies the errors and results in an erratic curve. If you base your decisions on this data, you will be haunted by noise: your decisions will be as erratic as the curve. Calling it agile won’t make it any better.
Don’t believe a statistic you haven’t understood yourself
So, what can we do? First of all, it’s important to understand that this kind of over-interpretation happens frequently. It is by no means limited to growth rates. Nassim Nicholas Taleb’s “Fooled by Randomness” contains many more examples, and it is a pleasurable read by a highly cultured author. But the solution to handling this kind of problem is much older. It comes from 18th century philosopher Immanuel Kant, the most important thinker of the enlightenment. He chose the old Latin phrase “Sapere aude!” as his motto and translated it as “Dare to think for yourself!”.
Sapere aude!
And that is what we need to do: Use data only after we’ve understood it ourselves, including its flaws and limitations. Sometimes, this means that you just can’t use the data in the way you would like to. Find other ways, or better data, but never give in to the temptation of just ignoring the error.
