W Edwards Deming is often quoted as saying: “In god we trust, all others must bring data”.
His point was that if you want to improve an operation; hunches, beliefs and superstitions just won’t cut it. You need to have the data that shows what is really going on. No data, no improvement.
Unfortunately the data isn’t enough. Data by itself won’t tell you much, not unless you can interpret it properly.
Tragically, most people think that interpreting data is easy, but in truth, it is downright hard.
It is hard not to jump to the wrong conclusion
The Psychologist Thomas Gilovich studies the way people make decisions. He has shown how, when faced with difficult problems and data, most people jump to completely the wrong conclusion.
One of his nastier examples is the bombardment of London during World War Two by V1 and V2 rockets, or, as they were less than affectionately known, “doodlebugs”.
Flying bombs showered down on London in 1944 and 1945 killing thousands of people. The map below shows where they landed in central London. (Warning, it is less than pin point accurate, it was the best I could manage with a paper map and Power Point).
Precision bombing
The bombs clustered along the banks of the Thames and to the North West over Euston and Regents Park. This is even clearer if I divide the map into quadrants.
It was of obvious concern to Londoners that the enemy were so precise when firing their rockets from a hundred miles away in Northern France. Many speculated that German spies were living in Finsbury and Clerkenwell (north and east of the river). That would explain why that area was protected from the devastation.
But it was little more than point and shoot
The truth though was very different, all the Germans were doing was pointing the bombs in the right general direction and hoping they had put in enough fuel. If I cut the map diagonally, the evidence behind the “precision bombing” looks a lot less compelling.
When is a pattern really a pattern?
So the pattern that everybody saw and recognised wasn’t really a pattern at all.
Which leads to my question:
Is that data about your business telling you something significant or is it just noise?
This is not the place for a detailed statistics lesson (and I would quickly reach the end of my knowledge). But if you don’t know how to answer my question you could do worse than employ and analyst or two who can.
If nothing else, it will save you from the embarrassment of accusing your neighbours of spying on you.
If you enjoyed this post click here for updates delivered straight to your inbox
Read another opinion
Hello James
Great post which shows the limitations of data. Data is useless without someone to make it into a story. Why? Because human beings most easily relate to / make sense of stories. Put differently, story was the original structure/vehicle for communication. It is what we do best.
So data ends up as story. The fascinating matter is that given the same data different people will make different stories. Which is to say it is ALL interpretation at the human level. And as you point out there are in built biases when it comes to interpretation – these occur automatically.
Of all these biases, the one that shows up the strongest is the one that threatens the comfort/security/self-esteem of the storyteller. You can be confident that it is a rare storyteller that comes up with a story that makes him look foolish, threatens his income/livelihood, and creates graft for him.
In my days leading a data mining and predictive analytics practice the most valuable lesson I learnt was this: come up with a simple story that appeals to the ‘background’ of the people listening. So executives like the ‘strategy/numbers’ story, marketing like the ‘brand/creative/targeting’ story etc….
Maz
Tell the people the story they want to hear.
There is a very slippery slope
But a very good point
Thank you Maz
Hi James,
Great post. I guess what I’d like to know is how many V1s and V2s were actually launched, how many landed on London, how many exploded etc etc
Even then it might not tell us anything useful….bit like trying to read something useful from your tea leaves.
Adrian
Hi
Interesting post James. I like what you said there Adrian; I believe that the more you dig the more information you are likely to find; them be right or faulty there is always something that can be concluded.
Thanks for your comment Ibrahim, I suppose the trick is knowing when to stop digging
The data that were used in the analysis can be seen; search for
flying bombs distribution
and use the site from Coventry University (nestor.coventry.ac.uk).
Thanks for the tip David
Here is the link http://nestor.coventry.ac.uk/~nhunt/poisson/dbug.html
James
Funny, that everybody believes the numbers. If you would use your system 2 and count on your own , you would see, there is a mistake in the last figur. Without mistake you may read it in the book of Thaler and Sunstein (2008) Nudges,
Truth be told I probably read it there Johannes :) Thanks for reading