Beware the “Success Story” (The Mismeasure of Candidates, Part 2)

Here is an amazing true story: I flipped a coin 20 times, and it came up heads 20 times. Twenty times!

10coins

This was not a trick coin or anything like that. It was a regular “fair” coin that normally would come up heads about 50% of the time, and tails the rest of the time. I must be a great coin-flipper. The chances of this just happening randomly are less than one in a million (1:1,048,576 to be exact).

Imagine if the outcome of a coin flip represented something very valuable. Something that you really want. How much would you pay me to flip coins for you? Before you answer, one more small detail: I didn’t mention any of the times that the coin came up tails.

The above probability is actually 1:1 (inevitable) because I just kept flipping until I got enough of what I was after. That was a cheap trick, I admit, though one that some people might pull on purpose. However it also provides a tangible example of what goes wrong when we fall prey to confirmation bias. Even people with the best intentions tend to pay more attention to expected or desired outcomes, while downplaying or ignoring other evidence.

This is something to keep in mind whenever you are evaluating a product or service. You will often see “success stories” representing satisfied customers, or examples of specific times that a product or service provided very desirable results. It is always important to ask “Where are the tails?” (if we are sticking with the silly example above). How many dissatisfied customers? How often are the results this beneficial? How often do undesired outcomes occur?

Variation is to be expected. The more often and more consistently someone claims to provide a specific outcome, the less likely it is to actually be true.

Leave a Reply

Your email address will not be published. Required fields are marked *