Surprising Studies are Usually Wrong

The problem with things like Ted talks is that they almost always focus on some surprising study. And the problem with surprising studies is that they’re usually wrong.

Consider, for example, the “power pose” study, in which the authors claimed that striking a powerful pose (e.g., arms on your hips and legs spread out) can trick your brain into feeling more powerful and help you perform better in things like negotiations and interviews. The Ted talk went viral. But a few years later, when replication was attempted, the study was debunked. Even some of the original authors have stated they no longer believe in the effect.

This example isn’t alone. The fact that studies with surprising outcomes are usually wrong has a good mathematical reason. Suppose a study is focusing on some surprising phenomenon , which apriori had a probability of of being true.

Let’s say that whenever is true, the study finds it with probability roughly . On the other hand, whenever is false, the study still has a probability of incorrectly finding to be true. (In particular, that’s because a publishable -value of or less will happen by pure chance with probability .) Thus the probability that the study finds to be true (i.e., rejecting the null hypothesis) is

On the other hand, the probability that the study finds to be true and actually is true is . Thus, if we condition on the study finding to be true, then the probability that actually is true is

So even after seeing that the results of the study support , it’s still more likely than not that is false.

But it gets worse. Because the math above ignores human error. In particular it ignores the fact that humans are natural -hackers. Some recent research selected 21 random studies from high profile journals (Nature and Science) and tested whether they were replicable. They didn’t try to select particularly surprising studies (though the fact that the studies got Nature and Science may have implicitly done this for them). It was just a random sampling of 21 studies.

The result? Only 13 passed retesting. More importantly, it turns out that non-experts can guess which ones were replicable based on a one-sentence summary of the study. You can try it for yourself at this link. The reason that non-experts could guess so well was that it’s actually pretty easy. If a study makes you think, “Wow, that’s surprising,” or “Man, psychology is pretty amazing,” then it’s probably wrong.