Summary
Overview
Planet Money reporters visit a 'Replication Game' in Montreal, where economist Abel Brodeur has created an innovative system to address the replication crisis in social science research. Through crowdsourced hackathon-style events, teams of researchers verify whether published papers' findings can be reproduced, creating an accountability mechanism that's changing how academics conduct research.
The Replication Crisis Discovery
Economist Abel Brodeur first encountered the replication crisis while working on his master's thesis about smoking bans in 2011. After manipulating his data to achieve statistical significance, he realized he was doing something fundamentally wrong that undermined the goal of science. This personal experience led him to investigate how widespread the problem was, eventually discovering that the academic incentive system was pushing researchers to torture data until it produced publishable, significant results.
- Abel collected data on smoking bans but initially found no effect, contradicting published research
- He manipulated his data for six months until finding a statistically significant result
- Statistical significance requires results that would occur by chance less than 5% of the time - a crucial threshold for publication
- Abel realized this data manipulation was undermining the goal of discovering true knowledge about human behavior
- He discovered other students were doing the same thing: 'oh yeah, that's how you publish'
" I was finding absolutely no effect. None. It was like nobody stopped smoking. I've played with the data for six months, and I find nothing. "
" This is so stupid. What am I doing? I'm writing a piece saying that smoking bans are decreasing smoking prevalence because I managed to find one that worked. I was like, this is dumb. I'm doing something wrong. "
Star Wars: The Empirics Strike Back
Abel and colleagues researched the research itself, scraping significance data from top academic journals to measure the extent of the problem. They discovered a suspicious hump in the distribution just above the 5% significance threshold, suggesting widespread data manipulation. Despite resistance from academic publishing, they eventually published their findings in 2016, and the paper became an unexpected hit in academic circles.
- Abel scraped data from top journals to analyze the distribution of statistical significance
- They found a noticeable hump just above the 5% significance threshold
- Academic publishers initially rejected their research with a 'resounding series of no'
- The paper was published in 2016 and became well-known, with people asking 'are you a Star Wars guy?'
" It took a long time before I realized actually the paper was like well-known. Before people started talking to me at conferences like, are you a Star Wars guy? "
📚 6 more sections below
Sign up to unlock the complete summary with all insights, key points, and quotes
Get this summary + all future Planet Money episodes in your inbox
100% Free • Unsubscribe Anytime
Sign up now and we'll send you the complete summary of this episode, plus get notified when new Planet Money episodes are released—delivered straight to your inbox within minutes.