Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi
β‘ Contested βπ Appears in:
Plain English Summary
When a well-known psychologist published data apparently showing people can see the future, Wagenmakers and colleagues said: not so fast. They re-crunched all nine of Daryl Bem's precognition experiments using Bayesian statistics -- a method that directly measures how much evidence supports one idea over another. The verdict was brutal: only one out of ten tests showed even modest support for psychic powers, while three actually favored the boring explanation that nothing paranormal happened. The rest were a statistical shrug. The team pinpointed where Bem went wrong: mixing up exploratory fishing expeditions with rigorous hypothesis testing, and leaning on p-values that dramatically oversell the evidence. They capped it off with a practical reform blueprint, including pre-registering studies and inviting skeptics to collaborate on experiments. This paper became a rallying cry in psychology's broader reckoning with its own statistical habits.
Actual Paper Abstract
Does psi exist? In a recent article, Dr. Bem conducted nine studies with over a thousand participants in an attempt to demonstrate that future events retroactively aο¬ect people's responses. Here we discuss several limitations of Bem's experiments on psi; in particular, we show that the data analysis was partly exploratory, and that one-sided p-values may overstate the statistical evidence against the null hypothesis. We reanalyze Bem's data using a default Bayesian t-test and show that the evidence for psi is weak to nonexistent. We argue that in order to convince a skeptical audience of a controversial claim, one needs to conduct strictly conο¬rmatory studies and analyze the results with statistical tests that are conservative rather than liberal. We conclude that Bem's p-values do not indicate evidence in favor of precognition; instead, they indicate that experimental psychologists need to change the way they conduct their experiments and analyze their data.
Research Notes
Landmark skeptical critique that became central to the replication crisis narrative in psychology. Demonstrates how standard frequentist methods can produce 'significant' results for implausible claims. Directly triggered Bem, Utts & Johnson's rebuttal and the Wagenmakers et al. (2012) confirmatory research agenda.
Reanalysis of Bem's (2011) nine precognition experiments using a default Bayesian t-test reveals that the statistical evidence for psi is weak to nonexistent. Of 10 critical tests, only one yields 'substantial' Bayesian evidence for psi (BF01 = 0.17); three yield 'substantial' evidence for the null hypothesis (BF01 = 3.14 to 7.61); the remaining six produce only 'anecdotal' evidence in either direction. The paper identifies three flaws in Bem's approach: conflation of exploratory and confirmatory analyses, the fallacy of the transposed conditional, and reliance on p-values that overstate evidence against the null. Proposes six guidelines for confirmatory research including pre-registration, Bayesian testing, and adversarial collaboration.
Links
Related Papers
Companion
- A Bayes Factor Meta-Analysis of Bem's ESP Claim β Rouder, Jeffrey N (2011)
- Back from the Future: Parapsychology and the Bem Affair β Alcock, James E (2011)
- Fearing the Future of Empirical Psychology: Bem's (2011) Evidence of Psi as a Case Study of Deficiencies in Modal Research Practice β LeBel, Etienne P (2011)
- Correcting the Past: Failures to Replicate Psi β Galak, Jeff (2012)
- False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant β Simmons, Joseph P (2011)
- Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence? β Pashler, Harold (2012)
- Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling β John, Leslie K (2012)
- The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No "Fishing Expedition" or "P-Hacking" and the Research Hypothesis Was Posited Ahead of Time β Gelman, Andrew (2013)
- Failing the Future: Three Unsuccessful Attempts to Replicate Bem's 'Retroactive Facilitation of Recall' Effect β Ritchie, Stuart J (2012)
- Results from a Confirmatory Replication Study of Bem (2011): Precognitive Detection of Erotic Stimuli? β Wagenmakers, Eric-Jan (2012)
- Mindless Statistics β Gigerenzer, Gerd (2004)
- Theoretical Risks and Tabular Asterisks: Sir Karl, Sir Ronald, and the Slow Progress of Soft Psychology β Meehl, Paul E (1978)
Cites
- Meta-Analysis of Free-Response Studies, 1992β2008: Assessing the Noise Reduction Model in Parapsychology β Storm, Lance (2010)
- Replication and Meta-Analysis in Parapsychology β Utts, Jessica (1991)
- Why Is Psi So Elusive? A Review and Proposed Model β Kennedy, James E (2001)
- Why Most Published Research Findings Are False β Ioannidis, John P.A (2005)
Cited By
- Cross-Examining the Case for Precognition: Comment on Mossbridge and Radin (2018) β Houran, James (2018)
- Bem's 'Feeling the Future' (2011) Five Years Later: Its Impact on Scientific Literature β Silva, Bruno A (2017)
- Why Most Research Findings About Psi Are False: The Replicability Crisis, the Psi Paradox and the Myth of Sisyphus β Rabeyron, Thomas (2020)
- Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses β Lakens, DaniΓ«l (2017)
- The Experimental Evidence for Parapsychological Phenomena: A Review β CardeΓ±a, Etzel (2018)
- The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No "Fishing Expedition" or "P-Hacking" and the Research Hypothesis Was Posited Ahead of Time β Gelman, Andrew (2013)
- Bayesian and Classical Hypothesis Testing: Practical Differences for a Controversial Area of Research β Kennedy, J.E (2014)
- Replication Unreliability in Psychology: Elusive Phenomena or "Elusive" Statistical Power? β Tressoldi, Patrizio E (2012)
- Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability β Nosek, Brian A (2012)
Also by these authors
More in Skeptical
Cognitive Styles and Psi: Psi Researchers Are More Similar to Skeptics Than to Lay Believers
Searching for the Impossible: Parapsychology's Elusive Quest
False-Positive Effect in the Radin Double-Slit Experiment on Observer Consciousness as Determined with the Advanced Meta-Experimental Protocol
N,N-Dimethyltryptamine and the Pineal Gland: Separating Fact from Myth
Paranormal psychic believers and skeptics: a large-scale test of the cognitive differences hypothesis
π Cite this paper
Wagenmakers, Eric-Jan, Wetzels, Ruud, Borsboom, Denny, van der Maas, Han (2011). Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi. Journal of Personality and Social Psychology. https://doi.org/10.1037/a0022790
@article{wagenmakers_2011_why_psychologists,
title = {Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi},
author = {Wagenmakers, Eric-Jan and Wetzels, Ruud and Borsboom, Denny and van der Maas, Han},
year = {2011},
journal = {Journal of Personality and Social Psychology},
doi = {10.1037/a0022790},
}