Fearing the Future of Empirical Psychology: Bem's (2011) Evidence of Psi as a Case Study of Deficiencies in Modal Research Practice
β‘ Contested βπ Appears in:
Plain English Summary
When Daryl Bem published a blockbuster 2011 paper claiming experimental evidence for psychic abilities, LeBel and Peters used it as an X-ray of what's broken in mainstream psychology research. Their argument goes beyond statistics into philosophy of science: they identify three deep problems with standard research practices. First, psychology over-relies on loose "conceptual" replications (tweaking the experiment each time) instead of running the same study again to see if it holds up. Second, researchers rarely check whether their measurement tools actually work reliably β and Bem reported zero reliability data. Third, the standard statistical method (null hypothesis testing) is essentially rigged: test enough people and even a trivially tiny effect like a 51.7% hit rate becomes "significant." Together, these flaws create what they call "interpretation bias" β a cozy protective bubble that makes virtually any theory unfalsifiable. Their provocative conclusion? It's more reasonable to doubt psychology's methods than to accept that humans can see the future.
Actual Paper Abstract
In this methodological commentary, we use Bem's (2011) recent article reporting experimental evidence for psi as a case study for discussing important deο¬ciencies in modal research practice in empirical psychology. We focus on (a) overemphasis on conceptual rather than close replication, (b) insufο¬cient attention to verifying the soundness of measurement and experimental procedures, and (c) ο¬awed implementation of null hypothesis signiο¬cance testing. We argue that these deο¬ciencies contribute to weak method-relevant beliefs that, in conjunction with overly strong theory-relevant beliefs, lead to a systemic and pernicious bias in the interpretation of data that favors a researcher's theory. Ultimately, this interpretation bias increases the risk of drawing incorrect conclusions about human psychology. Our analysis points to concrete recommendations for improving research practice in empirical psychology. We recommend (a) a stronger emphasis on close replication, (b) routinely verifying the integrity of measurement instruments and experimental procedures, and (c) using stronger, more diagnostic forms of null hypothesis testing.
Research Notes
A landmark methodological critique grounded in philosophy of science (Duhem-Quine underdetermination, Quine-Ullian conservatism) rather than statistics alone. The 'interpretation bias' framework became influential in the broader replication reform movement. Directly relevant to Controversy #2 (Bem) and #10 (meta-debate). Compare with Wagenmakers et al. (2011) for the Bayesian-statistical version of the same argument.
Bem's (2011) nine-experiment report of evidence for psi is used as a diagnostic case study for three systemic deficiencies in modal research practice (MRP) in empirical psychology: (a) overemphasis on conceptual rather than close replication, which allows failed extensions to be filed away while successful ones count as replications; (b) failure to independently verify measurement instrument integrity, with no reliability estimates reported for any dependent variable; and (c) flawed NHST implementation that tests against a nil hypothesis virtually guaranteed to be false, making rejection contingent only on sample size (e.g., a 51.7% hit rate reaching p < .05 at N = 150). These deficiencies produce an 'interpretation bias' that buffers any theory from falsification. Conservatism in theory choice favors revising beliefs about MRP over revising beliefs about causality and time.
Links
Related Papers
Cites
- Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect β Bem, Daryl J (2011)
- A Practical Solution to the Pervasive Problems of p Values β Wagenmakers, Eric-Jan (2007)
- Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi β Wagenmakers, Eric-Jan (2011)
- Why Most Published Research Findings Are False β Ioannidis, John P.A (2005)
Companion
- Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi β Wagenmakers, Eric-Jan (2011)
- Back from the Future: Parapsychology and the Bem Affair β Alcock, James E (2011)
- Failing the Future: Three Unsuccessful Attempts to Replicate Bem's 'Retroactive Facilitation of Recall' Effect β Ritchie, Stuart J (2012)
- Correcting the Past: Failures to Replicate Psi β Galak, Jeff (2012)
- Results from a Confirmatory Replication Study of Bem (2011): Precognitive Detection of Erotic Stimuli? β Wagenmakers, Eric-Jan (2012)
- False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant β Simmons, Joseph P (2011)
- Too Good to Be True: Publication Bias in Two Prominent Studies from Experimental Psychology β Francis, Gregory (2012)
- We Should Have Seen This Coming β Schwarzkopf, D. Samuel (2014)
- A Bayes Factor Meta-Analysis of Bem's ESP Claim β Rouder, Jeffrey N (2011)
- Bem's 'Feeling the Future' (2011) Five Years Later: Its Impact on Scientific Literature β Silva, Bruno A (2017)
Also by these authors
More in Methodology
Paranormal belief, conspiracy endorsement, and positive wellbeing: a network analysis
Planning Falsifiable Confirmatory Research
Addressing Researcher Fraud: Retrospective, Real-Time, and Preventive Strategies β Including Legal Points and Data Management That Prevents Fraud
Quantum Aspects of the Brain-Mind Relationship: A Hypothesis with Supporting Evidence
Paranormal beliefs and cognitive function: A systematic review and assessment of study quality across four decades of research
π Cite this paper
LeBel, Etienne P, Peters, Kurt R (2011). Fearing the Future of Empirical Psychology: Bem's (2011) Evidence of Psi as a Case Study of Deficiencies in Modal Research Practice. Review of General Psychology. https://doi.org/10.1037/a0025172
@article{lebel_peters_2011_fearing_future,
title = {Fearing the Future of Empirical Psychology: Bem's (2011) Evidence of Psi as a Case Study of Deficiencies in Modal Research Practice},
author = {LeBel, Etienne P and Peters, Kurt R},
year = {2011},
journal = {Review of General Psychology},
doi = {10.1037/a0025172},
}