Must Psychologists Change the Way They Analyze Their Data?
📄 Original study📌 Appears in:
Plain English Summary
When Wagenmakers and colleagues claimed a Bayesian reanalysis of Bem's precognition experiments showed no evidence for psi, Bem fired back. The crux? It all comes down to your "prior" — the assumptions baked into the math before looking at data. Wagenmakers used a broad prior placing 57% probability on huge effect sizes, which Bem's team called absurd. Swap in a realistic prior based on known effect sizes, and the result flips dramatically: from zero evidence to a Bayes factor of 13,669 — enormous support for psi. The unsettling takeaway: prior assumptions can completely reverse your conclusions.
Actual Paper Abstract
Wagenmakers, Wetzels, Borsboom, and van der Maas (2011) argued that psychologists should replace the familiar "frequentist" statistical analyses of their data with Bayesian analyses. To illustrate their argument, they reanalyzed a set of psi experiments published recently in this journal by Bem (2011), maintaining that, contrary to his conclusion, his data do not yield evidence in favor of the psi hypothesis. We argue that they have incorrectly selected an unrealistic prior distribution for their analysis and that a Bayesian analysis using a more reasonable distribution yields strong evidence in favor of the psi hypothesis. More generally, we argue that there are advantages to Bayesian analyses that merit their increased use in the future. However, as Wagenmakers et al.'s analysis inadvertently revealed, they contain hidden traps that must be better understood before being more widely substituted for the familiar frequentist analyses currently employed by most research psychologists.
Research Notes
Key document in the Bem FTF statistical debate. Shows how prior choice flips Bayesian conclusions from null support (Wagenmakers' Cauchy: BF = 0.63) to extreme psi support (knowledge-based: BF = 13,669). Speaks directly to Controversies #2 and #10.
A reply to Wagenmakers et al. (2011), who argued that a Bayesian reanalysis of Bem's (2011) nine precognition experiments yields no evidence for psi. Bem, Utts, and Johnson contend that Wagenmakers et al.'s diffuse Cauchy prior is unrealistic, placing 57% probability on effect sizes >= 0.8 and triggering the Lindley-Jeffreys paradox. Using a knowledge-based normal prior (90th percentile of |d| at 0.5, informed by known psychological and psi effect sizes), the combined Bayes factor across all nine experiments is 13,669 with posterior P(H0) = 7.3 x 10^-5 — extreme evidence for psi. The authors argue Bayesian methods are valuable but contain hidden traps when priors are poorly specified.
Links
Related Papers
Cites
- Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect — Bem, Daryl J (2011)
- A Bayes Factor Meta-Analysis of Bem's ESP Claim — Rouder, Jeffrey N (2011)
- Predictive Physiological Anticipation Preceding Seemingly Unpredictable Stimuli: A Meta-Analysis — Mossbridge, Julia (2012)
Cited By
- Results from a Confirmatory Replication Study of Bem (2011): Precognitive Detection of Erotic Stimuli? — Wagenmakers, Eric-Jan (2012)
- The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No "Fishing Expedition" or "P-Hacking" and the Research Hypothesis Was Posited Ahead of Time — Gelman, Andrew (2013)
Also by these authors
More in Methodology
Paranormal belief, conspiracy endorsement, and positive wellbeing: a network analysis
Planning Falsifiable Confirmatory Research
Addressing Researcher Fraud: Retrospective, Real-Time, and Preventive Strategies — Including Legal Points and Data Management That Prevents Fraud
Quantum Aspects of the Brain-Mind Relationship: A Hypothesis with Supporting Evidence
Paranormal beliefs and cognitive function: A systematic review and assessment of study quality across four decades of research
📋 Cite this paper
Bem, Daryl J, Utts, Jessica, Johnson, Wesley O (2011). Must Psychologists Change the Way They Analyze Their Data?. Journal of Personality and Social Psychology. https://doi.org/10.1037/a0024777
@article{bem_utts_johnson_2011_must_psychologists,
title = {Must Psychologists Change the Way They Analyze Their Data?},
author = {Bem, Daryl J and Utts, Jessica and Johnson, Wesley O},
year = {2011},
journal = {Journal of Personality and Social Psychology},
doi = {10.1037/a0024777},
}