Bayesian and Classical Hypothesis Testing: Practical Differences for a Controversial Area of Research
📄 Original study ↗📌 Appears in:
Plain English Summary
Kennedy wades into the stats wars between classical (frequentist) and Bayesian hypothesis testing. Classical methods ask 'how surprising is this data if nothing is going on?' while Bayesian methods fold in prior beliefs about how likely an effect is. The practical stakes are huge: 5,100 hits out of 10,000 tries passes the classical test but favors the null under standard Bayesian analysis by nearly 11 to 1 — a jaw-dropping reversal depending on which philosophy you pick. Kennedy argues neither approach is wrong when done carefully, but both parapsychology and mainstream psychology rarely do truly confirmatory research with pre-specified rules. He recommends the FDA's clinical trial framework as a model.
Actual Paper Abstract
The use of Bayesian analysis and debates involving Bayesian analysis are increasing for controversial areas of research such as parapsychology. This paper conceptually describes the philosophical and modeling differences between Bayesian and classical analyses, and the practical implications of these differences. Widely accepted statistical conventions have not yet been established for Bayesian analysis in scientific research. The recommendations from the FDA guidance on using Bayesian methods are appropriate for confirmatory experiments. This guidance recommends that the study design and protocol include (a) specification of the prior probabilities and models that will be used, (b) specification of the criteria that will be considered acceptable evidence, (c) operating characteristics for the probability of Type I error and power of the analysis, and (d) an estimate of the relative roles of prior probability versus the data from the current experiment in producing the final results. Both classical and Bayesian methods are valid when properly applied with confirmatory methodology that includes prespecification of statistical methods, and prospective evaluations of inferential errors and power. Evaluations of inferential errors and power measure the validity of a planned hypothesis test, including Bayesian analysis. Unfortunately, the use of confirmatory methodology has been rare in psychology and parapsychology.
Research Notes
A balanced and accessible methodological commentary comparing Bayesian and classical hypothesis testing, tailored to psi research controversies. Directly relevant to the Wagenmakers-Bem debate and Bayes factor usage in parapsychology. The catalog ID reflects the PDF filename (a mislabel from ingestion), not the actual paper title.
Conceptual analysis comparing Bayesian and classical (frequentist) hypothesis testing for controversial research areas such as parapsychology. Describes the philosophical differences (objective vs subjective probability), mathematical models, and practical implications of each approach. Demonstrates that a uniform Bayesian prior can bias against small effects: 5100 hits in 10000 trials yields classical p=.046 but Bayesian BF=10.8 for the null. Recommends the FDA guidance on Bayesian clinical trials as a framework for confirmatory psi experiments: specify priors, acceptance criteria, operating characteristics (Type I error and power), and relative roles of priors vs data prospectively. Argues that both approaches are valid when properly applied, but psychology and parapsychology lack confirmatory methodology, undermining both statistical traditions.
Related Papers
Cites
- The Capricious, Actively Evasive, Unsustainable Nature of Psi: A Summary and Hypotheses — Kennedy, J.E (2003)
- Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect — Bem, Daryl J (2011)
- Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi — Wagenmakers, Eric-Jan (2011)
- Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence? — Pashler, Harold (2012)
- Can Parapsychology Move Beyond the Controversies of Retrospective Meta-Analyses? — Kennedy, J.E (2013)
Companion
- A Practical Solution to the Pervasive Problems of p Values — Wagenmakers, Eric-Jan (2007)
- A Bayes Factor Meta-Analysis of Recent Extrasensory Perception Experiments: Comment on Storm, Tressoldi, and Di Risio (2010) — Rouder, Jeffrey N (2013)
- Replication Unreliability in Psychology: Elusive Phenomena or "Elusive" Statistical Power? — Tressoldi, Patrizio E (2012)
- Lessons from the First Two Years of Operating a Study Registry — Watt, Caroline (2015)
- False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant — Simmons, Joseph P (2011)
- Too Good to Be True: Publication Bias in Two Prominent Studies from Experimental Psychology — Francis, Gregory (2012)
- The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No "Fishing Expedition" or "P-Hacking" and the Research Hypothesis Was Posited Ahead of Time — Gelman, Andrew (2013)
- Why Most Published Research Findings Are False — Ioannidis, John P.A (2005)
Also by these authors
More in Methodology
Paranormal belief, conspiracy endorsement, and positive wellbeing: a network analysis
Addressing Researcher Fraud: Retrospective, Real-Time, and Preventive Strategies — Including Legal Points and Data Management That Prevents Fraud
Quantum Aspects of the Brain-Mind Relationship: A Hypothesis with Supporting Evidence
Paranormal beliefs and cognitive function: A systematic review and assessment of study quality across four decades of research
Experimental evidence of non-classical brain functions
📋 Cite this paper
Kennedy, J.E (2014). Bayesian and Classical Hypothesis Testing: Practical Differences for a Controversial Area of Research. Journal of Parapsychology.
@article{kennedy_2014_nature_psi,
title = {Bayesian and Classical Hypothesis Testing: Practical Differences for a Controversial Area of Research},
author = {Kennedy, J.E},
year = {2014},
journal = {Journal of Parapsychology},
}