Is the Methodological Revolution in Psychology Over or Just Beginning?
📄 Original study ↗📌 Appears in:
Plain English Summary
When Bem published his "feeling the future" experiments in 2011, psychology had a wake-up call about research rigor. But Kennedy argues pre-registering studies was just the start. He identifies eight more weak spots both mainstream psychology and psi research (the study of psychic phenomena) still haven't fixed -- sloppy record-keeping, unvalidated software, no safeguards against cheating, and cherry-picked statistics among them. His fix? Borrow the tough playbook from FDA-regulated drug trials. His punchline is deliciously strategic: once psychologists see these flaws laid bare, they'll rush to adopt reforms rather than risk their work looking as shaky as the psi studies that sparked the crisis.
Actual Paper Abstract
Significant results from parapsychological experiments using standard psychological research methods motivated psychologists to recognize some widespread methodological deficiencies and the need for preregistered well-powered confirmatory research. Psychological researchers have not yet recognized several other common methodological weaknesses that can be expected to cause this cycle to be repeated. When confronted with the choice between psi versus overlooked methodological deficiencies, psychologists will recognize the need for methodological improvements. These overlooked methodological factors include: (a) deficient study registration practices, (b) bias from dropouts and incomplete data, (c) the need for software validation, (d) measures to prevent experimenter fraud, (e) appropriate statistical methods for confirmatory research, (f) failure to consider inferential errors with Bayesian analyses, (g) the weaknesses of retrospective meta-analysis and strengths of prospective meta-analysis, and (h) problems from statistical dependence for the outcome variables in statistical analyses. Psychological and parapsychological researchers can easily avoid this inefficient process of methodological evolution driven by controversies about parapsychological findings. Research practices that address these methodological deficiencies are available and will eventually be recognized as needed for psychological and parapsychological research. Recommended practices for addressing these methodological weaknesses are described.
Research Notes
Most systematic single-paper inventory of methodological weaknesses in psi research, applying FDA-level clinical trial standards. Directly extends Kennedy (2013) on retrospective meta-analysis and anticipates Kennedy (2017, 2024) on fraud standards. Central to the meta-debate (Controversy #10).
Prompted by Bem's (2011) 'feeling the future' experiments, psychology recognized the need for pre-registered confirmatory research. Eight additional methodological deficiencies remain unaddressed in both fields: deficient study registration, bias from dropouts and incomplete data, lack of software validation, absence of fraud prevention measures, inappropriate statistical methods for confirmatory studies, overlooked Bayesian inferential errors, weaknesses of retrospective meta-analysis versus prospective meta-analysis, and statistical dependence problems in outcome variables. Drawing on regulated medical research standards (FDA clinical trial guidance), specific practices are recommended for each deficiency. When confronted with the choice between psi and methodological flaws, psychologists will inevitably choose reform — making proactive adoption of these standards the efficient path forward.
Related Papers
Cites
- Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect — Bem, Daryl J (2011)
- Can Parapsychology Move Beyond the Controversies of Retrospective Meta-Analyses? — Kennedy, J.E (2013)
- Estimating the Reproducibility of Psychological Science — Open Science Collaboration (2015)
- Lessons from the First Two Years of Operating a Study Registry — Watt, Caroline (2015)
Same Research Program
- Experimenter Fraud: What Are Appropriate Methodological Standards? — Kennedy, J.E (2017)
- Conclusions about Paranormal Phenomena — Kennedy, J.E (2013)
- Planning Falsifiable Confirmatory Research — Kennedy, James E (2024)
- Addressing Researcher Fraud: Retrospective, Real-Time, and Preventive Strategies — Including Legal Points and Data Management That Prevents Fraud — Kennedy, James E (2024)
Companion
Also by these authors
More in Methodology
Paranormal belief, conspiracy endorsement, and positive wellbeing: a network analysis
Quantum Aspects of the Brain-Mind Relationship: A Hypothesis with Supporting Evidence
Paranormal beliefs and cognitive function: A systematic review and assessment of study quality across four decades of research
Experimental evidence of non-classical brain functions
Self-Ascribed Paranormal Ability: Reflexive Thematic Analysis
📋 Cite this paper
Kennedy, J.E (2016). Is the Methodological Revolution in Psychology Over or Just Beginning?. Journal of Parapsychology.
@article{kennedy_2016_psi_research_ready,
title = {Is the Methodological Revolution in Psychology Over or Just Beginning?},
author = {Kennedy, J.E},
year = {2016},
journal = {Journal of Parapsychology},
}