Theoretical Risks and Tabular Asterisks: Sir Karl, Sir Ronald, and the Slow Progress of Soft Psychology
📄 Original study ↗Plain English Summary
Published in 1978, this remarkably prescient paper essentially predicted the replication crisis that wouldn't fully erupt for another three decades. Meehl's target: the way 'soft' psychology (think personality, social, and clinical research) leans almost entirely on null hypothesis significance testing -- that familiar ritual of checking whether your result earns a little asterisk in a table. His devastating point is that the null hypothesis (the claim that there's zero effect) is almost always literally false, so rejecting it tells you very little about whether your actual theory is any good. It's like celebrating that you found a needle in a haystack when the haystack is made of needles. He proposed an alternative using his taxometric method, which nailed 94% accuracy with zero false negatives across 600 simulations -- genuinely impressive. This critique lands squarely on psi research debates too, where both believers and skeptics tally up p-values and trade 'significant' results like baseball cards, often missing the deeper question of what those numbers actually mean.
Actual Paper Abstract
Theories in "soft" areas of psychology lack the cumulative character of scientific knowledge. They tend neither to be refuted nor corroborated, but instead merely fade away as people lose interest. Even though intrinsic subject matter difficulties (20 listed) contribute to this, the excessive reliance on significance testing is partly responsible, being a poor way of doing science. Karl Popper's approach, with modifications, would be prophylactic. Since the null hypothesis is quasi-always false, tables summarizing research in terms of patterns of "significant differences" are little more than complex, causally uninterpretable outcomes of statistical power functions. Multiple paths to estimating numerical point values ("consistency tests") are better, even if approximate with rough tolerances; and lacking this, ranges, orderings, second-order differences, curve peaks and valleys, and function forms should be used. Such methods are usual in developed sciences that seldom report statistical significance. Consistency tests of a conjectural taxometric model yielded 94% success with zero false negatives.
Research Notes
A foundational critique of significance testing that anticipates the replication crisis by three decades. Directly relevant to the library's meta-debate controversy: the same nose-counting of p-values that Meehl decries is central to how both psi proponents and skeptics evaluate the evidence. His distinction between statistical and substantive hypotheses illuminates why ganzfeld batting averages and Bem replication tallies are contested.
Arguing that theories in soft psychology (clinical, social, personality) neither cumulate nor get clearly refuted but merely fade away, Meehl identifies 20 intrinsic difficulties of the subject matter and then turns to an extrinsic one: the near-universal reliance on null hypothesis significance testing. Drawing on Popper's falsificationism and Bayesian reasoning, he shows that since the null hypothesis is quasi-always false, refuting it depends on statistical power, not theoretical verisimilitude, making significance tests a feeble method of theory corroboration. As an alternative, he demonstrates 'consistency tests' within his MAXCOV-HITMAX taxometric framework, achieving 94% accuracy with zero false negatives across 600 Monte Carlo samples. He prescribes multiple non-redundant estimates of theoretical quantities over significance tests.
Links
Related Papers
Companion
- Why Science Is Not Necessarily Self-Correcting — Ioannidis, John P.A (2012)
- The "File Drawer Problem" and Tolerance for Null Results — Rosenthal, Robert (1979)
- Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability — Nosek, Brian A (2012)
- Small Telescopes: Detectability and the Evaluation of Replication Results — Simonsohn, Uri (2015)
- Options for Prospective Meta-Analysis and Introduction of Registration-Based Prospective Meta-Analysis — Watt, Caroline A (2017)
- Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi — Wagenmakers, Eric-Jan (2011)
- Scientists behaving badly — Martinson, Brian C (2005)
More in Methodology
Paranormal belief, conspiracy endorsement, and positive wellbeing: a network analysis
Planning Falsifiable Confirmatory Research
Addressing Researcher Fraud: Retrospective, Real-Time, and Preventive Strategies — Including Legal Points and Data Management That Prevents Fraud
Quantum Aspects of the Brain-Mind Relationship: A Hypothesis with Supporting Evidence
Paranormal beliefs and cognitive function: A systematic review and assessment of study quality across four decades of research
📋 Cite this paper
Meehl, Paul E (1978). Theoretical Risks and Tabular Asterisks: Sir Karl, Sir Ronald, and the Slow Progress of Soft Psychology. Journal of Consulting and Clinical Psychology. https://doi.org/10.1037/0022-006X.46.4.806
@article{meehl_1978_theoretical_risks,
title = {Theoretical Risks and Tabular Asterisks: Sir Karl, Sir Ronald, and the Slow Progress of Soft Psychology},
author = {Meehl, Paul E},
year = {1978},
journal = {Journal of Consulting and Clinical Psychology},
doi = {10.1037/0022-006X.46.4.806},
}