A Bayes Factor Meta-Analysis of Bem's ESP Claim

📌 Appears in:

Plain English Summary

When Daryl Bem claimed people can sense the future, it sparked a firestorm. This study responds with a fresh tool — a "meta-analytic Bayes factor" pooling evidence across experiments. The method has since become standard in psychology, a big deal on its own. Reanalyzing Bem's nine experiments by image type, evidence for precognition with erotic images was weak, with neutral images basically zero, but with emotional non-erotic images it hit 40 to 1 in favor. The paper also showed an earlier skeptical critique underestimated cumulative evidence. The authors land in an unusual spot: that critique was technically flawed, but they still reject ESP — 40-to-1 odds aren't enough given how implausible psychic powers are without any physical mechanism.

Actual Paper Abstract

In recent years, statisticians and psychologists have provided the critique that p-values do not capture the evidence afforded by data and are, consequently, ill suited for analysis in scientific endeavors. The issue is particular salient in the assessment of the recent evidence provided for ESP by Bem (2011) in the mainstream Journal of Personality and Social Psychology. Wagenmakers, Wetzels, Borsboom, and van der Maas (Journal of Personality and Social Psychology, 100, 426–432, 2011) have provided an alternative Bayes factor assessment of Bem's data, but their assessment was limited to examining each experiment in isolation. We show here that the variant of the Bayes factor employed by Wagenmakers et al. is inappropriate for making assessments across multiple experiments, and cannot be used to gain an accurate assessment of the total evidence in Bem's data. We develop a meta-analytic Bayes factor that describes how researchers should update their prior beliefs about the odds of hypotheses in light of data across several experiments. We find that the evidence that people can feel the future with neutral and erotic stimuli to be slight, with Bayes factors of 3.23 and 1.57, respectively. There is some evidence, however, for the hypothesis that people can feel the future with emotionally valenced nonerotic stimuli, with a Bayes factor of about 40. Although this value is certainly noteworthy, we believe it is orders of magnitude lower than what is required to overcome appropriate skepticism of ESP.

Research Notes

Methodologically important for introducing the meta-analytic JZS Bayes factor, now widely adopted in psychology. Occupies a distinctive middle position in the Bem debate: critiques Wagenmakers et al.'s approach as inadequate while still concluding against ESP. Central to Controversy #2.

Bayesian meta-analysis of the nine experiments in Bem's (2011) 'Feeling the Future' paper, using a newly developed meta-analytic extension of the JZS default Bayes factor t-test. Excluding three retroactive mere-exposure experiments as uninterpretable, the remaining data were categorized by stimulus type. Evidence for ESP with erotic stimuli was slight (BF = 3.23), with neutral stimuli negligible (BF = 1.57), but with emotionally valenced nonerotic stimuli noteworthy (BF = 38.7). The analysis also demonstrated that simply multiplying individual Bayes factors across experiments — as Wagenmakers et al. (2011) implicitly did — systematically underestimates cumulative evidence. Despite the BF of ~40 for emotional stimuli, the authors argued this is insufficient to overcome appropriate prior skepticism about ESP given the absence of plausible physical mechanisms.