A 20-Year Review of Outcome Reporting Bias in Moderated Multiple Regression

Ernest O’Boyle, George C. Banks, Kameron Carter, Sheryl Walter, Zhenyu Yuan

Research output: Contribution to journalArticle

6 Scopus citations

Abstract

Moderated multiple regression (MMR) remains the most popular method of testing interactions in management and applied psychology. Recent discussions of MMR have centered on their small effect sizes and typically being statistically underpowered (e.g., Murphy & Russell, Organizational Research Methods, 2016). Although many MMR tests are likely plagued by type II errors, they may also be particularly prone to outcome reporting bias (ORB) resulting in elevated false positives (type I errors). We tested the state of MMR through a 20-year review of six leading journals. Based on 1218 MMR tests nested within 343 studies, we found that despite low statistical power, most MMR tests (54%) were reported as statistically significant. Further, although sample size has remained relatively unchanged (r = −.002), statistically significant MMR tests have risen from 41% (1995–1999) to 49% (2000–2004), to 60% (2005–2009), and to 69% (2010–2014). This could indicate greater methodological and theoretical precision but leaves open the possibility of ORB. In our review, we found evidence that both increased rigor and theoretical precision play an important role in MMR effect size magnitudes, but also found evidence for ORB. Specifically, (a) smaller sample sizes are associated with larger effect sizes, (b) there is a substantial frequency spike in p values just below the.05 threshold, and (c) recalculated p values less than.05 always converged with authors’ conclusions of statistical significance but recalculated p values between.05 and.10 only converged with authors’ conclusions about half (54%) of the time. The findings of this research provide important implications for future application of MMR.

Original languageEnglish (US)
Pages (from-to)19-37
Number of pages19
JournalJournal of Business and Psychology
Volume34
Issue number1
DOIs
StatePublished - Feb 15 2019

All Science Journal Classification (ASJC) codes

  • Business and International Management
  • Business, Management and Accounting(all)
  • Applied Psychology
  • Psychology(all)

Fingerprint Dive into the research topics of 'A 20-Year Review of Outcome Reporting Bias in Moderated Multiple Regression'. Together they form a unique fingerprint.

  • Cite this