A 20-Year Review of Outcome Reporting Bias in Moderated Multiple Regression

Ernest O’Boyle, George C. Banks, Kameron Carter, Sheryl Walter, Zhenyu Yuan

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

Moderated multiple regression (MMR) remains the most popular method of testing interactions in management and applied psychology. Recent discussions of MMR have centered on their small effect sizes and typically being statistically underpowered (e.g., Murphy & Russell, Organizational Research Methods, 2016). Although many MMR tests are likely plagued by type II errors, they may also be particularly prone to outcome reporting bias (ORB) resulting in elevated false positives (type I errors). We tested the state of MMR through a 20-year review of six leading journals. Based on 1218 MMR tests nested within 343 studies, we found that despite low statistical power, most MMR tests (54%) were reported as statistically significant. Further, although sample size has remained relatively unchanged (r = −.002), statistically significant MMR tests have risen from 41% (1995–1999) to 49% (2000–2004), to 60% (2005–2009), and to 69% (2010–2014). This could indicate greater methodological and theoretical precision but leaves open the possibility of ORB. In our review, we found evidence that both increased rigor and theoretical precision play an important role in MMR effect size magnitudes, but also found evidence for ORB. Specifically, (a) smaller sample sizes are associated with larger effect sizes, (b) there is a substantial frequency spike in p values just below the.05 threshold, and (c) recalculated p values less than.05 always converged with authors’ conclusions of statistical significance but recalculated p values between.05 and.10 only converged with authors’ conclusions about half (54%) of the time. The findings of this research provide important implications for future application of MMR.

Original languageEnglish (US)
Pages (from-to)19-37
Number of pages19
JournalJournal of Business and Psychology
Volume34
Issue number1
DOIs
StatePublished - Feb 15 2019

Fingerprint

Sample Size
Applied Psychology
Research
Reporting bias
Multiple regression
P value
Effect size

All Science Journal Classification (ASJC) codes

  • Business and International Management
  • Business, Management and Accounting(all)
  • Applied Psychology
  • Psychology(all)

Cite this

O’Boyle, Ernest ; Banks, George C. ; Carter, Kameron ; Walter, Sheryl ; Yuan, Zhenyu. / A 20-Year Review of Outcome Reporting Bias in Moderated Multiple Regression. In: Journal of Business and Psychology. 2019 ; Vol. 34, No. 1. pp. 19-37.
@article{c20c7ebe665148ab995e3db320347fe1,
title = "A 20-Year Review of Outcome Reporting Bias in Moderated Multiple Regression",
abstract = "Moderated multiple regression (MMR) remains the most popular method of testing interactions in management and applied psychology. Recent discussions of MMR have centered on their small effect sizes and typically being statistically underpowered (e.g., Murphy & Russell, Organizational Research Methods, 2016). Although many MMR tests are likely plagued by type II errors, they may also be particularly prone to outcome reporting bias (ORB) resulting in elevated false positives (type I errors). We tested the state of MMR through a 20-year review of six leading journals. Based on 1218 MMR tests nested within 343 studies, we found that despite low statistical power, most MMR tests (54{\%}) were reported as statistically significant. Further, although sample size has remained relatively unchanged (r = −.002), statistically significant MMR tests have risen from 41{\%} (1995–1999) to 49{\%} (2000–2004), to 60{\%} (2005–2009), and to 69{\%} (2010–2014). This could indicate greater methodological and theoretical precision but leaves open the possibility of ORB. In our review, we found evidence that both increased rigor and theoretical precision play an important role in MMR effect size magnitudes, but also found evidence for ORB. Specifically, (a) smaller sample sizes are associated with larger effect sizes, (b) there is a substantial frequency spike in p values just below the.05 threshold, and (c) recalculated p values less than.05 always converged with authors’ conclusions of statistical significance but recalculated p values between.05 and.10 only converged with authors’ conclusions about half (54{\%}) of the time. The findings of this research provide important implications for future application of MMR.",
author = "Ernest O’Boyle and Banks, {George C.} and Kameron Carter and Sheryl Walter and Zhenyu Yuan",
year = "2019",
month = "2",
day = "15",
doi = "10.1007/s10869-018-9539-8",
language = "English (US)",
volume = "34",
pages = "19--37",
journal = "Journal of Business and Psychology",
issn = "0889-3268",
publisher = "Kluwer Academic/Human Sciences Press Inc.",
number = "1",

}

A 20-Year Review of Outcome Reporting Bias in Moderated Multiple Regression. / O’Boyle, Ernest; Banks, George C.; Carter, Kameron; Walter, Sheryl; Yuan, Zhenyu.

In: Journal of Business and Psychology, Vol. 34, No. 1, 15.02.2019, p. 19-37.

Research output: Contribution to journalArticle

TY - JOUR

T1 - A 20-Year Review of Outcome Reporting Bias in Moderated Multiple Regression

AU - O’Boyle, Ernest

AU - Banks, George C.

AU - Carter, Kameron

AU - Walter, Sheryl

AU - Yuan, Zhenyu

PY - 2019/2/15

Y1 - 2019/2/15

N2 - Moderated multiple regression (MMR) remains the most popular method of testing interactions in management and applied psychology. Recent discussions of MMR have centered on their small effect sizes and typically being statistically underpowered (e.g., Murphy & Russell, Organizational Research Methods, 2016). Although many MMR tests are likely plagued by type II errors, they may also be particularly prone to outcome reporting bias (ORB) resulting in elevated false positives (type I errors). We tested the state of MMR through a 20-year review of six leading journals. Based on 1218 MMR tests nested within 343 studies, we found that despite low statistical power, most MMR tests (54%) were reported as statistically significant. Further, although sample size has remained relatively unchanged (r = −.002), statistically significant MMR tests have risen from 41% (1995–1999) to 49% (2000–2004), to 60% (2005–2009), and to 69% (2010–2014). This could indicate greater methodological and theoretical precision but leaves open the possibility of ORB. In our review, we found evidence that both increased rigor and theoretical precision play an important role in MMR effect size magnitudes, but also found evidence for ORB. Specifically, (a) smaller sample sizes are associated with larger effect sizes, (b) there is a substantial frequency spike in p values just below the.05 threshold, and (c) recalculated p values less than.05 always converged with authors’ conclusions of statistical significance but recalculated p values between.05 and.10 only converged with authors’ conclusions about half (54%) of the time. The findings of this research provide important implications for future application of MMR.

AB - Moderated multiple regression (MMR) remains the most popular method of testing interactions in management and applied psychology. Recent discussions of MMR have centered on their small effect sizes and typically being statistically underpowered (e.g., Murphy & Russell, Organizational Research Methods, 2016). Although many MMR tests are likely plagued by type II errors, they may also be particularly prone to outcome reporting bias (ORB) resulting in elevated false positives (type I errors). We tested the state of MMR through a 20-year review of six leading journals. Based on 1218 MMR tests nested within 343 studies, we found that despite low statistical power, most MMR tests (54%) were reported as statistically significant. Further, although sample size has remained relatively unchanged (r = −.002), statistically significant MMR tests have risen from 41% (1995–1999) to 49% (2000–2004), to 60% (2005–2009), and to 69% (2010–2014). This could indicate greater methodological and theoretical precision but leaves open the possibility of ORB. In our review, we found evidence that both increased rigor and theoretical precision play an important role in MMR effect size magnitudes, but also found evidence for ORB. Specifically, (a) smaller sample sizes are associated with larger effect sizes, (b) there is a substantial frequency spike in p values just below the.05 threshold, and (c) recalculated p values less than.05 always converged with authors’ conclusions of statistical significance but recalculated p values between.05 and.10 only converged with authors’ conclusions about half (54%) of the time. The findings of this research provide important implications for future application of MMR.

UR - http://www.scopus.com/inward/record.url?scp=85045755870&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85045755870&partnerID=8YFLogxK

U2 - 10.1007/s10869-018-9539-8

DO - 10.1007/s10869-018-9539-8

M3 - Article

AN - SCOPUS:85045755870

VL - 34

SP - 19

EP - 37

JO - Journal of Business and Psychology

JF - Journal of Business and Psychology

SN - 0889-3268

IS - 1

ER -