Selecting Null Distributions When Calculating rwg: A Tutorial and Review

Rustin D. Meyer, Troy V. Mumford, Carla J. Burrus, Michael A. Campion, Lawrence R. James

Research output: Contribution to journalArticle

18 Citations (Scopus)

Abstract

rwg is a common metric used to quantify interrater agreement in the organizational sciences. Finn developed rwg but based it on the assumption that raters’ deviations from their true perceptions are influenced by random chance only. James, Demaree, and Wolf extended Finn’s work by describing procedures to account for the additional influence of response biases. We demonstrate that organizational scientists have relied largely on Finn’s procedures, at least in part because of a lack of specific guidance regarding the conditions under which various response biases might be present. In an effort to address this gap in the literature, we introduce the concept of target-irrelevant, nonrandom forces (those aspects of the research context that are likely to lead to response biases), then describe how the familiar “5Ws and an H” framework (i.e., who, what, when, where, why, and how) can be used to identify these biases a priori. It is our hope that this system will permit those who calculate rwg to account for the effects of response biases in a manner that is simultaneously rigorous, consistent, and transparent.

Original languageEnglish (US)
Pages (from-to)324-345
Number of pages22
JournalOrganizational Research Methods
Volume17
Issue number3
DOIs
StatePublished - Jul 12 2014

Fingerprint

Tutorial
Response bias
Deviation
Interrater agreement
Guidance

All Science Journal Classification (ASJC) codes

  • Decision Sciences(all)
  • Strategy and Management
  • Management of Technology and Innovation

Cite this

Meyer, Rustin D. ; Mumford, Troy V. ; Burrus, Carla J. ; Campion, Michael A. ; James, Lawrence R. / Selecting Null Distributions When Calculating rwg : A Tutorial and Review. In: Organizational Research Methods. 2014 ; Vol. 17, No. 3. pp. 324-345.
@article{e0a0fdf7b5454d8a9bb88a1ec4116711,
title = "Selecting Null Distributions When Calculating rwg: A Tutorial and Review",
abstract = "rwg is a common metric used to quantify interrater agreement in the organizational sciences. Finn developed rwg but based it on the assumption that raters’ deviations from their true perceptions are influenced by random chance only. James, Demaree, and Wolf extended Finn’s work by describing procedures to account for the additional influence of response biases. We demonstrate that organizational scientists have relied largely on Finn’s procedures, at least in part because of a lack of specific guidance regarding the conditions under which various response biases might be present. In an effort to address this gap in the literature, we introduce the concept of target-irrelevant, nonrandom forces (those aspects of the research context that are likely to lead to response biases), then describe how the familiar “5Ws and an H” framework (i.e., who, what, when, where, why, and how) can be used to identify these biases a priori. It is our hope that this system will permit those who calculate rwg to account for the effects of response biases in a manner that is simultaneously rigorous, consistent, and transparent.",
author = "Meyer, {Rustin D.} and Mumford, {Troy V.} and Burrus, {Carla J.} and Campion, {Michael A.} and James, {Lawrence R.}",
year = "2014",
month = "7",
day = "12",
doi = "10.1177/1094428114526927",
language = "English (US)",
volume = "17",
pages = "324--345",
journal = "Organizational Research Methods",
issn = "1094-4281",
publisher = "SAGE Publications Inc.",
number = "3",

}

Selecting Null Distributions When Calculating rwg : A Tutorial and Review. / Meyer, Rustin D.; Mumford, Troy V.; Burrus, Carla J.; Campion, Michael A.; James, Lawrence R.

In: Organizational Research Methods, Vol. 17, No. 3, 12.07.2014, p. 324-345.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Selecting Null Distributions When Calculating rwg

T2 - A Tutorial and Review

AU - Meyer, Rustin D.

AU - Mumford, Troy V.

AU - Burrus, Carla J.

AU - Campion, Michael A.

AU - James, Lawrence R.

PY - 2014/7/12

Y1 - 2014/7/12

N2 - rwg is a common metric used to quantify interrater agreement in the organizational sciences. Finn developed rwg but based it on the assumption that raters’ deviations from their true perceptions are influenced by random chance only. James, Demaree, and Wolf extended Finn’s work by describing procedures to account for the additional influence of response biases. We demonstrate that organizational scientists have relied largely on Finn’s procedures, at least in part because of a lack of specific guidance regarding the conditions under which various response biases might be present. In an effort to address this gap in the literature, we introduce the concept of target-irrelevant, nonrandom forces (those aspects of the research context that are likely to lead to response biases), then describe how the familiar “5Ws and an H” framework (i.e., who, what, when, where, why, and how) can be used to identify these biases a priori. It is our hope that this system will permit those who calculate rwg to account for the effects of response biases in a manner that is simultaneously rigorous, consistent, and transparent.

AB - rwg is a common metric used to quantify interrater agreement in the organizational sciences. Finn developed rwg but based it on the assumption that raters’ deviations from their true perceptions are influenced by random chance only. James, Demaree, and Wolf extended Finn’s work by describing procedures to account for the additional influence of response biases. We demonstrate that organizational scientists have relied largely on Finn’s procedures, at least in part because of a lack of specific guidance regarding the conditions under which various response biases might be present. In an effort to address this gap in the literature, we introduce the concept of target-irrelevant, nonrandom forces (those aspects of the research context that are likely to lead to response biases), then describe how the familiar “5Ws and an H” framework (i.e., who, what, when, where, why, and how) can be used to identify these biases a priori. It is our hope that this system will permit those who calculate rwg to account for the effects of response biases in a manner that is simultaneously rigorous, consistent, and transparent.

UR - http://www.scopus.com/inward/record.url?scp=84910093199&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84910093199&partnerID=8YFLogxK

U2 - 10.1177/1094428114526927

DO - 10.1177/1094428114526927

M3 - Article

AN - SCOPUS:84910093199

VL - 17

SP - 324

EP - 345

JO - Organizational Research Methods

JF - Organizational Research Methods

SN - 1094-4281

IS - 3

ER -