Answers to 20 questions about interrater reliability and interrater agreement

James M. LeBreton, Jenell L. Senter

Research output: Contribution to journalArticle

1267 Citations (Scopus)

Abstract

The use of interrater reliability (IRR) and interrater agreement (IRA) indices has increased dramatically during the past 20 years. This popularity is, at least in part, because of the increased role of multilevel modeling techniques (e.g., hierarchical linear modeling and multilevel structural equation modeling) in organizational research. IRR and IRA indices are often used to justify aggregating lower-level data used in composition models. The purpose of the current article is to expose researchers to the various issues surrounding the use of IRR and IRA indices often used in conjunction with multilevel models. To achieve this goal, the authors adopt a question-and-answer format and provide a tutorial in the appendices illustrating how these indices may be computed using the SPSS software.

Original languageEnglish (US)
Pages (from-to)815-852
Number of pages38
JournalOrganizational Research Methods
Volume11
Issue number4
DOIs
StatePublished - Oct 1 2008

Fingerprint

Chemical analysis
Interrater reliability
Interrater agreement
Hierarchical linear modeling
Multilevel modeling
Structural equation modeling
Tutorial
Software
Multilevel models
Organizational research

All Science Journal Classification (ASJC) codes

  • Decision Sciences(all)
  • Strategy and Management
  • Management of Technology and Innovation

Cite this

@article{fb5d32ecefa04c3a86c0433bdf8c9782,
title = "Answers to 20 questions about interrater reliability and interrater agreement",
abstract = "The use of interrater reliability (IRR) and interrater agreement (IRA) indices has increased dramatically during the past 20 years. This popularity is, at least in part, because of the increased role of multilevel modeling techniques (e.g., hierarchical linear modeling and multilevel structural equation modeling) in organizational research. IRR and IRA indices are often used to justify aggregating lower-level data used in composition models. The purpose of the current article is to expose researchers to the various issues surrounding the use of IRR and IRA indices often used in conjunction with multilevel models. To achieve this goal, the authors adopt a question-and-answer format and provide a tutorial in the appendices illustrating how these indices may be computed using the SPSS software.",
author = "LeBreton, {James M.} and Senter, {Jenell L.}",
year = "2008",
month = "10",
day = "1",
doi = "10.1177/1094428106296642",
language = "English (US)",
volume = "11",
pages = "815--852",
journal = "Organizational Research Methods",
issn = "1094-4281",
publisher = "SAGE Publications Inc.",
number = "4",

}

Answers to 20 questions about interrater reliability and interrater agreement. / LeBreton, James M.; Senter, Jenell L.

In: Organizational Research Methods, Vol. 11, No. 4, 01.10.2008, p. 815-852.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Answers to 20 questions about interrater reliability and interrater agreement

AU - LeBreton, James M.

AU - Senter, Jenell L.

PY - 2008/10/1

Y1 - 2008/10/1

N2 - The use of interrater reliability (IRR) and interrater agreement (IRA) indices has increased dramatically during the past 20 years. This popularity is, at least in part, because of the increased role of multilevel modeling techniques (e.g., hierarchical linear modeling and multilevel structural equation modeling) in organizational research. IRR and IRA indices are often used to justify aggregating lower-level data used in composition models. The purpose of the current article is to expose researchers to the various issues surrounding the use of IRR and IRA indices often used in conjunction with multilevel models. To achieve this goal, the authors adopt a question-and-answer format and provide a tutorial in the appendices illustrating how these indices may be computed using the SPSS software.

AB - The use of interrater reliability (IRR) and interrater agreement (IRA) indices has increased dramatically during the past 20 years. This popularity is, at least in part, because of the increased role of multilevel modeling techniques (e.g., hierarchical linear modeling and multilevel structural equation modeling) in organizational research. IRR and IRA indices are often used to justify aggregating lower-level data used in composition models. The purpose of the current article is to expose researchers to the various issues surrounding the use of IRR and IRA indices often used in conjunction with multilevel models. To achieve this goal, the authors adopt a question-and-answer format and provide a tutorial in the appendices illustrating how these indices may be computed using the SPSS software.

UR - http://www.scopus.com/inward/record.url?scp=51649129337&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=51649129337&partnerID=8YFLogxK

U2 - 10.1177/1094428106296642

DO - 10.1177/1094428106296642

M3 - Article

AN - SCOPUS:51649129337

VL - 11

SP - 815

EP - 852

JO - Organizational Research Methods

JF - Organizational Research Methods

SN - 1094-4281

IS - 4

ER -