A recent article by Lindell and Brandt raised two concerns regarding the use of James, Demaree, and Wolf's interrater agreement indices r WG and r WG(J). First, they noted that the multi-item r WG(J) equation is mathematically equivalent to inserting r WG into the Spearman-Brown prophecy formula and questioned whether applying a formula developed for a reliability index was also appropriate for an agreement index. Second, they questioned the appropriateness of James et al.'s suggestion of replacing obtained negative values of r WG with zeros. This article addresses these concerns by demonstrating that r WG(J) can be derived independently from the Spearman-Brown prophecy formula and that negative values of r WG can be avoided by reparameterizing the structural equation underlying the data when there is systematic disagreement between subgroups of raters.
All Science Journal Classification (ASJC) codes
- Decision Sciences(all)
- Strategy and Management
- Management of Technology and Innovation