Choosing the Best Method for Local Validity Estimation: Relative Accuracy of Meta-Analysis Versus a Local Study Versus Bayes-Analysis

Daniel A. Newman, Rick R. Jacobs, Dave Bartram

Research output: Contribution to journalArticle

36 Citations (Scopus)

Abstract

This study assessed the relative accuracy of 3 techniques-local validity studies, meta-analysis, and Bayesian analysis-for estimating test validity, incremental validity, and adverse impact in the local selection context. Bayes-analysis involves combining a local study with nonlocal (meta-analytic) validity data. Using tests of cognitive ability and personality (conscientiousness) as predictors, an empirically driven selection scenario illustrates conditions in which each of the 3 estimation techniques performs best. General recommendations are offered for how to estimate local parameters, based on true population variability (σp2) and the number of studies in the meta-analytic prior (k). Benefits of empirical Bayesian analysis for personnel selection are demonstrated, and equations are derived to help guide the choice of a local validity technique (i.e., meta-analysis vs. local study vs. Bayes-analysis).

Original languageEnglish (US)
Pages (from-to)1394-1413
Number of pages20
JournalJournal of Applied Psychology
Volume92
Issue number5
DOIs
StatePublished - Sep 1 2007

Fingerprint

Bayes Theorem
Meta-Analysis
Personnel Selection
Aptitude
Personality
Population

All Science Journal Classification (ASJC) codes

  • Applied Psychology

Cite this

@article{21db106ad8364352837f2540fed1aac3,
title = "Choosing the Best Method for Local Validity Estimation: Relative Accuracy of Meta-Analysis Versus a Local Study Versus Bayes-Analysis",
abstract = "This study assessed the relative accuracy of 3 techniques-local validity studies, meta-analysis, and Bayesian analysis-for estimating test validity, incremental validity, and adverse impact in the local selection context. Bayes-analysis involves combining a local study with nonlocal (meta-analytic) validity data. Using tests of cognitive ability and personality (conscientiousness) as predictors, an empirically driven selection scenario illustrates conditions in which each of the 3 estimation techniques performs best. General recommendations are offered for how to estimate local parameters, based on true population variability (σp2) and the number of studies in the meta-analytic prior (k). Benefits of empirical Bayesian analysis for personnel selection are demonstrated, and equations are derived to help guide the choice of a local validity technique (i.e., meta-analysis vs. local study vs. Bayes-analysis).",
author = "Newman, {Daniel A.} and Jacobs, {Rick R.} and Dave Bartram",
year = "2007",
month = "9",
day = "1",
doi = "10.1037/0021-9010.92.5.1394",
language = "English (US)",
volume = "92",
pages = "1394--1413",
journal = "Journal of Applied Psychology",
issn = "0021-9010",
publisher = "American Psychological Association Inc.",
number = "5",

}

Choosing the Best Method for Local Validity Estimation : Relative Accuracy of Meta-Analysis Versus a Local Study Versus Bayes-Analysis. / Newman, Daniel A.; Jacobs, Rick R.; Bartram, Dave.

In: Journal of Applied Psychology, Vol. 92, No. 5, 01.09.2007, p. 1394-1413.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Choosing the Best Method for Local Validity Estimation

T2 - Relative Accuracy of Meta-Analysis Versus a Local Study Versus Bayes-Analysis

AU - Newman, Daniel A.

AU - Jacobs, Rick R.

AU - Bartram, Dave

PY - 2007/9/1

Y1 - 2007/9/1

N2 - This study assessed the relative accuracy of 3 techniques-local validity studies, meta-analysis, and Bayesian analysis-for estimating test validity, incremental validity, and adverse impact in the local selection context. Bayes-analysis involves combining a local study with nonlocal (meta-analytic) validity data. Using tests of cognitive ability and personality (conscientiousness) as predictors, an empirically driven selection scenario illustrates conditions in which each of the 3 estimation techniques performs best. General recommendations are offered for how to estimate local parameters, based on true population variability (σp2) and the number of studies in the meta-analytic prior (k). Benefits of empirical Bayesian analysis for personnel selection are demonstrated, and equations are derived to help guide the choice of a local validity technique (i.e., meta-analysis vs. local study vs. Bayes-analysis).

AB - This study assessed the relative accuracy of 3 techniques-local validity studies, meta-analysis, and Bayesian analysis-for estimating test validity, incremental validity, and adverse impact in the local selection context. Bayes-analysis involves combining a local study with nonlocal (meta-analytic) validity data. Using tests of cognitive ability and personality (conscientiousness) as predictors, an empirically driven selection scenario illustrates conditions in which each of the 3 estimation techniques performs best. General recommendations are offered for how to estimate local parameters, based on true population variability (σp2) and the number of studies in the meta-analytic prior (k). Benefits of empirical Bayesian analysis for personnel selection are demonstrated, and equations are derived to help guide the choice of a local validity technique (i.e., meta-analysis vs. local study vs. Bayes-analysis).

UR - http://www.scopus.com/inward/record.url?scp=34748826719&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=34748826719&partnerID=8YFLogxK

U2 - 10.1037/0021-9010.92.5.1394

DO - 10.1037/0021-9010.92.5.1394

M3 - Article

C2 - 17845093

AN - SCOPUS:34748826719

VL - 92

SP - 1394

EP - 1413

JO - Journal of Applied Psychology

JF - Journal of Applied Psychology

SN - 0021-9010

IS - 5

ER -