Evolutionary regression? Assessing the problem of hidden biases in criminal justice applications using propensity scores

Thomas A. Loughran, Theodore Wilson, Daniel S. Nagin, Alex R. Piquero

Research output: Contribution to journalArticle

23 Citations (Scopus)

Abstract

Objectives: Propensity score methods rely on an untestable assumption of unconfoundedness for making causal inference. Yet, empirical applications using propensity scores in criminology routinely invoke this assumption without careful scrutiny. Methods: We use a dataset with a wide range of observable, potential confounders, which allows us to evaluate recidivism outcomes for adolescent offenders who are sentenced to either placement or probation. We then systematically withhold important known confounders from the matching process to demonstrate the effectiveness of sensitivity checks in sizing up the robustness of these treatment effect estimates in the case where hidden biases clearly exist. Results: We find important variability in the estimated treatment effect, and a large degree of imbalance in ‘unobserved’ covariates, which we did not explicitly control for. The hidden biases observed in our controlled analysis would have at least been suggested in an actual application by the low gamma statistics that attended our analysis, a statistic that is not reported in most criminological applications of propensity score analysis. Conclusions: Researchers who use propensity score methods should openly discuss potential limitations of their analysis due to hidden bias and report bias sensitivity checks based on the gamma statistic when statistically significant treatment effects are reported.

Original languageEnglish (US)
Pages (from-to)631-652
Number of pages22
JournalJournal of Experimental Criminology
Volume11
Issue number4
DOIs
StatePublished - Dec 1 2015

Fingerprint

justice
regression
statistics
trend
probation
criminology
offender
adolescent

All Science Journal Classification (ASJC) codes

  • Law

Cite this

@article{7aa213e82fc748438119f69e96c55f23,
title = "Evolutionary regression? Assessing the problem of hidden biases in criminal justice applications using propensity scores",
abstract = "Objectives: Propensity score methods rely on an untestable assumption of unconfoundedness for making causal inference. Yet, empirical applications using propensity scores in criminology routinely invoke this assumption without careful scrutiny. Methods: We use a dataset with a wide range of observable, potential confounders, which allows us to evaluate recidivism outcomes for adolescent offenders who are sentenced to either placement or probation. We then systematically withhold important known confounders from the matching process to demonstrate the effectiveness of sensitivity checks in sizing up the robustness of these treatment effect estimates in the case where hidden biases clearly exist. Results: We find important variability in the estimated treatment effect, and a large degree of imbalance in ‘unobserved’ covariates, which we did not explicitly control for. The hidden biases observed in our controlled analysis would have at least been suggested in an actual application by the low gamma statistics that attended our analysis, a statistic that is not reported in most criminological applications of propensity score analysis. Conclusions: Researchers who use propensity score methods should openly discuss potential limitations of their analysis due to hidden bias and report bias sensitivity checks based on the gamma statistic when statistically significant treatment effects are reported.",
author = "Loughran, {Thomas A.} and Theodore Wilson and Nagin, {Daniel S.} and Piquero, {Alex R.}",
year = "2015",
month = "12",
day = "1",
doi = "10.1007/s11292-015-9242-y",
language = "English (US)",
volume = "11",
pages = "631--652",
journal = "Journal of Experimental Criminology",
issn = "1573-3750",
publisher = "Springer Netherlands",
number = "4",

}

Evolutionary regression? Assessing the problem of hidden biases in criminal justice applications using propensity scores. / Loughran, Thomas A.; Wilson, Theodore; Nagin, Daniel S.; Piquero, Alex R.

In: Journal of Experimental Criminology, Vol. 11, No. 4, 01.12.2015, p. 631-652.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Evolutionary regression? Assessing the problem of hidden biases in criminal justice applications using propensity scores

AU - Loughran, Thomas A.

AU - Wilson, Theodore

AU - Nagin, Daniel S.

AU - Piquero, Alex R.

PY - 2015/12/1

Y1 - 2015/12/1

N2 - Objectives: Propensity score methods rely on an untestable assumption of unconfoundedness for making causal inference. Yet, empirical applications using propensity scores in criminology routinely invoke this assumption without careful scrutiny. Methods: We use a dataset with a wide range of observable, potential confounders, which allows us to evaluate recidivism outcomes for adolescent offenders who are sentenced to either placement or probation. We then systematically withhold important known confounders from the matching process to demonstrate the effectiveness of sensitivity checks in sizing up the robustness of these treatment effect estimates in the case where hidden biases clearly exist. Results: We find important variability in the estimated treatment effect, and a large degree of imbalance in ‘unobserved’ covariates, which we did not explicitly control for. The hidden biases observed in our controlled analysis would have at least been suggested in an actual application by the low gamma statistics that attended our analysis, a statistic that is not reported in most criminological applications of propensity score analysis. Conclusions: Researchers who use propensity score methods should openly discuss potential limitations of their analysis due to hidden bias and report bias sensitivity checks based on the gamma statistic when statistically significant treatment effects are reported.

AB - Objectives: Propensity score methods rely on an untestable assumption of unconfoundedness for making causal inference. Yet, empirical applications using propensity scores in criminology routinely invoke this assumption without careful scrutiny. Methods: We use a dataset with a wide range of observable, potential confounders, which allows us to evaluate recidivism outcomes for adolescent offenders who are sentenced to either placement or probation. We then systematically withhold important known confounders from the matching process to demonstrate the effectiveness of sensitivity checks in sizing up the robustness of these treatment effect estimates in the case where hidden biases clearly exist. Results: We find important variability in the estimated treatment effect, and a large degree of imbalance in ‘unobserved’ covariates, which we did not explicitly control for. The hidden biases observed in our controlled analysis would have at least been suggested in an actual application by the low gamma statistics that attended our analysis, a statistic that is not reported in most criminological applications of propensity score analysis. Conclusions: Researchers who use propensity score methods should openly discuss potential limitations of their analysis due to hidden bias and report bias sensitivity checks based on the gamma statistic when statistically significant treatment effects are reported.

UR - http://www.scopus.com/inward/record.url?scp=84953638088&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84953638088&partnerID=8YFLogxK

U2 - 10.1007/s11292-015-9242-y

DO - 10.1007/s11292-015-9242-y

M3 - Article

AN - SCOPUS:84953638088

VL - 11

SP - 631

EP - 652

JO - Journal of Experimental Criminology

JF - Journal of Experimental Criminology

SN - 1573-3750

IS - 4

ER -