A game-based experimental protocol for identifying and overcoming judgment biases in forensic decision analysis

Donald R. Kretz, B. J. Simpson, Colonel Jacob Graham

Research output: Chapter in Book/Report/Conference proceedingConference contribution

9 Citations (Scopus)

Abstract

In the aftermath of several recent catastrophic intelligence failures, the U.S. Government commissioned a series of studies to evaluate analytic methods and tradecraft. Those reports made specific recommendations to address consistent and systematic errors known as judgment biases found in all forms of analysis: predictive, estimative, and forensic. To correct for bias, a small number of methodological improvements have been suggested. There is, however, little experimental evidence to validate their impact on analytic quality. This lack of support motivates our present work, which seeks significant improvements in analytic performance by identifying common biases that emerge during analytic tasks, as well as measuring the effects of corresponding corrective measures (a.k.a.. "debiasing" techniques), which we refer to as analytic multipliers. This effort requires an experimental protocol suitable for studying the effects of many types of biases and debiasing techniques on realistic analytic problems. This paper presents our game-based paradigm for studying decision biases and developing analytic multipliers, and includes a description and results of a pilot game we developed to validate the approach.

Original languageEnglish (US)
Title of host publication2012 IEEE International Conference on Technologies for Homeland Security, HST 2012
Pages439-444
Number of pages6
DOIs
StatePublished - Dec 1 2012
Event2012 12th IEEE International Conference on Technologies for Homeland Security, HST 2012 - Waltham, MA, United States
Duration: Nov 13 2012Nov 15 2012

Publication series

Name2012 IEEE International Conference on Technologies for Homeland Security, HST 2012

Other

Other2012 12th IEEE International Conference on Technologies for Homeland Security, HST 2012
CountryUnited States
CityWaltham, MA
Period11/13/1211/15/12

Fingerprint

trend
multiplier
intelligence
paradigm
lack
performance
evidence

All Science Journal Classification (ASJC) codes

  • Law

Cite this

Kretz, D. R., Simpson, B. J., & Graham, C. J. (2012). A game-based experimental protocol for identifying and overcoming judgment biases in forensic decision analysis. In 2012 IEEE International Conference on Technologies for Homeland Security, HST 2012 (pp. 439-444). [6459889] (2012 IEEE International Conference on Technologies for Homeland Security, HST 2012). https://doi.org/10.1109/THS.2012.6459889
Kretz, Donald R. ; Simpson, B. J. ; Graham, Colonel Jacob. / A game-based experimental protocol for identifying and overcoming judgment biases in forensic decision analysis. 2012 IEEE International Conference on Technologies for Homeland Security, HST 2012. 2012. pp. 439-444 (2012 IEEE International Conference on Technologies for Homeland Security, HST 2012).
@inproceedings{1d9542e309234dd193427060b4c4e4fc,
title = "A game-based experimental protocol for identifying and overcoming judgment biases in forensic decision analysis",
abstract = "In the aftermath of several recent catastrophic intelligence failures, the U.S. Government commissioned a series of studies to evaluate analytic methods and tradecraft. Those reports made specific recommendations to address consistent and systematic errors known as judgment biases found in all forms of analysis: predictive, estimative, and forensic. To correct for bias, a small number of methodological improvements have been suggested. There is, however, little experimental evidence to validate their impact on analytic quality. This lack of support motivates our present work, which seeks significant improvements in analytic performance by identifying common biases that emerge during analytic tasks, as well as measuring the effects of corresponding corrective measures (a.k.a.. {"}debiasing{"} techniques), which we refer to as analytic multipliers. This effort requires an experimental protocol suitable for studying the effects of many types of biases and debiasing techniques on realistic analytic problems. This paper presents our game-based paradigm for studying decision biases and developing analytic multipliers, and includes a description and results of a pilot game we developed to validate the approach.",
author = "Kretz, {Donald R.} and Simpson, {B. J.} and Graham, {Colonel Jacob}",
year = "2012",
month = "12",
day = "1",
doi = "10.1109/THS.2012.6459889",
language = "English (US)",
isbn = "9781467327084",
series = "2012 IEEE International Conference on Technologies for Homeland Security, HST 2012",
pages = "439--444",
booktitle = "2012 IEEE International Conference on Technologies for Homeland Security, HST 2012",

}

Kretz, DR, Simpson, BJ & Graham, CJ 2012, A game-based experimental protocol for identifying and overcoming judgment biases in forensic decision analysis. in 2012 IEEE International Conference on Technologies for Homeland Security, HST 2012., 6459889, 2012 IEEE International Conference on Technologies for Homeland Security, HST 2012, pp. 439-444, 2012 12th IEEE International Conference on Technologies for Homeland Security, HST 2012, Waltham, MA, United States, 11/13/12. https://doi.org/10.1109/THS.2012.6459889

A game-based experimental protocol for identifying and overcoming judgment biases in forensic decision analysis. / Kretz, Donald R.; Simpson, B. J.; Graham, Colonel Jacob.

2012 IEEE International Conference on Technologies for Homeland Security, HST 2012. 2012. p. 439-444 6459889 (2012 IEEE International Conference on Technologies for Homeland Security, HST 2012).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - A game-based experimental protocol for identifying and overcoming judgment biases in forensic decision analysis

AU - Kretz, Donald R.

AU - Simpson, B. J.

AU - Graham, Colonel Jacob

PY - 2012/12/1

Y1 - 2012/12/1

N2 - In the aftermath of several recent catastrophic intelligence failures, the U.S. Government commissioned a series of studies to evaluate analytic methods and tradecraft. Those reports made specific recommendations to address consistent and systematic errors known as judgment biases found in all forms of analysis: predictive, estimative, and forensic. To correct for bias, a small number of methodological improvements have been suggested. There is, however, little experimental evidence to validate their impact on analytic quality. This lack of support motivates our present work, which seeks significant improvements in analytic performance by identifying common biases that emerge during analytic tasks, as well as measuring the effects of corresponding corrective measures (a.k.a.. "debiasing" techniques), which we refer to as analytic multipliers. This effort requires an experimental protocol suitable for studying the effects of many types of biases and debiasing techniques on realistic analytic problems. This paper presents our game-based paradigm for studying decision biases and developing analytic multipliers, and includes a description and results of a pilot game we developed to validate the approach.

AB - In the aftermath of several recent catastrophic intelligence failures, the U.S. Government commissioned a series of studies to evaluate analytic methods and tradecraft. Those reports made specific recommendations to address consistent and systematic errors known as judgment biases found in all forms of analysis: predictive, estimative, and forensic. To correct for bias, a small number of methodological improvements have been suggested. There is, however, little experimental evidence to validate their impact on analytic quality. This lack of support motivates our present work, which seeks significant improvements in analytic performance by identifying common biases that emerge during analytic tasks, as well as measuring the effects of corresponding corrective measures (a.k.a.. "debiasing" techniques), which we refer to as analytic multipliers. This effort requires an experimental protocol suitable for studying the effects of many types of biases and debiasing techniques on realistic analytic problems. This paper presents our game-based paradigm for studying decision biases and developing analytic multipliers, and includes a description and results of a pilot game we developed to validate the approach.

UR - http://www.scopus.com/inward/record.url?scp=84874580228&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84874580228&partnerID=8YFLogxK

U2 - 10.1109/THS.2012.6459889

DO - 10.1109/THS.2012.6459889

M3 - Conference contribution

SN - 9781467327084

T3 - 2012 IEEE International Conference on Technologies for Homeland Security, HST 2012

SP - 439

EP - 444

BT - 2012 IEEE International Conference on Technologies for Homeland Security, HST 2012

ER -

Kretz DR, Simpson BJ, Graham CJ. A game-based experimental protocol for identifying and overcoming judgment biases in forensic decision analysis. In 2012 IEEE International Conference on Technologies for Homeland Security, HST 2012. 2012. p. 439-444. 6459889. (2012 IEEE International Conference on Technologies for Homeland Security, HST 2012). https://doi.org/10.1109/THS.2012.6459889