Assessing causal claims about complex engineered systems with quantitative data: internal, external, and construct validity

David A. Broniatowski, Conrad S. Tucker

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

Engineers seek to design systems that will produce an intended change in the state of the world. How are we to know if a system will behave as intended? This article addresses ways that this question can be answered. Specifically, we focus on three types of research validity: (1) internal validity, or whether an observed association between two variables can be attributed to a causal link between them; (2) external validity, or whether a causal link generalizes across contexts; and (3) construct validity, or whether a specific set of metrics corresponds to what they are intended to measure. In each case, we discuss techniques that may be used to establish the corresponding type of validity: namely, quasi-experimental design, replication, and establishment of convergent-discriminant validity and reliability. These techniques typically require access to data, which has historically been limited for research on complex engineered systems. This is likely to change in the era of “big data.” Thus, we discuss the continued utility of these validity concepts in the face of advances in machine learning and big data as they pertain to complex engineered sociotechnical systems. Next, we discuss relationships between these validity concepts and other prominent approaches to evaluating research in the field. Finally, we propose a set of criteria by which one may evaluate research utilizing quantitative observation to test causal theory in the field of complex engineered systems.

Original languageEnglish (US)
Pages (from-to)483-496
Number of pages14
JournalSystems Engineering
Volume20
Issue number6
DOIs
StatePublished - Nov 1 2017

Fingerprint

Large scale systems
Design of experiments
Learning systems
Engineers
Big data

All Science Journal Classification (ASJC) codes

  • Hardware and Architecture
  • Computer Networks and Communications

Cite this

@article{11e6bd1b05be4dd9ac53047867cb73c9,
title = "Assessing causal claims about complex engineered systems with quantitative data: internal, external, and construct validity",
abstract = "Engineers seek to design systems that will produce an intended change in the state of the world. How are we to know if a system will behave as intended? This article addresses ways that this question can be answered. Specifically, we focus on three types of research validity: (1) internal validity, or whether an observed association between two variables can be attributed to a causal link between them; (2) external validity, or whether a causal link generalizes across contexts; and (3) construct validity, or whether a specific set of metrics corresponds to what they are intended to measure. In each case, we discuss techniques that may be used to establish the corresponding type of validity: namely, quasi-experimental design, replication, and establishment of convergent-discriminant validity and reliability. These techniques typically require access to data, which has historically been limited for research on complex engineered systems. This is likely to change in the era of “big data.” Thus, we discuss the continued utility of these validity concepts in the face of advances in machine learning and big data as they pertain to complex engineered sociotechnical systems. Next, we discuss relationships between these validity concepts and other prominent approaches to evaluating research in the field. Finally, we propose a set of criteria by which one may evaluate research utilizing quantitative observation to test causal theory in the field of complex engineered systems.",
author = "Broniatowski, {David A.} and Tucker, {Conrad S.}",
year = "2017",
month = "11",
day = "1",
doi = "10.1002/sys.21414",
language = "English (US)",
volume = "20",
pages = "483--496",
journal = "Systems Engineering",
issn = "1098-1241",
publisher = "John Wiley and Sons Inc.",
number = "6",

}

Assessing causal claims about complex engineered systems with quantitative data : internal, external, and construct validity. / Broniatowski, David A.; Tucker, Conrad S.

In: Systems Engineering, Vol. 20, No. 6, 01.11.2017, p. 483-496.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Assessing causal claims about complex engineered systems with quantitative data

T2 - internal, external, and construct validity

AU - Broniatowski, David A.

AU - Tucker, Conrad S.

PY - 2017/11/1

Y1 - 2017/11/1

N2 - Engineers seek to design systems that will produce an intended change in the state of the world. How are we to know if a system will behave as intended? This article addresses ways that this question can be answered. Specifically, we focus on three types of research validity: (1) internal validity, or whether an observed association between two variables can be attributed to a causal link between them; (2) external validity, or whether a causal link generalizes across contexts; and (3) construct validity, or whether a specific set of metrics corresponds to what they are intended to measure. In each case, we discuss techniques that may be used to establish the corresponding type of validity: namely, quasi-experimental design, replication, and establishment of convergent-discriminant validity and reliability. These techniques typically require access to data, which has historically been limited for research on complex engineered systems. This is likely to change in the era of “big data.” Thus, we discuss the continued utility of these validity concepts in the face of advances in machine learning and big data as they pertain to complex engineered sociotechnical systems. Next, we discuss relationships between these validity concepts and other prominent approaches to evaluating research in the field. Finally, we propose a set of criteria by which one may evaluate research utilizing quantitative observation to test causal theory in the field of complex engineered systems.

AB - Engineers seek to design systems that will produce an intended change in the state of the world. How are we to know if a system will behave as intended? This article addresses ways that this question can be answered. Specifically, we focus on three types of research validity: (1) internal validity, or whether an observed association between two variables can be attributed to a causal link between them; (2) external validity, or whether a causal link generalizes across contexts; and (3) construct validity, or whether a specific set of metrics corresponds to what they are intended to measure. In each case, we discuss techniques that may be used to establish the corresponding type of validity: namely, quasi-experimental design, replication, and establishment of convergent-discriminant validity and reliability. These techniques typically require access to data, which has historically been limited for research on complex engineered systems. This is likely to change in the era of “big data.” Thus, we discuss the continued utility of these validity concepts in the face of advances in machine learning and big data as they pertain to complex engineered sociotechnical systems. Next, we discuss relationships between these validity concepts and other prominent approaches to evaluating research in the field. Finally, we propose a set of criteria by which one may evaluate research utilizing quantitative observation to test causal theory in the field of complex engineered systems.

UR - http://www.scopus.com/inward/record.url?scp=85040843717&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85040843717&partnerID=8YFLogxK

U2 - 10.1002/sys.21414

DO - 10.1002/sys.21414

M3 - Article

AN - SCOPUS:85040843717

VL - 20

SP - 483

EP - 496

JO - Systems Engineering

JF - Systems Engineering

SN - 1098-1241

IS - 6

ER -