The High Cost of Complexity in Experimental Design and Data Analysis Type I and Type II Error Rates in Multiway ANOVA

Rachel A. Smith, Timothy R. Levine, Kenneth A. Lachlan, Thomas A. Fediuk

Research output: Contribution to journalArticle

38 Citations (Scopus)

Abstract

The availability of statistical software packages has led to a sharp increase in use of complex research designs and complex statistical analyses in communication research. An informal examination of studies from 2 leading communication journals suggests that the analysis of variance (ANOVA) is often the statistic of choice, and a substantial proportion of published research reports using ANOVA employ complex (k ≥ 3) factorial designs, often involving multiple dependent variables. This article reports a series of Monte Carlo simulations which demonstrate that this complexity may come at a heavier cost than many communication researchers realize. As frequently used, complex factorial ANOVA yield Type I and Type II error rates that many communication scholars would likely consider unacceptable. Consequently, quality of statistical inference in many studies is highly suspect. Communication researchers are warned about problems associated with design and statistical complexity and solutions are suggested.

Original languageEnglish (US)
Pages (from-to)515-530
Number of pages16
JournalHuman Communication Research
Volume28
Issue number4
DOIs
StatePublished - Oct 1 2002

Fingerprint

Analysis of variance (ANOVA)
analysis of variance
Design of experiments
Analysis of Variance
data analysis
Research Design
Costs and Cost Analysis
communication
Communication
costs
Research Personnel
Costs
communication research
Software
research planning
Software packages
statistics
Research
examination
simulation

All Science Journal Classification (ASJC) codes

  • Communication
  • Developmental and Educational Psychology
  • Anthropology
  • Linguistics and Language

Cite this

Smith, Rachel A. ; Levine, Timothy R. ; Lachlan, Kenneth A. ; Fediuk, Thomas A. / The High Cost of Complexity in Experimental Design and Data Analysis Type I and Type II Error Rates in Multiway ANOVA. In: Human Communication Research. 2002 ; Vol. 28, No. 4. pp. 515-530.
@article{df3bf806b5104440ac566d02abf70623,
title = "The High Cost of Complexity in Experimental Design and Data Analysis Type I and Type II Error Rates in Multiway ANOVA",
abstract = "The availability of statistical software packages has led to a sharp increase in use of complex research designs and complex statistical analyses in communication research. An informal examination of studies from 2 leading communication journals suggests that the analysis of variance (ANOVA) is often the statistic of choice, and a substantial proportion of published research reports using ANOVA employ complex (k ≥ 3) factorial designs, often involving multiple dependent variables. This article reports a series of Monte Carlo simulations which demonstrate that this complexity may come at a heavier cost than many communication researchers realize. As frequently used, complex factorial ANOVA yield Type I and Type II error rates that many communication scholars would likely consider unacceptable. Consequently, quality of statistical inference in many studies is highly suspect. Communication researchers are warned about problems associated with design and statistical complexity and solutions are suggested.",
author = "Smith, {Rachel A.} and Levine, {Timothy R.} and Lachlan, {Kenneth A.} and Fediuk, {Thomas A.}",
year = "2002",
month = "10",
day = "1",
doi = "10.1093/hcr/28.4.515",
language = "English (US)",
volume = "28",
pages = "515--530",
journal = "Human Communication Research",
issn = "0360-3989",
publisher = "Wiley-Blackwell",
number = "4",

}

The High Cost of Complexity in Experimental Design and Data Analysis Type I and Type II Error Rates in Multiway ANOVA. / Smith, Rachel A.; Levine, Timothy R.; Lachlan, Kenneth A.; Fediuk, Thomas A.

In: Human Communication Research, Vol. 28, No. 4, 01.10.2002, p. 515-530.

Research output: Contribution to journalArticle

TY - JOUR

T1 - The High Cost of Complexity in Experimental Design and Data Analysis Type I and Type II Error Rates in Multiway ANOVA

AU - Smith, Rachel A.

AU - Levine, Timothy R.

AU - Lachlan, Kenneth A.

AU - Fediuk, Thomas A.

PY - 2002/10/1

Y1 - 2002/10/1

N2 - The availability of statistical software packages has led to a sharp increase in use of complex research designs and complex statistical analyses in communication research. An informal examination of studies from 2 leading communication journals suggests that the analysis of variance (ANOVA) is often the statistic of choice, and a substantial proportion of published research reports using ANOVA employ complex (k ≥ 3) factorial designs, often involving multiple dependent variables. This article reports a series of Monte Carlo simulations which demonstrate that this complexity may come at a heavier cost than many communication researchers realize. As frequently used, complex factorial ANOVA yield Type I and Type II error rates that many communication scholars would likely consider unacceptable. Consequently, quality of statistical inference in many studies is highly suspect. Communication researchers are warned about problems associated with design and statistical complexity and solutions are suggested.

AB - The availability of statistical software packages has led to a sharp increase in use of complex research designs and complex statistical analyses in communication research. An informal examination of studies from 2 leading communication journals suggests that the analysis of variance (ANOVA) is often the statistic of choice, and a substantial proportion of published research reports using ANOVA employ complex (k ≥ 3) factorial designs, often involving multiple dependent variables. This article reports a series of Monte Carlo simulations which demonstrate that this complexity may come at a heavier cost than many communication researchers realize. As frequently used, complex factorial ANOVA yield Type I and Type II error rates that many communication scholars would likely consider unacceptable. Consequently, quality of statistical inference in many studies is highly suspect. Communication researchers are warned about problems associated with design and statistical complexity and solutions are suggested.

UR - http://www.scopus.com/inward/record.url?scp=0036412078&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0036412078&partnerID=8YFLogxK

U2 - 10.1093/hcr/28.4.515

DO - 10.1093/hcr/28.4.515

M3 - Article

AN - SCOPUS:0036412078

VL - 28

SP - 515

EP - 530

JO - Human Communication Research

JF - Human Communication Research

SN - 0360-3989

IS - 4

ER -