Multilevel factorial experiments for developing behavioral interventions

Power, sample size, and resource considerations

John Dziak, Inbal Nahum-Shani, Linda Marie Collins

Research output: Contribution to journalArticle

34 Citations (Scopus)

Abstract

Factorial experimental designs have many potential advantages for behavioral scientists. For example, such designs may be useful in building more potent interventions by helping investigators to screen several candidate intervention components simultaneously and to decide which are likely to offer greater benefit before evaluating the intervention as a whole. However, sample size and power considerations may challenge investigators attempting to apply such designs, especially when the population of interest is multilevel (e.g., when students are nested within schools, or when employees are nested within organizations). In this article, we examine the feasibility of factorial experimental designs with multiple factors in a multilevel, clustered setting (i.e., of multilevel, multifactor experiments). We conduct Monte Carlo simulations to demonstrate how design elements-such as the number of clusters, the number of lower-level units, and the intraclass correlation-affect power. Our results suggest that multilevel, multifactor experiments are feasible for factor-screening purposes because of the economical properties of complete and fractional factorial experimental designs. We also discuss resources for sample size planning and power estimation for multilevel factorial experiments. These results are discussed from a resource management perspective, in which the goal is to choose a design that maximizes the scientific benefit using the resources available for an investigation.

Original languageEnglish (US)
Pages (from-to)153-175
Number of pages23
JournalPsychological Methods
Volume17
Issue number2
DOIs
StatePublished - Jun 1 2012

Fingerprint

Sample Size
Research Design
Research Personnel
Students
Population
Power (Psychology)

All Science Journal Classification (ASJC) codes

  • Psychology (miscellaneous)

Cite this

@article{9ddefb85626f477796dbb1c7a58be937,
title = "Multilevel factorial experiments for developing behavioral interventions: Power, sample size, and resource considerations",
abstract = "Factorial experimental designs have many potential advantages for behavioral scientists. For example, such designs may be useful in building more potent interventions by helping investigators to screen several candidate intervention components simultaneously and to decide which are likely to offer greater benefit before evaluating the intervention as a whole. However, sample size and power considerations may challenge investigators attempting to apply such designs, especially when the population of interest is multilevel (e.g., when students are nested within schools, or when employees are nested within organizations). In this article, we examine the feasibility of factorial experimental designs with multiple factors in a multilevel, clustered setting (i.e., of multilevel, multifactor experiments). We conduct Monte Carlo simulations to demonstrate how design elements-such as the number of clusters, the number of lower-level units, and the intraclass correlation-affect power. Our results suggest that multilevel, multifactor experiments are feasible for factor-screening purposes because of the economical properties of complete and fractional factorial experimental designs. We also discuss resources for sample size planning and power estimation for multilevel factorial experiments. These results are discussed from a resource management perspective, in which the goal is to choose a design that maximizes the scientific benefit using the resources available for an investigation.",
author = "John Dziak and Inbal Nahum-Shani and Collins, {Linda Marie}",
year = "2012",
month = "6",
day = "1",
doi = "10.1037/a0026972",
language = "English (US)",
volume = "17",
pages = "153--175",
journal = "Psychological Methods",
issn = "1082-989X",
publisher = "American Psychological Association Inc.",
number = "2",

}

Multilevel factorial experiments for developing behavioral interventions : Power, sample size, and resource considerations. / Dziak, John; Nahum-Shani, Inbal; Collins, Linda Marie.

In: Psychological Methods, Vol. 17, No. 2, 01.06.2012, p. 153-175.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Multilevel factorial experiments for developing behavioral interventions

T2 - Power, sample size, and resource considerations

AU - Dziak, John

AU - Nahum-Shani, Inbal

AU - Collins, Linda Marie

PY - 2012/6/1

Y1 - 2012/6/1

N2 - Factorial experimental designs have many potential advantages for behavioral scientists. For example, such designs may be useful in building more potent interventions by helping investigators to screen several candidate intervention components simultaneously and to decide which are likely to offer greater benefit before evaluating the intervention as a whole. However, sample size and power considerations may challenge investigators attempting to apply such designs, especially when the population of interest is multilevel (e.g., when students are nested within schools, or when employees are nested within organizations). In this article, we examine the feasibility of factorial experimental designs with multiple factors in a multilevel, clustered setting (i.e., of multilevel, multifactor experiments). We conduct Monte Carlo simulations to demonstrate how design elements-such as the number of clusters, the number of lower-level units, and the intraclass correlation-affect power. Our results suggest that multilevel, multifactor experiments are feasible for factor-screening purposes because of the economical properties of complete and fractional factorial experimental designs. We also discuss resources for sample size planning and power estimation for multilevel factorial experiments. These results are discussed from a resource management perspective, in which the goal is to choose a design that maximizes the scientific benefit using the resources available for an investigation.

AB - Factorial experimental designs have many potential advantages for behavioral scientists. For example, such designs may be useful in building more potent interventions by helping investigators to screen several candidate intervention components simultaneously and to decide which are likely to offer greater benefit before evaluating the intervention as a whole. However, sample size and power considerations may challenge investigators attempting to apply such designs, especially when the population of interest is multilevel (e.g., when students are nested within schools, or when employees are nested within organizations). In this article, we examine the feasibility of factorial experimental designs with multiple factors in a multilevel, clustered setting (i.e., of multilevel, multifactor experiments). We conduct Monte Carlo simulations to demonstrate how design elements-such as the number of clusters, the number of lower-level units, and the intraclass correlation-affect power. Our results suggest that multilevel, multifactor experiments are feasible for factor-screening purposes because of the economical properties of complete and fractional factorial experimental designs. We also discuss resources for sample size planning and power estimation for multilevel factorial experiments. These results are discussed from a resource management perspective, in which the goal is to choose a design that maximizes the scientific benefit using the resources available for an investigation.

UR - http://www.scopus.com/inward/record.url?scp=84866436598&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84866436598&partnerID=8YFLogxK

U2 - 10.1037/a0026972

DO - 10.1037/a0026972

M3 - Article

VL - 17

SP - 153

EP - 175

JO - Psychological Methods

JF - Psychological Methods

SN - 1082-989X

IS - 2

ER -