Investigating variation in replicability: A "many labs" replication project

Richard A. Klein, Kate A. Ratliff, Michelangelo Vianello, Reginald B. Adams, Štěpán Bahník, Michael J. Bernstein, Konrad Bocian, Mark J. Brandt, Beach Brooks, Claudia Chloe Brumbaugh, Zeynep Cemalcilar, Jesse Chandler, Winnee Cheong, William E. Davis, Thierry Devos, Matthew Eisner, Natalia Frankowska, David Furrow, Elisa Maria Galliani, Fred HasselmanJoshua A. Hicks, James F. Hovermale, S. Jane Hunt, Jeffrey R. Huntsinger, Hans Ijzerman, Melissa Sue John, Jennifer A. Joy-Gaba, Heather Barry Kappes, Lacy E. Krueger, Jaime Kurtz, Carmel A. Levitan, Robyn K. Mallett, Wendy L. Morris, Anthony J. Nelson, Jason A. Nier, Grant Packard, Ronaldo Pilati, Abraham M. Rutchick, Kathleen Schmidt, Jeanine L. Skorinko, Robert Smith, Troy G. Steiner, Justin Storbeck, Lyn M. Van Swol, Donna Thompson, A. E. Van 'T Veer, Leigh Ann Vaughn, Marek Vranka, Aaron L. Wichman, Julie A. Woodzicka, Brian A. Nosek

Research output: Contribution to journalArticle

312 Citations (Scopus)

Abstract

Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of 13 classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, 10 effects replicated consistently. One effect - imagined contact reducing prejudice - showed weak support for replicability. And two effects - flag priming influencing conservatism and currency priming influencing system justification - did not replicate. We compared whether the conditions such as lab versus online or US versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.

Original languageEnglish (US)
Pages (from-to)142-152
Number of pages11
JournalSocial Psychology
Volume45
Issue number3
DOIs
StatePublished - 2014

Fingerprint

Politics
Psychology
Research
Replication
conservatism
prejudice
currency
psychology
contact
science

All Science Journal Classification (ASJC) codes

  • Social Psychology
  • Arts and Humanities (miscellaneous)
  • Sociology and Political Science
  • Psychology(all)

Cite this

Klein, Richard A. ; Ratliff, Kate A. ; Vianello, Michelangelo ; Adams, Reginald B. ; Bahník, Štěpán ; Bernstein, Michael J. ; Bocian, Konrad ; Brandt, Mark J. ; Brooks, Beach ; Brumbaugh, Claudia Chloe ; Cemalcilar, Zeynep ; Chandler, Jesse ; Cheong, Winnee ; Davis, William E. ; Devos, Thierry ; Eisner, Matthew ; Frankowska, Natalia ; Furrow, David ; Galliani, Elisa Maria ; Hasselman, Fred ; Hicks, Joshua A. ; Hovermale, James F. ; Hunt, S. Jane ; Huntsinger, Jeffrey R. ; Ijzerman, Hans ; John, Melissa Sue ; Joy-Gaba, Jennifer A. ; Kappes, Heather Barry ; Krueger, Lacy E. ; Kurtz, Jaime ; Levitan, Carmel A. ; Mallett, Robyn K. ; Morris, Wendy L. ; Nelson, Anthony J. ; Nier, Jason A. ; Packard, Grant ; Pilati, Ronaldo ; Rutchick, Abraham M. ; Schmidt, Kathleen ; Skorinko, Jeanine L. ; Smith, Robert ; Steiner, Troy G. ; Storbeck, Justin ; Van Swol, Lyn M. ; Thompson, Donna ; Van 'T Veer, A. E. ; Vaughn, Leigh Ann ; Vranka, Marek ; Wichman, Aaron L. ; Woodzicka, Julie A. ; Nosek, Brian A. / Investigating variation in replicability : A "many labs" replication project. In: Social Psychology. 2014 ; Vol. 45, No. 3. pp. 142-152.
@article{6ec44fad54a943dcace8ba7287b1fe9c,
title = "Investigating variation in replicability: A {"}many labs{"} replication project",
abstract = "Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of 13 classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, 10 effects replicated consistently. One effect - imagined contact reducing prejudice - showed weak support for replicability. And two effects - flag priming influencing conservatism and currency priming influencing system justification - did not replicate. We compared whether the conditions such as lab versus online or US versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.",
author = "Klein, {Richard A.} and Ratliff, {Kate A.} and Michelangelo Vianello and Adams, {Reginald B.} and Štěp{\'a}n Bahn{\'i}k and Bernstein, {Michael J.} and Konrad Bocian and Brandt, {Mark J.} and Beach Brooks and Brumbaugh, {Claudia Chloe} and Zeynep Cemalcilar and Jesse Chandler and Winnee Cheong and Davis, {William E.} and Thierry Devos and Matthew Eisner and Natalia Frankowska and David Furrow and Galliani, {Elisa Maria} and Fred Hasselman and Hicks, {Joshua A.} and Hovermale, {James F.} and Hunt, {S. Jane} and Huntsinger, {Jeffrey R.} and Hans Ijzerman and John, {Melissa Sue} and Joy-Gaba, {Jennifer A.} and Kappes, {Heather Barry} and Krueger, {Lacy E.} and Jaime Kurtz and Levitan, {Carmel A.} and Mallett, {Robyn K.} and Morris, {Wendy L.} and Nelson, {Anthony J.} and Nier, {Jason A.} and Grant Packard and Ronaldo Pilati and Rutchick, {Abraham M.} and Kathleen Schmidt and Skorinko, {Jeanine L.} and Robert Smith and Steiner, {Troy G.} and Justin Storbeck and {Van Swol}, {Lyn M.} and Donna Thompson and {Van 'T Veer}, {A. E.} and Vaughn, {Leigh Ann} and Marek Vranka and Wichman, {Aaron L.} and Woodzicka, {Julie A.} and Nosek, {Brian A.}",
year = "2014",
doi = "10.1027/1864-9335/a000178",
language = "English (US)",
volume = "45",
pages = "142--152",
journal = "Social Psychology",
issn = "1864-9335",
publisher = "Verlag Hans Huber",
number = "3",

}

Klein, RA, Ratliff, KA, Vianello, M, Adams, RB, Bahník, Š, Bernstein, MJ, Bocian, K, Brandt, MJ, Brooks, B, Brumbaugh, CC, Cemalcilar, Z, Chandler, J, Cheong, W, Davis, WE, Devos, T, Eisner, M, Frankowska, N, Furrow, D, Galliani, EM, Hasselman, F, Hicks, JA, Hovermale, JF, Hunt, SJ, Huntsinger, JR, Ijzerman, H, John, MS, Joy-Gaba, JA, Kappes, HB, Krueger, LE, Kurtz, J, Levitan, CA, Mallett, RK, Morris, WL, Nelson, AJ, Nier, JA, Packard, G, Pilati, R, Rutchick, AM, Schmidt, K, Skorinko, JL, Smith, R, Steiner, TG, Storbeck, J, Van Swol, LM, Thompson, D, Van 'T Veer, AE, Vaughn, LA, Vranka, M, Wichman, AL, Woodzicka, JA & Nosek, BA 2014, 'Investigating variation in replicability: A "many labs" replication project', Social Psychology, vol. 45, no. 3, pp. 142-152. https://doi.org/10.1027/1864-9335/a000178

Investigating variation in replicability : A "many labs" replication project. / Klein, Richard A.; Ratliff, Kate A.; Vianello, Michelangelo; Adams, Reginald B.; Bahník, Štěpán; Bernstein, Michael J.; Bocian, Konrad; Brandt, Mark J.; Brooks, Beach; Brumbaugh, Claudia Chloe; Cemalcilar, Zeynep; Chandler, Jesse; Cheong, Winnee; Davis, William E.; Devos, Thierry; Eisner, Matthew; Frankowska, Natalia; Furrow, David; Galliani, Elisa Maria; Hasselman, Fred; Hicks, Joshua A.; Hovermale, James F.; Hunt, S. Jane; Huntsinger, Jeffrey R.; Ijzerman, Hans; John, Melissa Sue; Joy-Gaba, Jennifer A.; Kappes, Heather Barry; Krueger, Lacy E.; Kurtz, Jaime; Levitan, Carmel A.; Mallett, Robyn K.; Morris, Wendy L.; Nelson, Anthony J.; Nier, Jason A.; Packard, Grant; Pilati, Ronaldo; Rutchick, Abraham M.; Schmidt, Kathleen; Skorinko, Jeanine L.; Smith, Robert; Steiner, Troy G.; Storbeck, Justin; Van Swol, Lyn M.; Thompson, Donna; Van 'T Veer, A. E.; Vaughn, Leigh Ann; Vranka, Marek; Wichman, Aaron L.; Woodzicka, Julie A.; Nosek, Brian A.

In: Social Psychology, Vol. 45, No. 3, 2014, p. 142-152.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Investigating variation in replicability

T2 - A "many labs" replication project

AU - Klein, Richard A.

AU - Ratliff, Kate A.

AU - Vianello, Michelangelo

AU - Adams, Reginald B.

AU - Bahník, Štěpán

AU - Bernstein, Michael J.

AU - Bocian, Konrad

AU - Brandt, Mark J.

AU - Brooks, Beach

AU - Brumbaugh, Claudia Chloe

AU - Cemalcilar, Zeynep

AU - Chandler, Jesse

AU - Cheong, Winnee

AU - Davis, William E.

AU - Devos, Thierry

AU - Eisner, Matthew

AU - Frankowska, Natalia

AU - Furrow, David

AU - Galliani, Elisa Maria

AU - Hasselman, Fred

AU - Hicks, Joshua A.

AU - Hovermale, James F.

AU - Hunt, S. Jane

AU - Huntsinger, Jeffrey R.

AU - Ijzerman, Hans

AU - John, Melissa Sue

AU - Joy-Gaba, Jennifer A.

AU - Kappes, Heather Barry

AU - Krueger, Lacy E.

AU - Kurtz, Jaime

AU - Levitan, Carmel A.

AU - Mallett, Robyn K.

AU - Morris, Wendy L.

AU - Nelson, Anthony J.

AU - Nier, Jason A.

AU - Packard, Grant

AU - Pilati, Ronaldo

AU - Rutchick, Abraham M.

AU - Schmidt, Kathleen

AU - Skorinko, Jeanine L.

AU - Smith, Robert

AU - Steiner, Troy G.

AU - Storbeck, Justin

AU - Van Swol, Lyn M.

AU - Thompson, Donna

AU - Van 'T Veer, A. E.

AU - Vaughn, Leigh Ann

AU - Vranka, Marek

AU - Wichman, Aaron L.

AU - Woodzicka, Julie A.

AU - Nosek, Brian A.

PY - 2014

Y1 - 2014

N2 - Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of 13 classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, 10 effects replicated consistently. One effect - imagined contact reducing prejudice - showed weak support for replicability. And two effects - flag priming influencing conservatism and currency priming influencing system justification - did not replicate. We compared whether the conditions such as lab versus online or US versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.

AB - Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of 13 classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, 10 effects replicated consistently. One effect - imagined contact reducing prejudice - showed weak support for replicability. And two effects - flag priming influencing conservatism and currency priming influencing system justification - did not replicate. We compared whether the conditions such as lab versus online or US versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.

UR - http://www.scopus.com/inward/record.url?scp=84901716350&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84901716350&partnerID=8YFLogxK

U2 - 10.1027/1864-9335/a000178

DO - 10.1027/1864-9335/a000178

M3 - Article

AN - SCOPUS:84901716350

VL - 45

SP - 142

EP - 152

JO - Social Psychology

JF - Social Psychology

SN - 1864-9335

IS - 3

ER -