Variability in the analysis of a single neuroimaging dataset by many teams

Rotem Botvinik-Nezer; Felix Holzmeister; Colin F. Camerer; Anna Dreber; Juergen Huber; Magnus Johannesson; Michael Kirchler; Roni Iwanir; Jeanette A. Mumford; R. Alison Adcock; Paolo Avesani; Blazej M. Baczkowski; Aahana Bajracharya; Leah Bakst; Sheryl Ball; Marco Barilari; Nadège Bault; Derek Beaton; Julia Beitner; Roland G. Benoit; Ruud M.W.J. Berkers; Jamil P. Bhanji; Bharat B. Biswal; Sebastian Bobadilla-Suarez; Tiago Bortolini; Katherine L. Bottenhorn; Alexander Bowring; Senne Braem; Hayley R. Brooks; Emily G. Brudner; Cristian B. Calderon; Julia A. Camilleri; Jaime J. Castrellon; Luca Cecchetti; Edna C. Cieslik; Zachary J. Cole; Olivier Collignon; Robert W. Cox; William A. Cunningham; Stefan Czoschke; Kamalaker Dadi; Charles P. Davis; Alberto De Luca; Mauricio R. Delgado; Lysia Demetriou; Jeffrey B. Dennison; Xin Di; Erin W. Dickie; Ekaterina Dobryakova; Claire L. Donnat; Juergen Dukart; Niall W. Duncan; Joke Durnez; Amr Eed; Simon B. Eickhoff; Andrew Erhart; Laura Fontanesi; G. Matthew Fricke; Shiguang Fu; Adriana Galván; Remi Gau; Sarah Genon; Tristan Glatard; Enrico Glerean; Jelle J. Goeman; Sergej A.E. Golowin; Carlos González-García; Krzysztof J. Gorgolewski; Cheryl L. Grady; Mikella A. Green; João F. Guassi Moreira; Olivia Guest; Shabnam Hakimi; J. Paul Hamilton; Roeland Hancock; Giacomo Handjaras; Bronson B. Harry; Colin Hawco; Peer Herholz; Gabrielle Herman; Stephan Heunis; Felix Hoffstaedter; Jeremy Hogeveen; Susan Holmes; Chuan Peng Hu; Scott A. Huettel; Matthew E. Hughes; Vittorio Iacovella; Alexandru D. Iordan; Peder M. Isager; Ayse I. Isik; Andrew Jahn; Matthew R. Johnson; Tom Johnstone; Michael J.E. Joseph; Anthony C. Juliano; Joseph W. Kable; Michalis Kassinopoulos; Cemal Koba; Xiang Zhen Kong; Timothy R. Koscik; Nuri Erkut Kucukboyaci; Brice A. Kuhl; Sebastian Kupek; Angela R. Laird; Claus Lamm; Robert Langner; Nina Lauharatanahirun; Hongmi Lee; Sangil Lee; Alexander Leemans; Andrea Leo; Elise Lesage; Flora Li; Monica Y.C. Li; Phui Cheng Lim; Evan N. Lintz; Schuyler W. Liphardt; Annabel B. Losecaat Vermeer; Bradley C. Love; Michael L. Mack; Norberto Malpica; Theo Marins; Camille Maumet; Kelsey McDonald; Joseph T. McGuire; Helena Melero; Adriana S. Méndez Leal; Benjamin Meyer; Kristin N. Meyer; Glad Mihai; Georgios D. Mitsis; Jorge Moll; Dylan M. Nielson; Gustav Nilsonne; Michael P. Notter; Emanuele Olivetti; Adrian I. Onicas; Paolo Papale; Kaustubh R. Patil; Jonathan E. Peelle; Alexandre Pérez; Doris Pischedda; Jean Baptiste Poline; Yanina Prystauka; Shruti Ray; Patricia A. Reuter-Lorenz; Richard C. Reynolds; Emiliano Ricciardi; Jenny R. Rieck; Anais M. Rodriguez-Thompson; Anthony Romyn; Taylor Salo; Gregory R. Samanez-Larkin; Emilio Sanz-Morales; Margaret L. Schlichting; Douglas H. Schultz; Qiang Shen; Margaret A. Sheridan; Jennifer A. Silvers; Kenny Skagerlund; Alec Smith; David V. Smith; Peter Sokol-Hessner; Simon R. Steinkamp; Sarah M. Tashjian; Bertrand Thirion; John N. Thorp; Gustav Tinghög; Loreen Tisdall; Steven H. Tompson; Claudio Toro-Serey; Juan Jesus Torre Tresols; Leonardo Tozzi; Vuong Truong; Luca Turella; Anna E. van ‘t Veer; Tom Verguts; Jean M. Vettel; Sagana Vijayarajah; Khoi Vo; Matthew B. Wall; Wouter D. Weeda; Susanne Weis; David J. White; David Wisniewski; Alba Xifra-Porxas; Emily A. Yearling; Sangsuk Yoon; Rui Yuan; Kenneth S.L. Yuen; Lei Zhang; Xu Zhang; Joshua E. Zosky; Thomas E. Nichols; Russell A. Poldrack; Tom Schonberg

doi:10.1038/s41586-020-2314-9

Variability in the analysis of a single neuroimaging dataset by many teams

Rotem Botvinik-Nezer, Felix Holzmeister, Colin F. Camerer, Anna Dreber, Juergen Huber, Magnus Johannesson, Michael Kirchler, Roni Iwanir, Jeanette A. Mumford, R. Alison Adcock, Paolo Avesani, Blazej M. Baczkowski, Aahana Bajracharya, Leah Bakst, Sheryl Ball, Marco Barilari, Nadège Bault, Derek Beaton, Julia Beitner, Roland G. BenoitRuud M.W.J. Berkers, Jamil P. Bhanji, Bharat B. Biswal, Sebastian Bobadilla-Suarez, Tiago Bortolini, Katherine L. Bottenhorn, Alexander Bowring, Senne Braem, Hayley R. Brooks, Emily G. Brudner, Cristian B. Calderon, Julia A. Camilleri, Jaime J. Castrellon, Luca Cecchetti, Edna C. Cieslik, Zachary J. Cole, Olivier Collignon, Robert W. Cox, William A. Cunningham, Stefan Czoschke, Kamalaker Dadi, Charles P. Davis, Alberto De Luca, Mauricio R. Delgado, Lysia Demetriou, Jeffrey B. Dennison, Xin Di, Erin W. Dickie, Ekaterina Dobryakova, Claire L. Donnat, Juergen Dukart, Niall W. Duncan, Joke Durnez, Amr Eed, Simon B. Eickhoff, Andrew Erhart, Laura Fontanesi, G. Matthew Fricke, Shiguang Fu, Adriana Galván, Remi Gau, Sarah Genon, Tristan Glatard, Enrico Glerean, Jelle J. Goeman, Sergej A.E. Golowin, Carlos González-García, Krzysztof J. Gorgolewski, Cheryl L. Grady, Mikella A. Green, João F. Guassi Moreira, Olivia Guest, Shabnam Hakimi, J. Paul Hamilton, Roeland Hancock, Giacomo Handjaras, Bronson B. Harry, Colin Hawco, Peer Herholz, Gabrielle Herman, Stephan Heunis, Felix Hoffstaedter, Jeremy Hogeveen, Susan Holmes, Chuan Peng Hu, Scott A. Huettel, Matthew E. Hughes, Vittorio Iacovella, Alexandru D. Iordan, Peder M. Isager, Ayse I. Isik, Andrew Jahn, Matthew R. Johnson, Tom Johnstone, Michael J.E. Joseph, Anthony C. Juliano, Joseph W. Kable, Michalis Kassinopoulos, Cemal Koba, Xiang Zhen Kong, Timothy R. Koscik, Nuri Erkut Kucukboyaci, Brice A. Kuhl, Sebastian Kupek, Angela R. Laird, Claus Lamm, Robert Langner, Nina Lauharatanahirun, Hongmi Lee, Sangil Lee, Alexander Leemans, Andrea Leo, Elise Lesage, Flora Li, Monica Y.C. Li, Phui Cheng Lim, Evan N. Lintz, Schuyler W. Liphardt, Annabel B. Losecaat Vermeer, Bradley C. Love, Michael L. Mack, Norberto Malpica, Theo Marins, Camille Maumet, Kelsey McDonald, Joseph T. McGuire, Helena Melero, Adriana S. Méndez Leal, Benjamin Meyer, Kristin N. Meyer, Glad Mihai, Georgios D. Mitsis, Jorge Moll, Dylan M. Nielson, Gustav Nilsonne, Michael P. Notter, Emanuele Olivetti, Adrian I. Onicas, Paolo Papale, Kaustubh R. Patil, Jonathan E. Peelle, Alexandre Pérez, Doris Pischedda, Jean Baptiste Poline, Yanina Prystauka, Shruti Ray, Patricia A. Reuter-Lorenz, Richard C. Reynolds, Emiliano Ricciardi, Jenny R. Rieck, Anais M. Rodriguez-Thompson, Anthony Romyn, Taylor Salo, Gregory R. Samanez-Larkin, Emilio Sanz-Morales, Margaret L. Schlichting, Douglas H. Schultz, Qiang Shen, Margaret A. Sheridan, Jennifer A. Silvers, Kenny Skagerlund, Alec Smith, David V. Smith, Peter Sokol-Hessner, Simon R. Steinkamp, Sarah M. Tashjian, Bertrand Thirion, John N. Thorp, Gustav Tinghög, Loreen Tisdall, Steven H. Tompson, Claudio Toro-Serey, Juan Jesus Torre Tresols, Leonardo Tozzi, Vuong Truong, Luca Turella, Anna E. van ‘t Veer, Tom Verguts, Jean M. Vettel, Sagana Vijayarajah, Khoi Vo, Matthew B. Wall, Wouter D. Weeda, Susanne Weis, David J. White, David Wisniewski, Alba Xifra-Porxas, Emily A. Yearling, Sangsuk Yoon, Rui Yuan, Kenneth S.L. Yuen, Lei Zhang, Xu Zhang, Joshua E. Zosky, Thomas E. Nichols, Russell A. Poldrack, Tom Schonberg

Research output: Contribution to journal › Article › peer-review

360 Scopus citations

Abstract

Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses¹. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset^2–5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.

Original language	English (US)
Pages (from-to)	84-88
Number of pages	5
Journal	Nature
Volume	582
Issue number	7810
DOIs	https://doi.org/10.1038/s41586-020-2314-9
State	Published - Jun 4 2020

All Science Journal Classification (ASJC) codes

General

Access to Document

10.1038/s41586-020-2314-9

Cite this

Botvinik-Nezer, R., Holzmeister, F., Camerer, C. F., Dreber, A., Huber, J., Johannesson, M., Kirchler, M., Iwanir, R., Mumford, J. A., Adcock, R. A., Avesani, P., Baczkowski, B. M., Bajracharya, A., Bakst, L., Ball, S., Barilari, M., Bault, N., Beaton, D., Beitner, J., ... Schonberg, T. (2020). Variability in the analysis of a single neuroimaging dataset by many teams. Nature, 582(7810), 84-88. https://doi.org/10.1038/s41586-020-2314-9

@article{d0fc5941760345b3b5dc2390252a3443,

title = "Variability in the analysis of a single neuroimaging dataset by many teams",

abstract = "Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2–5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.",

author = "Rotem Botvinik-Nezer and Felix Holzmeister and Camerer, {Colin F.} and Anna Dreber and Juergen Huber and Magnus Johannesson and Michael Kirchler and Roni Iwanir and Mumford, {Jeanette A.} and Adcock, {R. Alison} and Paolo Avesani and Baczkowski, {Blazej M.} and Aahana Bajracharya and Leah Bakst and Sheryl Ball and Marco Barilari and Nad{\`e}ge Bault and Derek Beaton and Julia Beitner and Benoit, {Roland G.} and Berkers, {Ruud M.W.J.} and Bhanji, {Jamil P.} and Biswal, {Bharat B.} and Sebastian Bobadilla-Suarez and Tiago Bortolini and Bottenhorn, {Katherine L.} and Alexander Bowring and Senne Braem and Brooks, {Hayley R.} and Brudner, {Emily G.} and Calderon, {Cristian B.} and Camilleri, {Julia A.} and Castrellon, {Jaime J.} and Luca Cecchetti and Cieslik, {Edna C.} and Cole, {Zachary J.} and Olivier Collignon and Cox, {Robert W.} and Cunningham, {William A.} and Stefan Czoschke and Kamalaker Dadi and Davis, {Charles P.} and Luca, {Alberto De} and Delgado, {Mauricio R.} and Lysia Demetriou and Dennison, {Jeffrey B.} and Xin Di and Dickie, {Erin W.} and Ekaterina Dobryakova and Donnat, {Claire L.} and Juergen Dukart and Duncan, {Niall W.} and Joke Durnez and Amr Eed and Eickhoff, {Simon B.} and Andrew Erhart and Laura Fontanesi and Fricke, {G. Matthew} and Shiguang Fu and Adriana Galv{\'a}n and Remi Gau and Sarah Genon and Tristan Glatard and Enrico Glerean and Goeman, {Jelle J.} and Golowin, {Sergej A.E.} and Carlos Gonz{\'a}lez-Garc{\'i}a and Gorgolewski, {Krzysztof J.} and Grady, {Cheryl L.} and Green, {Mikella A.} and {Guassi Moreira}, {Jo{\~a}o F.} and Olivia Guest and Shabnam Hakimi and Hamilton, {J. Paul} and Roeland Hancock and Giacomo Handjaras and Harry, {Bronson B.} and Colin Hawco and Peer Herholz and Gabrielle Herman and Stephan Heunis and Felix Hoffstaedter and Jeremy Hogeveen and Susan Holmes and Hu, {Chuan Peng} and Huettel, {Scott A.} and Hughes, {Matthew E.} and Vittorio Iacovella and Iordan, {Alexandru D.} and Isager, {Peder M.} and Isik, {Ayse I.} and Andrew Jahn and Johnson, {Matthew R.} and Tom Johnstone and Joseph, {Michael J.E.} and Juliano, {Anthony C.} and Kable, {Joseph W.} and Michalis Kassinopoulos and Cemal Koba and Kong, {Xiang Zhen} and Koscik, {Timothy R.} and Kucukboyaci, {Nuri Erkut} and Kuhl, {Brice A.} and Sebastian Kupek and Laird, {Angela R.} and Claus Lamm and Robert Langner and Nina Lauharatanahirun and Hongmi Lee and Sangil Lee and Alexander Leemans and Andrea Leo and Elise Lesage and Flora Li and Li, {Monica Y.C.} and Lim, {Phui Cheng} and Lintz, {Evan N.} and Liphardt, {Schuyler W.} and {Losecaat Vermeer}, {Annabel B.} and Love, {Bradley C.} and Mack, {Michael L.} and Norberto Malpica and Theo Marins and Camille Maumet and Kelsey McDonald and McGuire, {Joseph T.} and Helena Melero and {M{\'e}ndez Leal}, {Adriana S.} and Benjamin Meyer and Meyer, {Kristin N.} and Glad Mihai and Mitsis, {Georgios D.} and Jorge Moll and Nielson, {Dylan M.} and Gustav Nilsonne and Notter, {Michael P.} and Emanuele Olivetti and Onicas, {Adrian I.} and Paolo Papale and Patil, {Kaustubh R.} and Peelle, {Jonathan E.} and Alexandre P{\'e}rez and Doris Pischedda and Poline, {Jean Baptiste} and Yanina Prystauka and Shruti Ray and Reuter-Lorenz, {Patricia A.} and Reynolds, {Richard C.} and Emiliano Ricciardi and Rieck, {Jenny R.} and Rodriguez-Thompson, {Anais M.} and Anthony Romyn and Taylor Salo and Samanez-Larkin, {Gregory R.} and Emilio Sanz-Morales and Schlichting, {Margaret L.} and Schultz, {Douglas H.} and Qiang Shen and Sheridan, {Margaret A.} and Silvers, {Jennifer A.} and Kenny Skagerlund and Alec Smith and Smith, {David V.} and Peter Sokol-Hessner and Steinkamp, {Simon R.} and Tashjian, {Sarah M.} and Bertrand Thirion and Thorp, {John N.} and Gustav Tingh{\"o}g and Loreen Tisdall and Tompson, {Steven H.} and Claudio Toro-Serey and {Torre Tresols}, {Juan Jesus} and Leonardo Tozzi and Vuong Truong and Luca Turella and {van {\textquoteleft}t Veer}, {Anna E.} and Tom Verguts and Vettel, {Jean M.} and Sagana Vijayarajah and Khoi Vo and Wall, {Matthew B.} and Weeda, {Wouter D.} and Susanne Weis and White, {David J.} and David Wisniewski and Alba Xifra-Porxas and Yearling, {Emily A.} and Sangsuk Yoon and Rui Yuan and Yuen, {Kenneth S.L.} and Lei Zhang and Xu Zhang and Zosky, {Joshua E.} and Nichols, {Thomas E.} and Poldrack, {Russell A.} and Tom Schonberg",

note = "Funding Information: Acknowledgements Neuroimaging data collection, performed at Tel Aviv University, was supported by the Austrian Science Fund (P29362-G27), the Israel Science Foundation (ISF 2004/15 to T. Schonberg) and the Swedish Foundation for Humanities and Social Sciences (NHS14-1719:1). Hosting of the data on OpenNeuro was supported by a National Institutes of Health (NIH) grant (R24MH117179). We thank M. C. Frank, Y. Assaf and N. Daw for comments on an earlier draft; the Texas Advanced Computing Center for providing computing resources for preprocessing of the data; the Stanford Research Computing Facility for hosting the data; and D. Roll for assisting with data processing. T. Schonberg thanks The Alfredo Federico Strauss Center for Computational Neuroimaging at Tel Aviv University; A.D. thanks the Knut and Alice Wallenberg Foundation and the Marianne and Marcus Wallenberg Foundation (A.D. is a Wallenberg Scholar), the Austrian Science Fund (FWF, SFB F63) and the Jan Wallander and Tom Hedelius Foundation (Svenska Handelsbankens Forskningsstiftelser); F. Holzmeister, J. Huber and M. Kirchler thank the Austrian Science Fund (FWF, SFB F63); D.W. was supported by the Research Foundation Flanders (FWO) and the European Union{\textquoteright}s Horizon 2020 research and innovation programme (https://ec.europa.eu/programmes/horizon2020/en) under the Marie Sk{\l}odowska-Curie grant agreement no. 665501; L. Tisdall was supported by the University of Basel Research Fund for Junior Researchers; C.B.C. was supported by grant 12O7719N from the Research Foundation Flanders; E.L. was supported by grant 12T2517N from the Research Foundation Flanders and Marie Sk{\l}odowska-Curie Actions under COFUND grant agreement 665501; A. Eed was supported by a predoctoral fellowship La Caixa-Severo Ochoa from Obra Social La Caixa and also acknowledges Comunidad de C{\'a}lculo Cient{\'i}fico del CSIC for the high-performance computing (HPC) use; C.L. was supported by the Vienna Science and Technology Fund (WWTF VRG13-007) and Austrian Science Fund (FWF P 32686); A.B.L.V. was supported by the Vienna Science and Technology Fund (WWTF VRG13-007); L.Z. was supported by the Vienna Science and Technology Fund (WWTF VRG13-007), the National Natural Science Foundation of China (no. 71801110), MOE (Ministry of Education in China) Project of Humanities and Social Sciences (no. 18YJC630268) and China Postdoctoral Science Foundation (no. 2018M633270); D.P. is currently supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany{\textquoteright}s Excellence Strategy {\textquoteleft}Science of Intelligence{\textquoteright} (EXC 2002/1, project number 390523135); P.H. was supported in part by funding provided by Brain Canada, in partnership with Health Canada, for the Canadian Open Neuroscience Platform initiative; J.-B.P. was partially funded by the NIH (NIH-NIBIB P41 EB019936 (ReproNim), NIH-NIMH R01 MH083320 (CANDIShare) and NIH RF1 MH120021 (NIDM)) and the National Institute Of Mental Health of the NIH under award number R01MH096906 (Neurosynth), as well as the Canada First Research Excellence Fund, awarded to McGill University for the Healthy Brains for Healthy Lives initiative and the Brain Canada Foundation with support from Health Canada; S.B.E. was supported by the European Union{\textquoteright}s Horizon 2020 Research and Innovation Programme under grant agreement no. 785907 (HBP SGA2); G.M. was supported by the Max Planck Society; S. Heunis has received funding from the Dutch foundation LSH-TKI (grant LSHM16053-SGF); J.F.G.M. was supported by a Graduate Research Fellowship from the NSF and T32 Predoctoral Fellowship from the NIH; B.M. was supported by the Deutsche Forschungsgemeinschaft (grant CRC1193, subproject B01); A.R.L. was supported by NSF 1631325 and NIH R01 DA041353; M.E.H., T.J. and D.J.W. were supported by the Australian National Imaging Facility, a National Collaborative Research Infrastructure Strategy (NCRIS) capability; P.M.I. was supported by VIDI grant 452-17-013 from the Netherlands Organisation for Scientific Research; B.M.B. was supported by the Max Planck Society; J.P.H. was supported by a grant from the Swedish Research Council; R.W.C. and R.C.R. were supported by NIH IRP project number ZICMH002888; D.M.N., R.W.C., and R.C.R. used the computational resources of the National Institutes of Health High Performance Computing Biowulf cluster (http://hpc.nih.gov); D.M.N. was supported by NIH IRP project number ZICMH002960; C.F.C. was supported by the Tianqiao and Chrissy Center for Social and Decision Neuroscience Center Leadership Chair; R.G.B. was supported by the Max Planck Society; R.M.W.J.B. was supported by the Max Planck Society; M.B., O.C. and R.G. were supported by the Belgian Excellence of Science program (EOS project 30991544) from the FNRS-Belgium; O.C. is a research associate at the FRS-FNRS of Belgium; A.D.L. was supported by grant R4195 “Repimpact” of EraNET Neuron; Q.S. was funded by grant no. 71971199,71602175 and 71942004 from the National Natural Science Foundation of China and no. 16YJC630103 of the Ministry of Education of Humanities and Social Science; and T.E.N. was supported by the Wellcome Trust award 100309/Z/12/Z. Publisher Copyright: {\textcopyright} 2020, The Author(s), under exclusive licence to Springer Nature Limited.",

year = "2020",

month = jun,

day = "4",

doi = "10.1038/s41586-020-2314-9",

language = "English (US)",

volume = "582",

pages = "84--88",

journal = "Nature",

issn = "0028-0836",

publisher = "Nature Publishing Group",

number = "7810",

}

Botvinik-Nezer, R, Holzmeister, F, Camerer, CF, Dreber, A, Huber, J, Johannesson, M, Kirchler, M, Iwanir, R, Mumford, JA, Adcock, RA, Avesani, P, Baczkowski, BM, Bajracharya, A, Bakst, L, Ball, S, Barilari, M, Bault, N, Beaton, D, Beitner, J, Benoit, RG, Berkers, RMWJ, Bhanji, JP, Biswal, BB, Bobadilla-Suarez, S, Bortolini, T, Bottenhorn, KL, Bowring, A, Braem, S, Brooks, HR, Brudner, EG, Calderon, CB, Camilleri, JA, Castrellon, JJ, Cecchetti, L, Cieslik, EC, Cole, ZJ, Collignon, O, Cox, RW, Cunningham, WA, Czoschke, S, Dadi, K, Davis, CP, Luca, AD, Delgado, MR, Demetriou, L, Dennison, JB, Di, X, Dickie, EW, Dobryakova, E, Donnat, CL, Dukart, J, Duncan, NW, Durnez, J, Eed, A, Eickhoff, SB, Erhart, A, Fontanesi, L, Fricke, GM, Fu, S, Galván, A, Gau, R, Genon, S, Glatard, T, Glerean, E, Goeman, JJ, Golowin, SAE, González-García, C, Gorgolewski, KJ, Grady, CL, Green, MA, Guassi Moreira, JF, Guest, O, Hakimi, S, Hamilton, JP, Hancock, R, Handjaras, G, Harry, BB, Hawco, C, Herholz, P, Herman, G, Heunis, S, Hoffstaedter, F, Hogeveen, J, Holmes, S, Hu, CP, Huettel, SA, Hughes, ME, Iacovella, V, Iordan, AD, Isager, PM, Isik, AI, Jahn, A, Johnson, MR, Johnstone, T, Joseph, MJE, Juliano, AC, Kable, JW, Kassinopoulos, M, Koba, C, Kong, XZ, Koscik, TR, Kucukboyaci, NE, Kuhl, BA, Kupek, S, Laird, AR, Lamm, C, Langner, R, Lauharatanahirun, N, Lee, H, Lee, S, Leemans, A, Leo, A, Lesage, E, Li, F, Li, MYC, Lim, PC, Lintz, EN, Liphardt, SW, Losecaat Vermeer, AB, Love, BC, Mack, ML, Malpica, N, Marins, T, Maumet, C, McDonald, K, McGuire, JT, Melero, H, Méndez Leal, AS, Meyer, B, Meyer, KN, Mihai, G, Mitsis, GD, Moll, J, Nielson, DM, Nilsonne, G, Notter, MP, Olivetti, E, Onicas, AI, Papale, P, Patil, KR, Peelle, JE, Pérez, A, Pischedda, D, Poline, JB, Prystauka, Y, Ray, S, Reuter-Lorenz, PA, Reynolds, RC, Ricciardi, E, Rieck, JR, Rodriguez-Thompson, AM, Romyn, A, Salo, T, Samanez-Larkin, GR, Sanz-Morales, E, Schlichting, ML, Schultz, DH, Shen, Q, Sheridan, MA, Silvers, JA, Skagerlund, K, Smith, A, Smith, DV, Sokol-Hessner, P, Steinkamp, SR, Tashjian, SM, Thirion, B, Thorp, JN, Tinghög, G, Tisdall, L, Tompson, SH, Toro-Serey, C, Torre Tresols, JJ, Tozzi, L, Truong, V, Turella, L, van ‘t Veer, AE, Verguts, T, Vettel, JM, Vijayarajah, S, Vo, K, Wall, MB, Weeda, WD, Weis, S, White, DJ, Wisniewski, D, Xifra-Porxas, A, Yearling, EA, Yoon, S, Yuan, R, Yuen, KSL, Zhang, L, Zhang, X, Zosky, JE, Nichols, TE, Poldrack, RA & Schonberg, T 2020, 'Variability in the analysis of a single neuroimaging dataset by many teams', Nature, vol. 582, no. 7810, pp. 84-88. https://doi.org/10.1038/s41586-020-2314-9

TY - JOUR

T1 - Variability in the analysis of a single neuroimaging dataset by many teams

AU - Botvinik-Nezer, Rotem

AU - Holzmeister, Felix

AU - Camerer, Colin F.

AU - Dreber, Anna

AU - Huber, Juergen

AU - Johannesson, Magnus

AU - Kirchler, Michael

AU - Iwanir, Roni

AU - Mumford, Jeanette A.

AU - Adcock, R. Alison

AU - Avesani, Paolo

AU - Baczkowski, Blazej M.

AU - Bajracharya, Aahana

AU - Bakst, Leah

AU - Ball, Sheryl

AU - Barilari, Marco

AU - Bault, Nadège

AU - Beaton, Derek

AU - Beitner, Julia

AU - Benoit, Roland G.

AU - Berkers, Ruud M.W.J.

AU - Bhanji, Jamil P.

AU - Biswal, Bharat B.

AU - Bobadilla-Suarez, Sebastian

AU - Bortolini, Tiago

AU - Bottenhorn, Katherine L.

AU - Bowring, Alexander

AU - Braem, Senne

AU - Brooks, Hayley R.

AU - Brudner, Emily G.

AU - Calderon, Cristian B.

AU - Camilleri, Julia A.

AU - Castrellon, Jaime J.

AU - Cecchetti, Luca

AU - Cieslik, Edna C.

AU - Cole, Zachary J.

AU - Collignon, Olivier

AU - Cox, Robert W.

AU - Cunningham, William A.

AU - Czoschke, Stefan

AU - Dadi, Kamalaker

AU - Davis, Charles P.

AU - Luca, Alberto De

AU - Delgado, Mauricio R.

AU - Demetriou, Lysia

AU - Dennison, Jeffrey B.

AU - Di, Xin

AU - Dickie, Erin W.

AU - Dobryakova, Ekaterina

AU - Donnat, Claire L.

AU - Dukart, Juergen

AU - Duncan, Niall W.

AU - Durnez, Joke

AU - Eed, Amr

AU - Eickhoff, Simon B.

AU - Erhart, Andrew

AU - Fontanesi, Laura

AU - Fricke, G. Matthew

AU - Fu, Shiguang

AU - Galván, Adriana

AU - Gau, Remi

AU - Genon, Sarah

AU - Glatard, Tristan

AU - Glerean, Enrico

AU - Goeman, Jelle J.

AU - Golowin, Sergej A.E.

AU - González-García, Carlos

AU - Gorgolewski, Krzysztof J.

AU - Grady, Cheryl L.

AU - Green, Mikella A.

AU - Guassi Moreira, João F.

AU - Guest, Olivia

AU - Hakimi, Shabnam

AU - Hamilton, J. Paul

AU - Hancock, Roeland

AU - Handjaras, Giacomo

AU - Harry, Bronson B.

AU - Hawco, Colin

AU - Herholz, Peer

AU - Herman, Gabrielle

AU - Heunis, Stephan

AU - Hoffstaedter, Felix

AU - Hogeveen, Jeremy

AU - Holmes, Susan

AU - Hu, Chuan Peng

AU - Huettel, Scott A.

AU - Hughes, Matthew E.

AU - Iacovella, Vittorio

AU - Iordan, Alexandru D.

AU - Isager, Peder M.

AU - Isik, Ayse I.

AU - Jahn, Andrew

AU - Johnson, Matthew R.

AU - Johnstone, Tom

AU - Joseph, Michael J.E.

AU - Juliano, Anthony C.

AU - Kable, Joseph W.

AU - Kassinopoulos, Michalis

AU - Koba, Cemal

AU - Kong, Xiang Zhen

AU - Koscik, Timothy R.

AU - Kucukboyaci, Nuri Erkut

AU - Kuhl, Brice A.

AU - Kupek, Sebastian

AU - Laird, Angela R.

AU - Lamm, Claus

AU - Langner, Robert

AU - Lauharatanahirun, Nina

AU - Lee, Hongmi

AU - Lee, Sangil

AU - Leemans, Alexander

AU - Leo, Andrea

AU - Lesage, Elise

AU - Li, Flora

AU - Li, Monica Y.C.

AU - Lim, Phui Cheng

AU - Lintz, Evan N.

AU - Liphardt, Schuyler W.

AU - Losecaat Vermeer, Annabel B.

AU - Love, Bradley C.

AU - Mack, Michael L.

AU - Malpica, Norberto

AU - Marins, Theo

AU - Maumet, Camille

AU - McDonald, Kelsey

AU - McGuire, Joseph T.

AU - Melero, Helena

AU - Méndez Leal, Adriana S.

AU - Meyer, Benjamin

AU - Meyer, Kristin N.

AU - Mihai, Glad

AU - Mitsis, Georgios D.

AU - Moll, Jorge

AU - Nielson, Dylan M.

AU - Nilsonne, Gustav

AU - Notter, Michael P.

AU - Olivetti, Emanuele

AU - Onicas, Adrian I.

AU - Papale, Paolo

AU - Patil, Kaustubh R.

AU - Peelle, Jonathan E.

AU - Pérez, Alexandre

AU - Pischedda, Doris

AU - Poline, Jean Baptiste

AU - Prystauka, Yanina

AU - Ray, Shruti

AU - Reuter-Lorenz, Patricia A.

AU - Reynolds, Richard C.

AU - Ricciardi, Emiliano

AU - Rieck, Jenny R.

AU - Rodriguez-Thompson, Anais M.

AU - Romyn, Anthony

AU - Salo, Taylor

AU - Samanez-Larkin, Gregory R.

AU - Sanz-Morales, Emilio

AU - Schlichting, Margaret L.

AU - Schultz, Douglas H.

AU - Shen, Qiang

AU - Sheridan, Margaret A.

AU - Silvers, Jennifer A.

AU - Skagerlund, Kenny

AU - Smith, Alec

AU - Smith, David V.

AU - Sokol-Hessner, Peter

AU - Steinkamp, Simon R.

AU - Tashjian, Sarah M.

AU - Thirion, Bertrand

AU - Thorp, John N.

AU - Tinghög, Gustav

AU - Tisdall, Loreen

AU - Tompson, Steven H.

AU - Toro-Serey, Claudio

AU - Torre Tresols, Juan Jesus

AU - Tozzi, Leonardo

AU - Truong, Vuong

AU - Turella, Luca

AU - van ‘t Veer, Anna E.

AU - Verguts, Tom

AU - Vettel, Jean M.

AU - Vijayarajah, Sagana

AU - Vo, Khoi

AU - Wall, Matthew B.

AU - Weeda, Wouter D.

AU - Weis, Susanne

AU - White, David J.

AU - Wisniewski, David

AU - Xifra-Porxas, Alba

AU - Yearling, Emily A.

AU - Yoon, Sangsuk

AU - Yuan, Rui

AU - Yuen, Kenneth S.L.

AU - Zhang, Lei

AU - Zhang, Xu

AU - Zosky, Joshua E.

AU - Nichols, Thomas E.

AU - Poldrack, Russell A.

AU - Schonberg, Tom

N1 - Funding Information: Acknowledgements Neuroimaging data collection, performed at Tel Aviv University, was supported by the Austrian Science Fund (P29362-G27), the Israel Science Foundation (ISF 2004/15 to T. Schonberg) and the Swedish Foundation for Humanities and Social Sciences (NHS14-1719:1). Hosting of the data on OpenNeuro was supported by a National Institutes of Health (NIH) grant (R24MH117179). We thank M. C. Frank, Y. Assaf and N. Daw for comments on an earlier draft; the Texas Advanced Computing Center for providing computing resources for preprocessing of the data; the Stanford Research Computing Facility for hosting the data; and D. Roll for assisting with data processing. T. Schonberg thanks The Alfredo Federico Strauss Center for Computational Neuroimaging at Tel Aviv University; A.D. thanks the Knut and Alice Wallenberg Foundation and the Marianne and Marcus Wallenberg Foundation (A.D. is a Wallenberg Scholar), the Austrian Science Fund (FWF, SFB F63) and the Jan Wallander and Tom Hedelius Foundation (Svenska Handelsbankens Forskningsstiftelser); F. Holzmeister, J. Huber and M. Kirchler thank the Austrian Science Fund (FWF, SFB F63); D.W. was supported by the Research Foundation Flanders (FWO) and the European Union’s Horizon 2020 research and innovation programme (https://ec.europa.eu/programmes/horizon2020/en) under the Marie Skłodowska-Curie grant agreement no. 665501; L. Tisdall was supported by the University of Basel Research Fund for Junior Researchers; C.B.C. was supported by grant 12O7719N from the Research Foundation Flanders; E.L. was supported by grant 12T2517N from the Research Foundation Flanders and Marie Skłodowska-Curie Actions under COFUND grant agreement 665501; A. Eed was supported by a predoctoral fellowship La Caixa-Severo Ochoa from Obra Social La Caixa and also acknowledges Comunidad de Cálculo Científico del CSIC for the high-performance computing (HPC) use; C.L. was supported by the Vienna Science and Technology Fund (WWTF VRG13-007) and Austrian Science Fund (FWF P 32686); A.B.L.V. was supported by the Vienna Science and Technology Fund (WWTF VRG13-007); L.Z. was supported by the Vienna Science and Technology Fund (WWTF VRG13-007), the National Natural Science Foundation of China (no. 71801110), MOE (Ministry of Education in China) Project of Humanities and Social Sciences (no. 18YJC630268) and China Postdoctoral Science Foundation (no. 2018M633270); D.P. is currently supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy ‘Science of Intelligence’ (EXC 2002/1, project number 390523135); P.H. was supported in part by funding provided by Brain Canada, in partnership with Health Canada, for the Canadian Open Neuroscience Platform initiative; J.-B.P. was partially funded by the NIH (NIH-NIBIB P41 EB019936 (ReproNim), NIH-NIMH R01 MH083320 (CANDIShare) and NIH RF1 MH120021 (NIDM)) and the National Institute Of Mental Health of the NIH under award number R01MH096906 (Neurosynth), as well as the Canada First Research Excellence Fund, awarded to McGill University for the Healthy Brains for Healthy Lives initiative and the Brain Canada Foundation with support from Health Canada; S.B.E. was supported by the European Union’s Horizon 2020 Research and Innovation Programme under grant agreement no. 785907 (HBP SGA2); G.M. was supported by the Max Planck Society; S. Heunis has received funding from the Dutch foundation LSH-TKI (grant LSHM16053-SGF); J.F.G.M. was supported by a Graduate Research Fellowship from the NSF and T32 Predoctoral Fellowship from the NIH; B.M. was supported by the Deutsche Forschungsgemeinschaft (grant CRC1193, subproject B01); A.R.L. was supported by NSF 1631325 and NIH R01 DA041353; M.E.H., T.J. and D.J.W. were supported by the Australian National Imaging Facility, a National Collaborative Research Infrastructure Strategy (NCRIS) capability; P.M.I. was supported by VIDI grant 452-17-013 from the Netherlands Organisation for Scientific Research; B.M.B. was supported by the Max Planck Society; J.P.H. was supported by a grant from the Swedish Research Council; R.W.C. and R.C.R. were supported by NIH IRP project number ZICMH002888; D.M.N., R.W.C., and R.C.R. used the computational resources of the National Institutes of Health High Performance Computing Biowulf cluster (http://hpc.nih.gov); D.M.N. was supported by NIH IRP project number ZICMH002960; C.F.C. was supported by the Tianqiao and Chrissy Center for Social and Decision Neuroscience Center Leadership Chair; R.G.B. was supported by the Max Planck Society; R.M.W.J.B. was supported by the Max Planck Society; M.B., O.C. and R.G. were supported by the Belgian Excellence of Science program (EOS project 30991544) from the FNRS-Belgium; O.C. is a research associate at the FRS-FNRS of Belgium; A.D.L. was supported by grant R4195 “Repimpact” of EraNET Neuron; Q.S. was funded by grant no. 71971199,71602175 and 71942004 from the National Natural Science Foundation of China and no. 16YJC630103 of the Ministry of Education of Humanities and Social Science; and T.E.N. was supported by the Wellcome Trust award 100309/Z/12/Z. Publisher Copyright: © 2020, The Author(s), under exclusive licence to Springer Nature Limited.

PY - 2020/6/4

Y1 - 2020/6/4

N2 - Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2–5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.

AB - Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2–5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.

UR - http://www.scopus.com/inward/record.url?scp=85085279867&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85085279867&partnerID=8YFLogxK

U2 - 10.1038/s41586-020-2314-9

DO - 10.1038/s41586-020-2314-9

M3 - Article

C2 - 32483374

AN - SCOPUS:85085279867

SN - 0028-0836

VL - 582

SP - 84

EP - 88

JO - Nature

JF - Nature

IS - 7810

ER -

Variability in the analysis of a single neuroimaging dataset by many teams

Abstract

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this