The Limited Value of Non-Replicable Field Experiments in Contexts With Low Temporal Validity

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

My first publication as a grad student was a field experiment using Twitter “bots” to socially sanction users engaged in racist harassment. The ascendant paradigm in quantitative social science emphasizes the need for research to be “internally valid,” with a preference for randomized control trials like the one I conducted. That research project was well recseived, both by the political science discipline and the public, but I no longer believe that one-off field experiments are as valuable a tool for studying online behavior as I once did. The problem is that the knowledge they generate decays too rapidly (alternatively, that the realms in which it can be applied are too few) because the object of study changes too rapidly. I have been developing the concept of “temporal validity” as form of “external validity” that is particularly relevant to the study of social media. I suggest two avenues for producing more temporally valid research: (1) faster, more transparent publication (adapting the CS model of conference proceedings); (2) a “hollowing out” of empirical research, replacing medium-scale experimentation like my own work with either purely descriptive work (ethnographic or large-scale) or with massive, collaborative, replicable experimentation.

Original languageEnglish (US)
JournalSocial Media and Society
Volume5
Issue number3
DOIs
StatePublished - Apr 1 2019

Fingerprint

twitter
experiment
social media
sanction
political science
empirical research
research project
social science
Experiments
paradigm
Social sciences
student
Students

All Science Journal Classification (ASJC) codes

  • Cultural Studies
  • Communication
  • Computer Science Applications

Cite this

@article{b07dbdec67904aefad5119f3fdd2b6f0,
title = "The Limited Value of Non-Replicable Field Experiments in Contexts With Low Temporal Validity",
abstract = "My first publication as a grad student was a field experiment using Twitter “bots” to socially sanction users engaged in racist harassment. The ascendant paradigm in quantitative social science emphasizes the need for research to be “internally valid,” with a preference for randomized control trials like the one I conducted. That research project was well recseived, both by the political science discipline and the public, but I no longer believe that one-off field experiments are as valuable a tool for studying online behavior as I once did. The problem is that the knowledge they generate decays too rapidly (alternatively, that the realms in which it can be applied are too few) because the object of study changes too rapidly. I have been developing the concept of “temporal validity” as form of “external validity” that is particularly relevant to the study of social media. I suggest two avenues for producing more temporally valid research: (1) faster, more transparent publication (adapting the CS model of conference proceedings); (2) a “hollowing out” of empirical research, replacing medium-scale experimentation like my own work with either purely descriptive work (ethnographic or large-scale) or with massive, collaborative, replicable experimentation.",
author = "Kevin Munger",
year = "2019",
month = "4",
day = "1",
doi = "10.1177/2056305119859294",
language = "English (US)",
volume = "5",
journal = "Social Media and Society",
issn = "2056-3051",
publisher = "SAGE Publications Ltd",
number = "3",

}

The Limited Value of Non-Replicable Field Experiments in Contexts With Low Temporal Validity. / Munger, Kevin.

In: Social Media and Society, Vol. 5, No. 3, 01.04.2019.

Research output: Contribution to journalArticle

TY - JOUR

T1 - The Limited Value of Non-Replicable Field Experiments in Contexts With Low Temporal Validity

AU - Munger, Kevin

PY - 2019/4/1

Y1 - 2019/4/1

N2 - My first publication as a grad student was a field experiment using Twitter “bots” to socially sanction users engaged in racist harassment. The ascendant paradigm in quantitative social science emphasizes the need for research to be “internally valid,” with a preference for randomized control trials like the one I conducted. That research project was well recseived, both by the political science discipline and the public, but I no longer believe that one-off field experiments are as valuable a tool for studying online behavior as I once did. The problem is that the knowledge they generate decays too rapidly (alternatively, that the realms in which it can be applied are too few) because the object of study changes too rapidly. I have been developing the concept of “temporal validity” as form of “external validity” that is particularly relevant to the study of social media. I suggest two avenues for producing more temporally valid research: (1) faster, more transparent publication (adapting the CS model of conference proceedings); (2) a “hollowing out” of empirical research, replacing medium-scale experimentation like my own work with either purely descriptive work (ethnographic or large-scale) or with massive, collaborative, replicable experimentation.

AB - My first publication as a grad student was a field experiment using Twitter “bots” to socially sanction users engaged in racist harassment. The ascendant paradigm in quantitative social science emphasizes the need for research to be “internally valid,” with a preference for randomized control trials like the one I conducted. That research project was well recseived, both by the political science discipline and the public, but I no longer believe that one-off field experiments are as valuable a tool for studying online behavior as I once did. The problem is that the knowledge they generate decays too rapidly (alternatively, that the realms in which it can be applied are too few) because the object of study changes too rapidly. I have been developing the concept of “temporal validity” as form of “external validity” that is particularly relevant to the study of social media. I suggest two avenues for producing more temporally valid research: (1) faster, more transparent publication (adapting the CS model of conference proceedings); (2) a “hollowing out” of empirical research, replacing medium-scale experimentation like my own work with either purely descriptive work (ethnographic or large-scale) or with massive, collaborative, replicable experimentation.

UR - http://www.scopus.com/inward/record.url?scp=85068965821&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85068965821&partnerID=8YFLogxK

U2 - 10.1177/2056305119859294

DO - 10.1177/2056305119859294

M3 - Article

AN - SCOPUS:85068965821

VL - 5

JO - Social Media and Society

JF - Social Media and Society

SN - 2056-3051

IS - 3

ER -