Microworkers as research participants: Does underpaying Turkers lead to cognitive dissonance?

Bingjie Liu, S. Shyam Sundar

Research output: Contribution to journalArticle

3 Citations (Scopus)

Abstract

Social science researchers increasingly rely on microworkers to serve as study participants, paying them very little compared to participants recruited from other venues. This has raised ethical concerns and questioned the validity of research based on microworkers. Informed by cognitive dissonance theory, we conducted two between-subjects experiments to examine the effects of underpaying Amazon Mechanical Turk workers (Turkers) on their perceptions and their actual performance on criteria crucial to online social science research. Data show that underpaid Turkers experienced ‘cognitive dissonance’ such that those paid as low as $0.25 rated their participation as more important than those who were paid higher. Perceived importance was associated with other positive perceptions and demand characteristics. Nevertheless, underpaying Turkers increased dropout rate, reduced their level of effort in answering open-ended questions and undermined perceived agency. We discuss the ethical and practical implications of underpaying microworkers.

Original languageEnglish (US)
Pages (from-to)61-69
Number of pages9
JournalComputers in Human Behavior
Volume88
DOIs
StatePublished - Nov 1 2018

Fingerprint

Cognitive Dissonance
Social sciences
Social Sciences
Research
Research Personnel
Experiments
Workers
Turks

All Science Journal Classification (ASJC) codes

  • Arts and Humanities (miscellaneous)
  • Human-Computer Interaction
  • Psychology(all)

Cite this

@article{3e2c0ed479ea441f90c7f5de611de4f5,
title = "Microworkers as research participants: Does underpaying Turkers lead to cognitive dissonance?",
abstract = "Social science researchers increasingly rely on microworkers to serve as study participants, paying them very little compared to participants recruited from other venues. This has raised ethical concerns and questioned the validity of research based on microworkers. Informed by cognitive dissonance theory, we conducted two between-subjects experiments to examine the effects of underpaying Amazon Mechanical Turk workers (Turkers) on their perceptions and their actual performance on criteria crucial to online social science research. Data show that underpaid Turkers experienced ‘cognitive dissonance’ such that those paid as low as $0.25 rated their participation as more important than those who were paid higher. Perceived importance was associated with other positive perceptions and demand characteristics. Nevertheless, underpaying Turkers increased dropout rate, reduced their level of effort in answering open-ended questions and undermined perceived agency. We discuss the ethical and practical implications of underpaying microworkers.",
author = "Bingjie Liu and Sundar, {S. Shyam}",
year = "2018",
month = "11",
day = "1",
doi = "10.1016/j.chb.2018.06.017",
language = "English (US)",
volume = "88",
pages = "61--69",
journal = "Computers in Human Behavior",
issn = "0747-5632",
publisher = "Elsevier Limited",

}

Microworkers as research participants : Does underpaying Turkers lead to cognitive dissonance? / Liu, Bingjie; Sundar, S. Shyam.

In: Computers in Human Behavior, Vol. 88, 01.11.2018, p. 61-69.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Microworkers as research participants

T2 - Does underpaying Turkers lead to cognitive dissonance?

AU - Liu, Bingjie

AU - Sundar, S. Shyam

PY - 2018/11/1

Y1 - 2018/11/1

N2 - Social science researchers increasingly rely on microworkers to serve as study participants, paying them very little compared to participants recruited from other venues. This has raised ethical concerns and questioned the validity of research based on microworkers. Informed by cognitive dissonance theory, we conducted two between-subjects experiments to examine the effects of underpaying Amazon Mechanical Turk workers (Turkers) on their perceptions and their actual performance on criteria crucial to online social science research. Data show that underpaid Turkers experienced ‘cognitive dissonance’ such that those paid as low as $0.25 rated their participation as more important than those who were paid higher. Perceived importance was associated with other positive perceptions and demand characteristics. Nevertheless, underpaying Turkers increased dropout rate, reduced their level of effort in answering open-ended questions and undermined perceived agency. We discuss the ethical and practical implications of underpaying microworkers.

AB - Social science researchers increasingly rely on microworkers to serve as study participants, paying them very little compared to participants recruited from other venues. This has raised ethical concerns and questioned the validity of research based on microworkers. Informed by cognitive dissonance theory, we conducted two between-subjects experiments to examine the effects of underpaying Amazon Mechanical Turk workers (Turkers) on their perceptions and their actual performance on criteria crucial to online social science research. Data show that underpaid Turkers experienced ‘cognitive dissonance’ such that those paid as low as $0.25 rated their participation as more important than those who were paid higher. Perceived importance was associated with other positive perceptions and demand characteristics. Nevertheless, underpaying Turkers increased dropout rate, reduced their level of effort in answering open-ended questions and undermined perceived agency. We discuss the ethical and practical implications of underpaying microworkers.

UR - http://www.scopus.com/inward/record.url?scp=85049904937&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85049904937&partnerID=8YFLogxK

U2 - 10.1016/j.chb.2018.06.017

DO - 10.1016/j.chb.2018.06.017

M3 - Article

AN - SCOPUS:85049904937

VL - 88

SP - 61

EP - 69

JO - Computers in Human Behavior

JF - Computers in Human Behavior

SN - 0747-5632

ER -