Defeating tyranny of the masses in crowdsourcing

Accounting for low-skilled and adversarial workers

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Crowdsourcing has emerged as a useful learning paradigm which allows us to instantly recruit workers on the web to solve large scale problems, such as quick annotation of image, web page, or document databases. Automated inference engines that fuse the answers or opinions from the crowd to make critical decisions are susceptible to unreliable, low-skilled and malicious workers who tend to mislead the system towards inaccurate inferences. We present a probabilistic generative framework to model worker responses for multicategory crowdsourcing tasks based on two novel paradigms. First, we decompose worker reliability into skill level and intention. Second, we introduce a stochastic model for answer generation that plausibly captures the interplay between worker skills, intentions, and task difficulties. This framework allows us to model and estimate a broad range of worker "types". A generalized Expectation Maximization algorithm is presented to jointly estimate the unknown ground truth answers along with worker and task parameters. As supported experimentally, the proposed scheme de-emphasizes answers from low skilled workers and leverages malicious workers to, in fact, improve crowd aggregation. Moreover, our approach is especially advantageous when there is an (a priori unknown) majority of low-skilled and/or malicious workers in the crowd.

Original languageEnglish (US)
Title of host publicationDecision and Game Theory for Security - 4th International Conference, GameSec 2013, Proceedings
PublisherSpringer Verlag
Pages140-153
Number of pages14
ISBN (Print)9783319027852
DOIs
StatePublished - Jan 1 2013
Event4th International Conference on Decision and Game Theory for Security, GameSec 2013 - Fort Worth, TX, United States
Duration: Nov 11 2013Nov 12 2013

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume8252 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other4th International Conference on Decision and Game Theory for Security, GameSec 2013
CountryUnited States
CityFort Worth, TX
Period11/11/1311/12/13

Fingerprint

Inference engines
Electric fuses
Stochastic models
Websites
Agglomeration
Paradigm
Unknown
Inference Engine
Expectation-maximization Algorithm
Large-scale Problems
Inaccurate
Leverage
Estimate
Stochastic Model
Annotation
Aggregation
Tend
Decompose
Model
Range of data

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Kurve, A., Miller, D. J., & Kesidis, G. (2013). Defeating tyranny of the masses in crowdsourcing: Accounting for low-skilled and adversarial workers. In Decision and Game Theory for Security - 4th International Conference, GameSec 2013, Proceedings (pp. 140-153). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8252 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-02786-9_9
Kurve, Aditya ; Miller, David Jonathan ; Kesidis, George. / Defeating tyranny of the masses in crowdsourcing : Accounting for low-skilled and adversarial workers. Decision and Game Theory for Security - 4th International Conference, GameSec 2013, Proceedings. Springer Verlag, 2013. pp. 140-153 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{1c6e8caff64044948daa27d87f00a890,
title = "Defeating tyranny of the masses in crowdsourcing: Accounting for low-skilled and adversarial workers",
abstract = "Crowdsourcing has emerged as a useful learning paradigm which allows us to instantly recruit workers on the web to solve large scale problems, such as quick annotation of image, web page, or document databases. Automated inference engines that fuse the answers or opinions from the crowd to make critical decisions are susceptible to unreliable, low-skilled and malicious workers who tend to mislead the system towards inaccurate inferences. We present a probabilistic generative framework to model worker responses for multicategory crowdsourcing tasks based on two novel paradigms. First, we decompose worker reliability into skill level and intention. Second, we introduce a stochastic model for answer generation that plausibly captures the interplay between worker skills, intentions, and task difficulties. This framework allows us to model and estimate a broad range of worker {"}types{"}. A generalized Expectation Maximization algorithm is presented to jointly estimate the unknown ground truth answers along with worker and task parameters. As supported experimentally, the proposed scheme de-emphasizes answers from low skilled workers and leverages malicious workers to, in fact, improve crowd aggregation. Moreover, our approach is especially advantageous when there is an (a priori unknown) majority of low-skilled and/or malicious workers in the crowd.",
author = "Aditya Kurve and Miller, {David Jonathan} and George Kesidis",
year = "2013",
month = "1",
day = "1",
doi = "10.1007/978-3-319-02786-9_9",
language = "English (US)",
isbn = "9783319027852",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "140--153",
booktitle = "Decision and Game Theory for Security - 4th International Conference, GameSec 2013, Proceedings",
address = "Germany",

}

Kurve, A, Miller, DJ & Kesidis, G 2013, Defeating tyranny of the masses in crowdsourcing: Accounting for low-skilled and adversarial workers. in Decision and Game Theory for Security - 4th International Conference, GameSec 2013, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 8252 LNCS, Springer Verlag, pp. 140-153, 4th International Conference on Decision and Game Theory for Security, GameSec 2013, Fort Worth, TX, United States, 11/11/13. https://doi.org/10.1007/978-3-319-02786-9_9

Defeating tyranny of the masses in crowdsourcing : Accounting for low-skilled and adversarial workers. / Kurve, Aditya; Miller, David Jonathan; Kesidis, George.

Decision and Game Theory for Security - 4th International Conference, GameSec 2013, Proceedings. Springer Verlag, 2013. p. 140-153 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8252 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Defeating tyranny of the masses in crowdsourcing

T2 - Accounting for low-skilled and adversarial workers

AU - Kurve, Aditya

AU - Miller, David Jonathan

AU - Kesidis, George

PY - 2013/1/1

Y1 - 2013/1/1

N2 - Crowdsourcing has emerged as a useful learning paradigm which allows us to instantly recruit workers on the web to solve large scale problems, such as quick annotation of image, web page, or document databases. Automated inference engines that fuse the answers or opinions from the crowd to make critical decisions are susceptible to unreliable, low-skilled and malicious workers who tend to mislead the system towards inaccurate inferences. We present a probabilistic generative framework to model worker responses for multicategory crowdsourcing tasks based on two novel paradigms. First, we decompose worker reliability into skill level and intention. Second, we introduce a stochastic model for answer generation that plausibly captures the interplay between worker skills, intentions, and task difficulties. This framework allows us to model and estimate a broad range of worker "types". A generalized Expectation Maximization algorithm is presented to jointly estimate the unknown ground truth answers along with worker and task parameters. As supported experimentally, the proposed scheme de-emphasizes answers from low skilled workers and leverages malicious workers to, in fact, improve crowd aggregation. Moreover, our approach is especially advantageous when there is an (a priori unknown) majority of low-skilled and/or malicious workers in the crowd.

AB - Crowdsourcing has emerged as a useful learning paradigm which allows us to instantly recruit workers on the web to solve large scale problems, such as quick annotation of image, web page, or document databases. Automated inference engines that fuse the answers or opinions from the crowd to make critical decisions are susceptible to unreliable, low-skilled and malicious workers who tend to mislead the system towards inaccurate inferences. We present a probabilistic generative framework to model worker responses for multicategory crowdsourcing tasks based on two novel paradigms. First, we decompose worker reliability into skill level and intention. Second, we introduce a stochastic model for answer generation that plausibly captures the interplay between worker skills, intentions, and task difficulties. This framework allows us to model and estimate a broad range of worker "types". A generalized Expectation Maximization algorithm is presented to jointly estimate the unknown ground truth answers along with worker and task parameters. As supported experimentally, the proposed scheme de-emphasizes answers from low skilled workers and leverages malicious workers to, in fact, improve crowd aggregation. Moreover, our approach is especially advantageous when there is an (a priori unknown) majority of low-skilled and/or malicious workers in the crowd.

UR - http://www.scopus.com/inward/record.url?scp=84893346449&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84893346449&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-02786-9_9

DO - 10.1007/978-3-319-02786-9_9

M3 - Conference contribution

SN - 9783319027852

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 140

EP - 153

BT - Decision and Game Theory for Security - 4th International Conference, GameSec 2013, Proceedings

PB - Springer Verlag

ER -

Kurve A, Miller DJ, Kesidis G. Defeating tyranny of the masses in crowdsourcing: Accounting for low-skilled and adversarial workers. In Decision and Game Theory for Security - 4th International Conference, GameSec 2013, Proceedings. Springer Verlag. 2013. p. 140-153. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-02786-9_9