Defeating tyranny of the masses in crowdsourcing: Accounting for low-skilled and adversarial workers

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations

Abstract

Crowdsourcing has emerged as a useful learning paradigm which allows us to instantly recruit workers on the web to solve large scale problems, such as quick annotation of image, web page, or document databases. Automated inference engines that fuse the answers or opinions from the crowd to make critical decisions are susceptible to unreliable, low-skilled and malicious workers who tend to mislead the system towards inaccurate inferences. We present a probabilistic generative framework to model worker responses for multicategory crowdsourcing tasks based on two novel paradigms. First, we decompose worker reliability into skill level and intention. Second, we introduce a stochastic model for answer generation that plausibly captures the interplay between worker skills, intentions, and task difficulties. This framework allows us to model and estimate a broad range of worker "types". A generalized Expectation Maximization algorithm is presented to jointly estimate the unknown ground truth answers along with worker and task parameters. As supported experimentally, the proposed scheme de-emphasizes answers from low skilled workers and leverages malicious workers to, in fact, improve crowd aggregation. Moreover, our approach is especially advantageous when there is an (a priori unknown) majority of low-skilled and/or malicious workers in the crowd.

Original languageEnglish (US)
Title of host publicationDecision and Game Theory for Security - 4th International Conference, GameSec 2013, Proceedings
PublisherSpringer Verlag
Pages140-153
Number of pages14
ISBN (Print)9783319027852
DOIs
StatePublished - Jan 1 2013
Event4th International Conference on Decision and Game Theory for Security, GameSec 2013 - Fort Worth, TX, United States
Duration: Nov 11 2013Nov 12 2013

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume8252 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other4th International Conference on Decision and Game Theory for Security, GameSec 2013
CountryUnited States
CityFort Worth, TX
Period11/11/1311/12/13

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint Dive into the research topics of 'Defeating tyranny of the masses in crowdsourcing: Accounting for low-skilled and adversarial workers'. Together they form a unique fingerprint.

  • Cite this

    Kurve, A., Miller, D. J., & Kesidis, G. (2013). Defeating tyranny of the masses in crowdsourcing: Accounting for low-skilled and adversarial workers. In Decision and Game Theory for Security - 4th International Conference, GameSec 2013, Proceedings (pp. 140-153). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8252 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-02786-9_9