CrowdSelect: Increasing accuracy of crowdsourcing tasks through behavior prediction and user selection

Chenxi Qiu, Anna C. Squicciarini, Barbara Carminati, James Caverlee, Dev Rishi Khare

Research output: Chapter in Book/Report/Conference proceedingConference contribution

9 Scopus citations

Abstract

Crowdsourcing allows many people to complete tasks of various difficulty with minimal recruitment and administration costs. However, the lack of participant accountability may entice people to complete as many tasks as possible without fully engaging in them, jeopardizing the quality of responses. In this paper, we present a dynamic and time efficient solution to the task assignment problem in crowdsourcing platforms. Our proposed approach, CrowdSelect, offers a theoretically proven algorithm to assign workers to tasks in a cost efficient manner, while ensuring high accuracy of the overall task. In contrast to existing works, our approach makes minimal assumptions on the probability of error for workers, and completely removes the assumptions that such probability is known apriori and that it remains consistent over time. Through experiments over real Amazon Mechanical Turk traces and synthetic data, we find that CrowdS-elect has a significant gain in term of accuracy compared to state-of-the-art algorithms, and can provide a 17.5% gain in answers' accuracy compared to previous methods, even when there are over 50% malicious workers.

Original languageEnglish (US)
Title of host publicationCIKM 2016 - Proceedings of the 2016 ACM Conference on Information and Knowledge Management
PublisherAssociation for Computing Machinery
Pages539-548
Number of pages10
ISBN (Electronic)9781450340731
DOIs
StatePublished - Oct 24 2016
Event25th ACM International Conference on Information and Knowledge Management, CIKM 2016 - Indianapolis, United States
Duration: Oct 24 2016Oct 28 2016

Publication series

NameInternational Conference on Information and Knowledge Management, Proceedings
Volume24-28-October-2016

Other

Other25th ACM International Conference on Information and Knowledge Management, CIKM 2016
CountryUnited States
CityIndianapolis
Period10/24/1610/28/16

All Science Journal Classification (ASJC) codes

  • Business, Management and Accounting(all)
  • Decision Sciences(all)

Fingerprint Dive into the research topics of 'CrowdSelect: Increasing accuracy of crowdsourcing tasks through behavior prediction and user selection'. Together they form a unique fingerprint.

  • Cite this

    Qiu, C., Squicciarini, A. C., Carminati, B., Caverlee, J., & Khare, D. R. (2016). CrowdSelect: Increasing accuracy of crowdsourcing tasks through behavior prediction and user selection. In CIKM 2016 - Proceedings of the 2016 ACM Conference on Information and Knowledge Management (pp. 539-548). (International Conference on Information and Knowledge Management, Proceedings; Vol. 24-28-October-2016). Association for Computing Machinery. https://doi.org/10.1145/2983323.2983830