Motivation-Aware task assignment in crowdsourcing

Julien Pilourdault, Sihem Amer-Yahia, Dongwon Lee, Senjuti Basu Roy

Research output: Chapter in Book/Report/Conference proceedingConference contribution

7 Citations (Scopus)

Abstract

We investigate how to leverage the notion of motivation in assigning tasks to workers and improving the performance of a crowdsourcing system. In particular, we propose to model motivation as the balance between task diversity–i.e., the difference in skills among the tasks to complete, and task payment–i.e., the difference between how much a chosen task offers to pay and how much other available tasks pay. We propose to test different task assignment strategies: (1) relevance, a strategy that assigns matching tasks, i.e., those that fit a worker’s profile, (2) diversity, a strategy that chooses matching and diverse tasks, and (3) div-pay, a strategy that selects matching tasks that offer the best compromise between diversity and payment. For each strategy, we study multiple iterations where tasks are re-assigned to workers as their motivation evolves. At each iteration, relevance and diversity assign tasks to a worker from an available pool of filtered tasks. div-pay, on the other hand, estimates each worker’s motivation on-the-fly at each iteration, and uses it to assign tasks to the worker. Our empirical experiments study the impact of each strategy on overall performance. We examine both requester-centric and worker-centric performance dimensions and find that different strategies prevail for different dimensions. In particular, relevance offers the best task throughput while div-pay achieves the best outcome quality.

Original languageEnglish (US)
Title of host publicationAdvances in Database Technology - EDBT 2017
Subtitle of host publication20th International Conference on Extending Database Technology, Proceedings
EditorsBernhard Mitschang, Volker Markl, Sebastian Bress, Periklis Andritsos, Kai-Uwe Sattler, Salvatore Orlando
PublisherOpenProceedings.org
Pages246-257
Number of pages12
ISBN (Electronic)9783893180738
DOIs
StatePublished - Jan 1 2017
Event20th International Conference on Extending Database Technology, EDBT 2017 - Venice, Italy
Duration: Mar 21 2017Mar 24 2017

Publication series

NameAdvances in Database Technology - EDBT
Volume2017-March
ISSN (Electronic)2367-2005

Other

Other20th International Conference on Extending Database Technology, EDBT 2017
CountryItaly
CityVenice
Period3/21/173/24/17

Fingerprint

Throughput
Experiments

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Software
  • Computer Science Applications

Cite this

Pilourdault, J., Amer-Yahia, S., Lee, D., & Roy, S. B. (2017). Motivation-Aware task assignment in crowdsourcing. In B. Mitschang, V. Markl, S. Bress, P. Andritsos, K-U. Sattler, & S. Orlando (Eds.), Advances in Database Technology - EDBT 2017: 20th International Conference on Extending Database Technology, Proceedings (pp. 246-257). (Advances in Database Technology - EDBT; Vol. 2017-March). OpenProceedings.org. https://doi.org/10.5441/002/edbt.2017.23
Pilourdault, Julien ; Amer-Yahia, Sihem ; Lee, Dongwon ; Roy, Senjuti Basu. / Motivation-Aware task assignment in crowdsourcing. Advances in Database Technology - EDBT 2017: 20th International Conference on Extending Database Technology, Proceedings. editor / Bernhard Mitschang ; Volker Markl ; Sebastian Bress ; Periklis Andritsos ; Kai-Uwe Sattler ; Salvatore Orlando. OpenProceedings.org, 2017. pp. 246-257 (Advances in Database Technology - EDBT).
@inproceedings{29eeb2eff1734ac383c36ab556a1d738,
title = "Motivation-Aware task assignment in crowdsourcing",
abstract = "We investigate how to leverage the notion of motivation in assigning tasks to workers and improving the performance of a crowdsourcing system. In particular, we propose to model motivation as the balance between task diversity–i.e., the difference in skills among the tasks to complete, and task payment–i.e., the difference between how much a chosen task offers to pay and how much other available tasks pay. We propose to test different task assignment strategies: (1) relevance, a strategy that assigns matching tasks, i.e., those that fit a worker’s profile, (2) diversity, a strategy that chooses matching and diverse tasks, and (3) div-pay, a strategy that selects matching tasks that offer the best compromise between diversity and payment. For each strategy, we study multiple iterations where tasks are re-assigned to workers as their motivation evolves. At each iteration, relevance and diversity assign tasks to a worker from an available pool of filtered tasks. div-pay, on the other hand, estimates each worker’s motivation on-the-fly at each iteration, and uses it to assign tasks to the worker. Our empirical experiments study the impact of each strategy on overall performance. We examine both requester-centric and worker-centric performance dimensions and find that different strategies prevail for different dimensions. In particular, relevance offers the best task throughput while div-pay achieves the best outcome quality.",
author = "Julien Pilourdault and Sihem Amer-Yahia and Dongwon Lee and Roy, {Senjuti Basu}",
year = "2017",
month = "1",
day = "1",
doi = "10.5441/002/edbt.2017.23",
language = "English (US)",
series = "Advances in Database Technology - EDBT",
publisher = "OpenProceedings.org",
pages = "246--257",
editor = "Bernhard Mitschang and Volker Markl and Sebastian Bress and Periklis Andritsos and Kai-Uwe Sattler and Salvatore Orlando",
booktitle = "Advances in Database Technology - EDBT 2017",

}

Pilourdault, J, Amer-Yahia, S, Lee, D & Roy, SB 2017, Motivation-Aware task assignment in crowdsourcing. in B Mitschang, V Markl, S Bress, P Andritsos, K-U Sattler & S Orlando (eds), Advances in Database Technology - EDBT 2017: 20th International Conference on Extending Database Technology, Proceedings. Advances in Database Technology - EDBT, vol. 2017-March, OpenProceedings.org, pp. 246-257, 20th International Conference on Extending Database Technology, EDBT 2017, Venice, Italy, 3/21/17. https://doi.org/10.5441/002/edbt.2017.23

Motivation-Aware task assignment in crowdsourcing. / Pilourdault, Julien; Amer-Yahia, Sihem; Lee, Dongwon; Roy, Senjuti Basu.

Advances in Database Technology - EDBT 2017: 20th International Conference on Extending Database Technology, Proceedings. ed. / Bernhard Mitschang; Volker Markl; Sebastian Bress; Periklis Andritsos; Kai-Uwe Sattler; Salvatore Orlando. OpenProceedings.org, 2017. p. 246-257 (Advances in Database Technology - EDBT; Vol. 2017-March).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Motivation-Aware task assignment in crowdsourcing

AU - Pilourdault, Julien

AU - Amer-Yahia, Sihem

AU - Lee, Dongwon

AU - Roy, Senjuti Basu

PY - 2017/1/1

Y1 - 2017/1/1

N2 - We investigate how to leverage the notion of motivation in assigning tasks to workers and improving the performance of a crowdsourcing system. In particular, we propose to model motivation as the balance between task diversity–i.e., the difference in skills among the tasks to complete, and task payment–i.e., the difference between how much a chosen task offers to pay and how much other available tasks pay. We propose to test different task assignment strategies: (1) relevance, a strategy that assigns matching tasks, i.e., those that fit a worker’s profile, (2) diversity, a strategy that chooses matching and diverse tasks, and (3) div-pay, a strategy that selects matching tasks that offer the best compromise between diversity and payment. For each strategy, we study multiple iterations where tasks are re-assigned to workers as their motivation evolves. At each iteration, relevance and diversity assign tasks to a worker from an available pool of filtered tasks. div-pay, on the other hand, estimates each worker’s motivation on-the-fly at each iteration, and uses it to assign tasks to the worker. Our empirical experiments study the impact of each strategy on overall performance. We examine both requester-centric and worker-centric performance dimensions and find that different strategies prevail for different dimensions. In particular, relevance offers the best task throughput while div-pay achieves the best outcome quality.

AB - We investigate how to leverage the notion of motivation in assigning tasks to workers and improving the performance of a crowdsourcing system. In particular, we propose to model motivation as the balance between task diversity–i.e., the difference in skills among the tasks to complete, and task payment–i.e., the difference between how much a chosen task offers to pay and how much other available tasks pay. We propose to test different task assignment strategies: (1) relevance, a strategy that assigns matching tasks, i.e., those that fit a worker’s profile, (2) diversity, a strategy that chooses matching and diverse tasks, and (3) div-pay, a strategy that selects matching tasks that offer the best compromise between diversity and payment. For each strategy, we study multiple iterations where tasks are re-assigned to workers as their motivation evolves. At each iteration, relevance and diversity assign tasks to a worker from an available pool of filtered tasks. div-pay, on the other hand, estimates each worker’s motivation on-the-fly at each iteration, and uses it to assign tasks to the worker. Our empirical experiments study the impact of each strategy on overall performance. We examine both requester-centric and worker-centric performance dimensions and find that different strategies prevail for different dimensions. In particular, relevance offers the best task throughput while div-pay achieves the best outcome quality.

UR - http://www.scopus.com/inward/record.url?scp=85030570261&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85030570261&partnerID=8YFLogxK

U2 - 10.5441/002/edbt.2017.23

DO - 10.5441/002/edbt.2017.23

M3 - Conference contribution

AN - SCOPUS:85030570261

T3 - Advances in Database Technology - EDBT

SP - 246

EP - 257

BT - Advances in Database Technology - EDBT 2017

A2 - Mitschang, Bernhard

A2 - Markl, Volker

A2 - Bress, Sebastian

A2 - Andritsos, Periklis

A2 - Sattler, Kai-Uwe

A2 - Orlando, Salvatore

PB - OpenProceedings.org

ER -

Pilourdault J, Amer-Yahia S, Lee D, Roy SB. Motivation-Aware task assignment in crowdsourcing. In Mitschang B, Markl V, Bress S, Andritsos P, Sattler K-U, Orlando S, editors, Advances in Database Technology - EDBT 2017: 20th International Conference on Extending Database Technology, Proceedings. OpenProceedings.org. 2017. p. 246-257. (Advances in Database Technology - EDBT). https://doi.org/10.5441/002/edbt.2017.23