TY - GEN
T1 - The benefits of a model of annotation
AU - Passonneau, Rebecca J.
AU - Carpenter, Bob
N1 - Funding Information:
The first author was partially supported by from NSF CRI 0708952 and CRI 1059312, and the second by NSF CNS-1205516 and DOE DESC0002099. We thank Shreya Prasad for data collection, Mitzi Morris for feedback on the paper, Marilyn Walker for advice on Mechanical Turk, and Nancy Ide, Keith Suderman, Tim Brown and Mitzi Morris for help with the sentence data.
Publisher Copyright:
© 2013 Association for Computational Linguistics
PY - 2020
Y1 - 2020
N2 - This paper presents a case study of a difficult and important categorical annotation task (word sense) to demonstrate a probabilistic annotation model applied to crowdsourced data. It is argued that standard (chance-adjusted) agreement levels are neither necessary nor sufficient to ensure high quality gold standard labels. Compared to conventional agreement measures, application of an annotation model to instances with crowdsourced labels yields higher quality labels at lower cost.
AB - This paper presents a case study of a difficult and important categorical annotation task (word sense) to demonstrate a probabilistic annotation model applied to crowdsourced data. It is argued that standard (chance-adjusted) agreement levels are neither necessary nor sufficient to ensure high quality gold standard labels. Compared to conventional agreement measures, application of an annotation model to instances with crowdsourced labels yields higher quality labels at lower cost.
UR - http://www.scopus.com/inward/record.url?scp=85084336743&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85084336743&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85084336743
T3 - LAW 2013 and ID 2013 - 7th Linguistic Annotation Workshop and Interoperability with Discourse, Proceedings of the Workshop
SP - 187
EP - 195
BT - LAW 2013 and ID 2013 - 7th Linguistic Annotation Workshop and Interoperability with Discourse, Proceedings of the Workshop
PB - Association for Computational Linguistics (ACL)
T2 - 7th Linguistic Annotation and Interoperability with Discourse Workshop, LAW 2013 and ID 2013
Y2 - 8 August 2013 through 9 August 2013
ER -