TY - GEN
T1 - The benefits of a model of annotation
AU - Passonneau, Rebecca J.
AU - Carpenter, Bob
N1 - Funding Information:
The first author was partially supported by from NSF CRI 0708952 and CRI 1059312, and the second by NSF CNS-1205516 and DOE DESC0002099. We thank Shreya Prasad for data collection, Mitzi Morris for feedback on the paper, Marilyn Walker for advice on Mechanical Turk, and Nancy Ide, Keith Suderman, Tim Brown and Mitzi Morris for help with the sentence data.
Publisher Copyright:
© 2013 Association for Computational Linguistics
PY - 2013
Y1 - 2013
N2 - This paper presents a case study of a difficult and important categorical annotation task (word sense) to demonstrate a probabilistic annotation model applied to crowdsourced data. It is argued that standard (chance-adjusted) agreement levels are neither necessary nor sufficient to ensure high quality gold standard labels. Compared to conventional agreement measures, application of an annotation model to instances with crowdsourced labels yields higher quality labels at lower cost.
AB - This paper presents a case study of a difficult and important categorical annotation task (word sense) to demonstrate a probabilistic annotation model applied to crowdsourced data. It is argued that standard (chance-adjusted) agreement levels are neither necessary nor sufficient to ensure high quality gold standard labels. Compared to conventional agreement measures, application of an annotation model to instances with crowdsourced labels yields higher quality labels at lower cost.
UR - http://www.scopus.com/inward/record.url?scp=84906926976&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84906926976&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:84906926976
SN - 9781937284589
T3 - 7th Linguistic Annotation Workshop and Interoperability with Discourse - Proceedings of the Workshop
SP - 187
EP - 195
BT - 7th Linguistic Annotation Workshop and Interoperability with Discourse - Proceedings of the Workshop
A2 - Dipper, Stefanie
A2 - Liakata, Maria
A2 - Liakata, Maria
A2 - Pareja-Lora, Antonio
PB - Association for Computational Linguistics (ACL)
T2 - 7th Linguistic Annotation Workshop and Interoperability with Discourse, LAW-ID 2013
Y2 - 8 August 2013 through 9 August 2013
ER -