TY - GEN
T1 - The benefits of a model of annotation
AU - Passonneau, Rebecca J.
AU - Carpenter, Bob
PY - 2020
Y1 - 2020
N2 - This paper presents a case study of a difficult and important categorical annotation task (word sense) to demonstrate a probabilistic annotation model applied to crowdsourced data. It is argued that standard (chance-adjusted) agreement levels are neither necessary nor sufficient to ensure high quality gold standard labels. Compared to conventional agreement measures, application of an annotation model to instances with crowdsourced labels yields higher quality labels at lower cost.
AB - This paper presents a case study of a difficult and important categorical annotation task (word sense) to demonstrate a probabilistic annotation model applied to crowdsourced data. It is argued that standard (chance-adjusted) agreement levels are neither necessary nor sufficient to ensure high quality gold standard labels. Compared to conventional agreement measures, application of an annotation model to instances with crowdsourced labels yields higher quality labels at lower cost.
UR - http://www.scopus.com/inward/record.url?scp=85084336743&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85084336743&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85084336743
T3 - LAW 2013 and ID 2013 - 7th Linguistic Annotation Workshop and Interoperability with Discourse, Proceedings of the Workshop
SP - 187
EP - 195
BT - LAW 2013 and ID 2013 - 7th Linguistic Annotation Workshop and Interoperability with Discourse, Proceedings of the Workshop
PB - Association for Computational Linguistics (ACL)
T2 - 7th Linguistic Annotation and Interoperability with Discourse Workshop, LAW 2013 and ID 2013
Y2 - 8 August 2013 through 9 August 2013
ER -