The benefits of a model of annotation

Rebecca J. Passonneau, Bob Carpenter

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations

Abstract

This paper presents a case study of a difficult and important categorical annotation task (word sense) to demonstrate a probabilistic annotation model applied to crowdsourced data. It is argued that standard (chance-adjusted) agreement levels are neither necessary nor sufficient to ensure high quality gold standard labels. Compared to conventional agreement measures, application of an annotation model to instances with crowdsourced labels yields higher quality labels at lower cost.

Original languageEnglish (US)
Title of host publication7th Linguistic Annotation Workshop and Interoperability with Discourse - Proceedings of the Workshop
EditorsStefanie Dipper, Maria Liakata, Maria Liakata, Antonio Pareja-Lora
PublisherAssociation for Computational Linguistics (ACL)
Pages187-195
Number of pages9
ISBN (Print)9781937284589
StatePublished - 2013
Event7th Linguistic Annotation Workshop and Interoperability with Discourse, LAW-ID 2013 - Sofia, Bulgaria
Duration: Aug 8 2013Aug 9 2013

Publication series

Name7th Linguistic Annotation Workshop and Interoperability with Discourse - Proceedings of the Workshop

Conference

Conference7th Linguistic Annotation Workshop and Interoperability with Discourse, LAW-ID 2013
CountryBulgaria
CitySofia
Period8/8/138/9/13

All Science Journal Classification (ASJC) codes

  • Language and Linguistics
  • Linguistics and Language

Fingerprint Dive into the research topics of 'The benefits of a model of annotation'. Together they form a unique fingerprint.

Cite this