Computing reliability for coreference annotation

Research output: Chapter in Book/Report/Conference proceedingConference contribution

56 Scopus citations

Abstract

Coreference annotation is annotation of language corpora to indicate wlücti expressions tiave been used to co-specify ttie same discourse entity. Wlien annotations of the same data are collected from two or more coders, the reliability of the data may need to be quantified. Two obstacles have stood in the way of applying reliability metrics: incommensurate units across annotations, and lack of a convenient representation of the coding values. Given N coders and M coding units, reliability is computed from an N-by-M matrix that records the value assigned to unit Mj by coder Nj,. The solution I present accommodates a wide range of coding choices for the annotator, while preserving the same units across codings. As a consequence, it permits a straightforward application of reliability measurement. In addition, in coreference annotation, disagreements can be complete or partial so I incorporate a distance metric to scale disagreements. This method has also been applied to a quite distinct coding task, namely semantic annotation of summaries.

Original languageEnglish (US)
Title of host publicationProceedings of the 4th International Conference on Language Resources and Evaluation, LREC 2004
EditorsMaria Francisca Xavier, Rute Costa, Fatima Ferreira, Maria Teresa Lino, Raquel Silva
PublisherEuropean Language Resources Association (ELRA)
Pages1503-1506
Number of pages4
ISBN (Electronic)2951740816, 9782951740815
StatePublished - Jan 1 2004
Event4th International Conference on Language Resources and Evaluation, LREC 2004 - Lisbon, Portugal
Duration: May 26 2004May 28 2004

Publication series

NameProceedings of the 4th International Conference on Language Resources and Evaluation, LREC 2004

Other

Other4th International Conference on Language Resources and Evaluation, LREC 2004
CountryPortugal
CityLisbon
Period5/26/045/28/04

All Science Journal Classification (ASJC) codes

  • Library and Information Sciences
  • Education
  • Language and Linguistics
  • Linguistics and Language

Fingerprint Dive into the research topics of 'Computing reliability for coreference annotation'. Together they form a unique fingerprint.

  • Cite this

    Passonneau, R. J. (2004). Computing reliability for coreference annotation. In M. F. Xavier, R. Costa, F. Ferreira, M. T. Lino, & R. Silva (Eds.), Proceedings of the 4th International Conference on Language Resources and Evaluation, LREC 2004 (pp. 1503-1506). (Proceedings of the 4th International Conference on Language Resources and Evaluation, LREC 2004). European Language Resources Association (ELRA).