PEAK: Pyramid evaluation via automated knowledge extraction

Qian Yang, Rebecca J. Passonneau, Gerard De Melo

7 Scopus citations

Abstract

Evaluating the selection of content in a summary is important both for human-written summaries, which can be a useful pedagogical tool for reading and writing skills, and machinegenerated summaries, which are increasingly being deployed in information management. The pyramid method assesses a summary by aggregating content units from the summaries of a wise crowd (a form of crowdsourcing). It has proven highly reliable but has largely depended on manual annotation. We propose PEAK, the first method to automatically assess summary content using the pyramid method that also generates the pyramid content models. PEAK relies on open information extraction and graph algorithms. The resulting scores correlate well with manually derived pyramid scores on both human and machine summaries, opening up the possibility of wide-spread use in numerous applications.

Original languageEnglish (US)
Title of host publication30th AAAI Conference on Artificial Intelligence, AAAI 2016
PublisherAAAI press
Pages2673-2679
Number of pages7
ISBN (Electronic)9781577357605
StatePublished - Jan 1 2016
Event30th AAAI Conference on Artificial Intelligence, AAAI 2016 - Phoenix, United States
Duration: Feb 12 2016Feb 17 2016

Publication series

Name30th AAAI Conference on Artificial Intelligence, AAAI 2016

Other

Other30th AAAI Conference on Artificial Intelligence, AAAI 2016
CountryUnited States
CityPhoenix
Period2/12/162/17/16

    Fingerprint

Cite this

Yang, Q., Passonneau, R. J., & De Melo, G. (2016). PEAK: Pyramid evaluation via automated knowledge extraction. In 30th AAAI Conference on Artificial Intelligence, AAAI 2016 (pp. 2673-2679). (30th AAAI Conference on Artificial Intelligence, AAAI 2016). AAAI press.