Do summaries help? A task-based evaluation of multi-document summarization

Kathleen McKeown, Rebecca J. Passonneau, David K. Elson, Ani Nenkova, Julia Hirschberg

Research output: Chapter in Book/Report/Conference proceedingConference contribution

50 Scopus citations

Abstract

We describe a task-based evaluation to determine whether multi-document summaries measurably improve user performance whe using online news browsing systems for directed research. We evaluated the multi-document summaries generated by Newsblaster, a robust news browsing system that clusters online news articles and summarizes multiple articles on each event. Four groups of subjects were asked to perform the same time-restricted fact-gathering tasks, reading news under different conditions: no summaries at all, single sentence summaries drawn from one of the articles, Newsblaster multi-document summaries, and human summaries. Our results show that, in comparison to source documents only, the quality of reports assembled using Newsblaster summaries was significantly better and user satisfaction was higher with both Newsblaster and human summaries.

Original languageEnglish (US)
Title of host publicationSIGIR 2005 - Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval
Pages210-217
Number of pages8
DOIs
StatePublished - Dec 1 2005
Event28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2005 - Salvador, Brazil
Duration: Aug 15 2005Aug 19 2005

Publication series

NameSIGIR 2005 - Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Other

Other28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2005
CountryBrazil
CitySalvador
Period8/15/058/19/05

All Science Journal Classification (ASJC) codes

  • Information Systems

Cite this

McKeown, K., Passonneau, R. J., Elson, D. K., Nenkova, A., & Hirschberg, J. (2005). Do summaries help? A task-based evaluation of multi-document summarization. In SIGIR 2005 - Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 210-217). (SIGIR 2005 - Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval). https://doi.org/10.1145/1076034.1076072