Measuring variability in sentence ordering for news summarization

Nitin Madnani, Rebecca Jane Passonneau, Necip Fazil Ayan, John M. Conroy, Bonnie J. Dorr, Judith L. Klavans, Dianne P. O'Leary, Judith D. Schlesinger

Research output: Contribution to conferencePaper

11 Citations (Scopus)

Abstract

The issue of sentence ordering is an important one for natural language tasks such as multi-document summarization, yet there has not been a quantitative exploration of the range of acceptable sentence orderings for short texts. We present results of a sentence reordering experiment with three experimental conditions. Our findings indicate a very high degree of variability in the orderings that the eighteen subjects produce. In addition, the variability of reorderings is significantly greater when the initial ordering seen by subjects is different from the original summary. We conclude that evaluation of sentence ordering should use multiple reference orderings. Our evaluation presents several metrics that might prove useful in assessing against multiple references. We conclude with a deeper set of questions: (a) what sorts of independent assessments of quality of the different reference orderings could be made and (b) whether a large enough test set would obviate the need for such independent means of quality assessment.

Original languageEnglish (US)
Pages81-88
Number of pages8
StatePublished - Dec 1 2007
Event11th European Workshop on Natural Language Generation, ENLG 07 - Schloss Dagstuhl, Germany
Duration: Jun 17 2007Jun 20 2007

Other

Other11th European Workshop on Natural Language Generation, ENLG 07
CountryGermany
CitySchloss Dagstuhl
Period6/17/076/20/07

Fingerprint

news
evaluation
experiment
language
Evaluation
Summarization
News
Summary
Quality Assessment
Experiment
Natural Language

All Science Journal Classification (ASJC) codes

  • Language and Linguistics
  • Linguistics and Language

Cite this

Madnani, N., Passonneau, R. J., Ayan, N. F., Conroy, J. M., Dorr, B. J., Klavans, J. L., ... Schlesinger, J. D. (2007). Measuring variability in sentence ordering for news summarization. 81-88. Paper presented at 11th European Workshop on Natural Language Generation, ENLG 07, Schloss Dagstuhl, Germany.
Madnani, Nitin ; Passonneau, Rebecca Jane ; Ayan, Necip Fazil ; Conroy, John M. ; Dorr, Bonnie J. ; Klavans, Judith L. ; O'Leary, Dianne P. ; Schlesinger, Judith D. / Measuring variability in sentence ordering for news summarization. Paper presented at 11th European Workshop on Natural Language Generation, ENLG 07, Schloss Dagstuhl, Germany.8 p.
@conference{134be6dfd0194a92ae42f5e61eb3a762,
title = "Measuring variability in sentence ordering for news summarization",
abstract = "The issue of sentence ordering is an important one for natural language tasks such as multi-document summarization, yet there has not been a quantitative exploration of the range of acceptable sentence orderings for short texts. We present results of a sentence reordering experiment with three experimental conditions. Our findings indicate a very high degree of variability in the orderings that the eighteen subjects produce. In addition, the variability of reorderings is significantly greater when the initial ordering seen by subjects is different from the original summary. We conclude that evaluation of sentence ordering should use multiple reference orderings. Our evaluation presents several metrics that might prove useful in assessing against multiple references. We conclude with a deeper set of questions: (a) what sorts of independent assessments of quality of the different reference orderings could be made and (b) whether a large enough test set would obviate the need for such independent means of quality assessment.",
author = "Nitin Madnani and Passonneau, {Rebecca Jane} and Ayan, {Necip Fazil} and Conroy, {John M.} and Dorr, {Bonnie J.} and Klavans, {Judith L.} and O'Leary, {Dianne P.} and Schlesinger, {Judith D.}",
year = "2007",
month = "12",
day = "1",
language = "English (US)",
pages = "81--88",
note = "11th European Workshop on Natural Language Generation, ENLG 07 ; Conference date: 17-06-2007 Through 20-06-2007",

}

Madnani, N, Passonneau, RJ, Ayan, NF, Conroy, JM, Dorr, BJ, Klavans, JL, O'Leary, DP & Schlesinger, JD 2007, 'Measuring variability in sentence ordering for news summarization' Paper presented at 11th European Workshop on Natural Language Generation, ENLG 07, Schloss Dagstuhl, Germany, 6/17/07 - 6/20/07, pp. 81-88.

Measuring variability in sentence ordering for news summarization. / Madnani, Nitin; Passonneau, Rebecca Jane; Ayan, Necip Fazil; Conroy, John M.; Dorr, Bonnie J.; Klavans, Judith L.; O'Leary, Dianne P.; Schlesinger, Judith D.

2007. 81-88 Paper presented at 11th European Workshop on Natural Language Generation, ENLG 07, Schloss Dagstuhl, Germany.

Research output: Contribution to conferencePaper

TY - CONF

T1 - Measuring variability in sentence ordering for news summarization

AU - Madnani, Nitin

AU - Passonneau, Rebecca Jane

AU - Ayan, Necip Fazil

AU - Conroy, John M.

AU - Dorr, Bonnie J.

AU - Klavans, Judith L.

AU - O'Leary, Dianne P.

AU - Schlesinger, Judith D.

PY - 2007/12/1

Y1 - 2007/12/1

N2 - The issue of sentence ordering is an important one for natural language tasks such as multi-document summarization, yet there has not been a quantitative exploration of the range of acceptable sentence orderings for short texts. We present results of a sentence reordering experiment with three experimental conditions. Our findings indicate a very high degree of variability in the orderings that the eighteen subjects produce. In addition, the variability of reorderings is significantly greater when the initial ordering seen by subjects is different from the original summary. We conclude that evaluation of sentence ordering should use multiple reference orderings. Our evaluation presents several metrics that might prove useful in assessing against multiple references. We conclude with a deeper set of questions: (a) what sorts of independent assessments of quality of the different reference orderings could be made and (b) whether a large enough test set would obviate the need for such independent means of quality assessment.

AB - The issue of sentence ordering is an important one for natural language tasks such as multi-document summarization, yet there has not been a quantitative exploration of the range of acceptable sentence orderings for short texts. We present results of a sentence reordering experiment with three experimental conditions. Our findings indicate a very high degree of variability in the orderings that the eighteen subjects produce. In addition, the variability of reorderings is significantly greater when the initial ordering seen by subjects is different from the original summary. We conclude that evaluation of sentence ordering should use multiple reference orderings. Our evaluation presents several metrics that might prove useful in assessing against multiple references. We conclude with a deeper set of questions: (a) what sorts of independent assessments of quality of the different reference orderings could be made and (b) whether a large enough test set would obviate the need for such independent means of quality assessment.

UR - http://www.scopus.com/inward/record.url?scp=71049125538&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=71049125538&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:71049125538

SP - 81

EP - 88

ER -

Madnani N, Passonneau RJ, Ayan NF, Conroy JM, Dorr BJ, Klavans JL et al. Measuring variability in sentence ordering for news summarization. 2007. Paper presented at 11th European Workshop on Natural Language Generation, ENLG 07, Schloss Dagstuhl, Germany.