📄 Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics

Summary

Lin and Hovy ask whether summaries can be scored automatically, the way the machine translation community had begun using BLEU/NIST, instead of relying on expensive and inconsistent human judges. They use DUC 2001 data, covering single-document summaries of 100 words and multi-document summaries of several lengths, with human coverage scores collected in the SEE tool as ground truth. Because DUC judging asks how much of a model summary a peer recovers, they turn BLEU's clipped n-gram precision into a recall measure computed against model units. This gives a family of scores, Ngram(i, j). They test these scores against two requirements: system rankings should match human rankings, and significance decisions should match those based on human scores. Running BLEU without changes correlates positively with human judgments, but not reliably. Simple unigram recall, Ngram(1,1), shows the strongest and most stable agreement. This made cheap, repeatable summary evaluation look feasible and laid the groundwork for later recall-based n-gram metrics.

Key points

• Human ground truth is a coverage score C. It counts the model units an assessor marks as matched, each multiplied by a completeness factor E that equals 1 when the unit is fully covered, 3/4 when most of it is, 1/2 when some of it is, 1/4 when hardly any of it is, and 0 when none of it is.
• Unmodified IBM BLEU on DUC 2001 reached a Spearman ρ of 0.66 with one reference and 0.82 with three references on the single-document task. On the multi-document task it reached 0.67 and 0.70. All four values were significant at α = 0.01.
• The rankings cover 14 instances in the single-document task (11 systems, 2 humans, 1 baseline) and 16 in the multi-document task (12 systems, 2 humans, 2 baselines).
• The BLEU-like Ngram(1,4) gave Spearman values between about 0.53 and 0.88 depending on the task and summary length. Results were computed both with and without stopwords.
• In recall/precision plots of significance agreement (z-tests at several α levels), Ngram(1,1) and Ngram(2,2) fall in the best region, and Ngram(1,1) is the best overall.

Interesting ideas

• I1. Meta-evaluate a metric by its significance decisions, not just correlation. The method counts system pairs that the metric calls significantly different and checks how often human scores agree, giving a recall and a precision. This tells you whether a cheap metric can be trusted to declare one system better than another, and the check applies to any automatic metric.
• I2. Match the metric's orientation to the task. BLEU rewards precision because a translation should not add wrong material. Summary judging asks how much of the reference content was captured, so the authors flip the denominator to the reference side. Before adopting a metric from a neighbouring field, ask whether the human protocol is precision-like or recall-like.
• I3. Longer n-grams may measure fluency rather than content. The authors argue that extractive summaries copy grammatical sentences, so 3- and 4-gram matches add little information about content. When outputs are mostly copied text, plain unigram overlap may be enough, and adding more complex components should be justified with evidence.
• I4. Partial-credit human judgments. Graded completeness labels (all, most, some, hardly any) turn subjective matching into a numeric score. This is a lightweight annotation design that could be reused for other content-coverage tasks.
• I5. A recurring "metric track" built on existing human judgments. Each year's manual results would be used to rank newly proposed automatic metrics, and the winning metric would serve as the in-house evaluator for the following year. This gets more value from costly annotation; NIST spent about 3,000 hours on each DUC.