📄 The Automated Acquisition of Topic Signatures for Text Summarization

Summary

The paper addresses a core summarization problem: a text often describes the pieces of a complex concept (e.g., menu, eating, paying in a restaurant visit) without ever naming the concept itself. Hand-built scripts and extraction templates, as in FRUMP, SUMMONS or CIRCUS, are costly to engineer. Lin and Hovy instead learn "topic signatures" automatically: weighted lists of terms tied to a head topic. They are mined from documents pre-labeled as relevant or nonrelevant, here 6,194 TREC texts covering four topics. Each term is scored with Dunning's likelihood ratio (−2 log λ), and a chi-square confidence level sets the cutoff for inclusion. The signatures are plugged into the topic identification stage of the SUMMARIST system and compared with a position baseline and a tf·idf module using sentence-level F-scores against human model extracts. Topic signatures outperform both alternatives, while tf·idf does poorly. This makes the approach a cheap, parser-free first approximation of world knowledge for summarization.

Key points

• A topic signature is defined as a topic plus a vector of (term, weight) pairs. Terms may be stemmed words, bigrams or trigrams.
• Weights come from the −2 log λ likelihood ratio over a 2×2 term/relevance contingency table. The authors show this equals 2N·I(R;T), so it matches mutual information under binomial assumptions.
• Training data are 6,194 TREC texts on four topics. Named topics include 257 (Cigarette Consumption), 258 (Computer Security) and 271 (Solar Power).
• The evaluation uses three-fold cross-validation with unigram signatures only. Extract lengths run from 0% to 100% in 10% steps, scored by F-measure with β=1.
• Out of 40 test points (four topics times ten lengths), the signature module beat the position baseline 34 times.

Interesting ideas

• I1. Statistical cutoffs instead of arbitrary top-k. Because −2 log λ is asymptotically chi-square distributed, a significance level can decide how many terms to keep. This gives a principled vocabulary-pruning rule for any keyword or lexicon induction task.
• I2. Likelihood ratio for sparse counts. Preferring Dunning's statistic over χ² suits rare terms well. It could help low-resource languages or small domain corpora, where many useful words occur only a few times.
• I3. Sentence scoring by summed signature weights. Each sentence gets the total of its signature-term weights, normalized by the best-scoring sentence. This simple scheme can be reused for topic-focused extraction, highlighting or snippet selection.
• I4. Evaluating across the full length curve. The authors compare methods at every 10% extract length rather than at one length and count wins across topic/length points. This guards against conclusions that hold only at a single summary length.
• I5. Signatures as inverse query expansion, enriched with an ontology. The learned term clusters could be attached to ontology nodes such as SENSUS to group terms under concepts. They could also be reused for query expansion, linking summarization resources with IR.