Elamur Research⌕
☰
home / papers / 2009 / Exploring Content Models for Multi-Document Summarization

Exploring Content Models for Multi-Document Summarization

Summary

The paper investigates whether richer probabilistic content models improve generic extractive multi-document summarization, compared with simple frequency-based scoring. The authors start from SumBasic and build a ladder of increasingly structured models. KLSum picks sentences so that the summary's unigram distribution has minimal KL divergence from the document set's distribution. TopicSum keeps the same criterion but replaces raw frequencies with a learned "content" distribution from an LDA-style model that separates out background and document-specific words. HierSum adds a hierarchy of one general content topic plus several specific sub-topics. Models were developed on DUC 2006 and tested on DUC 2007, using ROUGE and a pairwise user study. ROUGE improved at each step. A bigram variant of HierSum matched the discriminative PYTHY system (Toutanova et al., 2007) on ROUGE, and human judges clearly preferred HierSum. Because it learns sub-topics, HierSum can also produce topic-specific summaries for exploring a collection. This shows that unsupervised generative topic models are competitive and easy to extend for summarization.

Key points

  • Development on DUC 2006 (50 sets × 25 documents), final tests on DUC 2007, with summaries limited to 250 words. Metrics were R-1, R-2 and R-SU4, each computed with and without stop words.
  • R-2 without stop words on DUC 2006 rose steadily:
    • SumBasic: 5.3
    • KLSum: 6.0, equal to SumFocus, the top R-2 system at DUC 2006
    • TopicSum: 6.3
    • HierSum: 6.4
  • TopicSum uses three topic types: BACKGROUND (shared across collections, which absorbs stop words), CONTENT (one per document set) and DOCSPECIFIC (one per document). Each sentence's mixture over these has Dirichlet pseudo-counts (1.0, 5.0, 10.0).
  • Running HierSum over bags of bigrams gave 9.3 R-2 without stop words on DUC 2007. This beat PYTHY, though not significantly.
  • In the manual study (69 pairwise preferences), HierSum beat PYTHY on all four questions, all significant by t-test:
    • overall: 49 vs 20
    • non-redundancy: 48 vs 21
    • coherence: 54 vs 15
    • focus: 41 vs 28

Interesting ideas

  • I1. Summarization as distribution matching: Picking sentences to minimise KL(document set ‖ summary) naturally rewards coverage and discourages repetition, with no hand-made redundancy penalty. It is a simple objective that could be reused wherever a subset should represent a whole, such as corpus sampling or selecting dataset examples.
  • I2. Learning what counts as content: Instead of using stop-word lists or raw counts, the model lets a shared background topic and per-document topics absorb uninformative words. Only words spread across many documents end up as core content. This idea suits languages or domains, including Ukrainian, where good stop-word lists are lacking.
  • I3. Positional and sequential priors in the generative story: Earlier sentences are biased toward general content, and each sentence's sub-topic depends on the previous sentence's. Cheap structural cues like these can be built into many document models.
  • I4. Swapping the unit from unigrams to bigrams: Treating sentences as bags of bigrams brought a large ROUGE gain without changing the model, even though the authors admit the model becomes formally degenerate. This is a pragmatic trick for aligning a model with the units the evaluation metric counts.
  • I5. Topics as a navigation interface: The learned sub-topics can be shown as key phrases, each linked to its own extractive summary. This lets users explore a collection before they know what to ask, which is useful for browsing news clusters or archives.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

130 papers of our library cite it (of the 140884 whose bibliographies we read); 9 of them have a page here, 121 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{haghighi2009exploring,
  title = {{Exploring Content Models for Multi-Document Summarization}},
  author = {Haghighi, Aria and Vanderwende, Lucy},
  booktitle = {Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
  doi = {10.3115/1620754.1620807},
  url = {https://doi.org/10.3115/1620754.1620807}
}

References

24 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Regina Barzilay and Lillian Lee. 2004. Catching the drift: Probabilistic content models, with applications to generation and summarization. In HLT-NAACL. find our card
  2. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. JMLR. find our card
  3. David M. Blei, Thomas L. Griffiths, Michael I. Jordan, and Joshua B. Tenenbaum. 2004. Hierarchical topic models and the nested chinese restaurant process. In NIPS. find
  4. S.R.K. Branavan, Pawan Deshpande, and Regina Barzilay. 2007. Generating a table-of-contents. In ACL. find
  5. Hal Daumé III and Daniel Marcu. 2006. Bayesian query-focused summarization. In Proceedings of the Conference of the Association for Computational Linguistics (ACL). find
  6. Jacob Eisenstein and Regina Barzilay. 2008. Bayesian unsupervised topic segmentation. In EMNLP-SIGDAT. find
  7. Thomas Griffiths. 2002. Gibbs sampling in the generative model of latent dirichlet allocation. find
  8. Sanda Harabagiu, Andrew Hickl, and Finley Lacatusu. 2007. Satisfying information needs with multi-document summaries. Inf. Process. Manage., 43(6). find
  9. Mirella Lapata. 2003. Probabilistic text structuring: Experiments with sentence ordering. In ACL. find our card
  10. Jurij Leskovec, Natasa Milic-frayling, and Marko Grobelnik. 2005. Impact of linguistic analysis on the semantic graph coverage and learning of document extracts. In In AAAI 05. find
  11. Anton Leuski, Chin-Yew Lin, and Eduard Hovy. 2003. ineats: Interactive multi-document summarization. In ACL. find
  12. Chin-Yew Lin, Guihong Cao, Jianfeng Gao, and Jian-Yun Nie. 2006. An information-theoretic approach to automatic evaluation of summaries. In HLT-NAACL. find
  13. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proc. ACL workshop on Text Summarization Branches Out. find
  14. H.P. Luhn. 1958. The automatic creation of literature abstracts. IBM Journal. find
  15. Ryan McDonald. 2007. A study of global inference algorithms in multi-document summarization. In ECIR. find
  16. Kathleen R. Mckeown, Judith L. Klavans, Vasileios Hatzivassiloglou, Regina Barzilay, and Eleazar Eskin. 1999. Towards multidocument summarization by reformulation: Progress and prospects. In In Proceedings of AAAI-99. find
  17. Vivi Nastase. 2008. Topic-driven multi-document summarization with encyclopedic knowledge and spreading activation. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. find
  18. A. Nenkova and L. Vanderwende. 2005. The impact of frequency on summarization. Technical report, Microsoft Research. find
  19. Dragomir R. Radev. 2004. Lexrank: graph-based centrality as salience in text summarization. Journal of Artificial Intelligence Research (JAIR. find
  20. M. Steyvers and T. Griffiths, 2007. Probabilistic Topic Models. find
  21. Kristina Toutanova, Chris Brockett, Michael Gamon Jagadeesh Jagarlamudi, Hisami Suzuki, and Lucy Vanderwende. 2007. The pythy summarization system: Microsoft research at duc 2007. In DUC. find
  22. Lucy Vanderwende, Hisami Suzuki, Chris Brockett, and Ani Nenkova. 2007. Beyond sumbasic: Task-focused summarization with sentence simplification and lexical expansion. volume 43. find
  23. Xiaojun Wan and Jianwu Yang. 2006. Improved affinity graph based multi-document summarization. In HLT-NAACL. find
  24. Xuerui Wang, Andrew McCallum, and Xing Wei. 2007. Topical n-grams: Phrase and topic discovery, with an application to information retrieval. In ICDM. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/haghighi-2009-exploring-content-models-multi-document/