Elamur Research⌕
☰
home / papers / 2004 / Catching the Drift: Probabilistic Content Models, with Applications to Generation and Summarization

Catching the Drift: Probabilistic Content Models, with Applications to Generation and Summarization

Summary

The paper tackles the problem of modeling how texts in a narrow domain are organized in terms of content: which topics they cover and in what order. Instead of hand-built schemas or knowledge bases, Barzilay and Lee learn this structure from unannotated documents. They model it as a Hidden Markov Model, called a content model. Each state stands for a domain topic and emits sentences through its own bigram language model. Transitions between states capture typical topic orderings.

Training starts by clustering sentences by lexical similarity and then refines the clusters with Viterbi-style re-estimation, so that sentence position also shapes topic assignments. The model is tested on two tasks:

  • Information ordering: it beats Lapata's (2003) method by a wide margin.
  • Extractive summarization: on the Earthquakes data, its sentence selection agrees with human summaries 88% of the time.

The work matters because it gives empirical support to the old idea that word distribution patterns reflect discourse structure, at least within one domain. It also yields a cheap, reusable representation that transfers across generation and summarization.

Key points

  • Content models are HMMs whose states are topics with state-specific bigram language models. A plain n-gram model is a degenerate, single-state case.
  • Initial topics come from complete-link clustering of sentences, using cosine similarity over word-bigram features. During clustering, generic tokens stand in for proper names, numbers and dates, so that clusters group event types rather than specific events.
  • On information ordering, content models outperform Lapata's (2003) approach, with a gap of 78 percentage points for one domain and metric.
  • On summarization (Earthquakes), extraction accuracy is 88%, compared with 76% for a sentence classifier using words plus location and 69% for the leading-sentences baseline. Trained on only one third of the data, the content-model method still beats the classifier trained on the full set.
  • When the number of states is varied on Earthquakes, the same size (64 states) gives the best result on both tasks: 72% OSO prediction rate for ordering and 88% extraction accuracy for summarization.

Interesting ideas

  • I1. Catch-all insertion state. Small clusters are merged into an "etcetera" cluster. Its state gets a language model that is the complement of all the other states, so it absorbs digressions and unseen content. Any unsupervised segmenter or tagger that needs a principled "other" class could reuse this trick.
  • I2. Transitions from document-level co-occurrence. Transition probabilities are estimated from how many documents contain one cluster's sentence directly followed by another's, rather than from raw counts. This keeps one long article from dominating the ordering statistics.
  • I3. Order-aware re-clustering. Sentences are reassigned to clusters using the Viterbi path, which lets position separate lexically similar sentences. For example, casualty mentions at the end of a quake report often describe an earlier quake. This idea suits any domain where similar wording means different things depending on where it appears.
  • I4. One model, several tasks. The authors check that the same model configuration ranks similarly on ordering and on summarization. This suggests tuning a structural model on a task with cheap evaluation and then reusing it where labeled data is scarce.
  • I5. Masking entities before clustering. Replacing names, numbers and dates with placeholders pushes clusters toward abstract information types. This is a simple preprocessing step for topic induction on news-like corpora, including Ukrainian ones.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

116 papers of our library cite it (of the 140884 whose bibliographies we read); 12 of them have a page here, 104 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{barzilay2004catching,
  title = {{Catching the Drift: Probabilistic Content Models, with Applications to Generation and Summarization}},
  author = {Barzilay, Regina and Lee, Lillian},
  booktitle = {Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004},
  year = {2004},
  url = {https://aclanthology.org/N04-1015/}
}

References

29 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. F. C. Bartlett. 1932. Remembering: a study in experimental and social psychology. Cambridge University Press. find
  2. R. Barzilay, L. Lee. 2003. Learning to paraphrase: An unsupervised approach using multiple-sequence alignment. In HLT-NAACL 2003: Main Proceedings, 16– 23. find our card
  3. D. Beeferman, A. Berger, J. Lafferty. 1997. Text segmentation using exponential models. In Proceedings of EMNLP, 35–46. find
  4. S. F. Chen, K. Seymore, R. Rosenfeld. 1998. Topic adaptation for language modeling using unnormalized exponential models. In Proceedings of ICASSP, volume 2, 681–684. find
  5. G. DeJong. 1982. An overview of the FRUMP system. In W. G. Lehnert, M. H. Ringle, eds., Strategies for Natural Language Processing, 149–176. Lawrence Erlbaum Associates, Hillsdale, New Jersey. find
  6. P. A. Duboue, K. R. McKeown. 2003. Statistical acquisition of content selection rules for natural language generation. In Proceedings of EMNLP, 121–128. find
  7. S. Fine, Y. Singer, N. Tishby. 1998. The hierarchical hidden Markov model: Analysis and applications. Machine Learning, 32(1):41–62. find
  8. R. Florian, D. Yarowsky. 1999. Dynamic non-local language modeling via hierarchical topic-based adaptation. In Proceedings of the ACL, 167–174. find
  9. D. Gildea, T. Hofmann. 1999. Topic-based language models using EM. In Proceedings of EUROSPEECH, 2167–2170. find
  10. Z. Harris. 1982. Discourse and sublanguage. In R. Kittredge, J. Lehrberger, eds., Sublanguage: Studies of Language in Restricted Semantic Domains, 231–236. Walter de Gruyter, Berlin; New York. find
  11. M. Hearst. 1994. Multi-paragraph segmentation of expository text. In Proceedings of the ACL, 9–16. find
  12. R. Iyer, M. Ostendorf. 1996. Modeling long distance dependence in language: Topic mixtures vs. dynamic cache models. In Proceedings of ICSLP, 236–239. find
  13. D. R. Jones, C. A. Thompson. 2003. Identifying events using similarity and context. In Proceedings of CoNLL, 135–141. find
  14. R. Kittredge, T. Korelsky, O. Rambow. 1991. On the need for domain communication language. Computational Intelligence, 7(4):305–314. find
  15. J. Kupiec, J. Pedersen, F. Chen. 1999. A trainable document summarizer. In I. Mani, M. T. Maybury, eds., Advances in Automatic Summarization, 55–60. MIT Press, Cambridge, MA. find
  16. M. Lapata. 2003. Probabilistic text structuring: Experiments with sentence ordering. In Proceeding of the ACL, 545–552. find our card
  17. W. C. Mann, S. A. Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. TEXT, 8(3):243–281. find our card
  18. D. Marcu. 1997. The rhetorical parsing of natural language texts. In Proceedings of the ACL/EACL, 96– 103. find
  19. K. R. McKeown. 1985. Text Generation: Using Discourse Strategies and Focus Constraints to Generate Natural Language Text. Cambridge University Press, Cambridge, UK. find
  20. W. H. Press, S. A. Teukolsky, W. T. Vetterling, B. P. Flannery. 1997. Numerical Recipes in C: The Art of Scientific Computing. Cambridge University Press, second edition. find
  21. L. Rabiner. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2):257–286. find
  22. D. R. Radev, K. R. McKeown. 1998. Generating natural language summaries from multiple on-line sources. Computational Linguistics, 24(3):469–500. find
  23. O. Rambow. 1990. Domain communication knowledge. In Fifth International Workshop on Natural Language Generation, 87–94. find
  24. J. Reynar, A. Ratnaparkhi. 1997. A maximum entropy approach to identifying sentence boundaries. In Proceedings of the Fifth Conference on Applied Natural Language Processing, 16–19. find our card
  25. R. E. Schapire, Y. Singer. 2000. BoosTexter: A boosting-based system for text categorization. Machine Learning, 2/3:135–168. find
  26. R. Soricut, D. Marcu. 2003. Sentence level discourse parsing using syntactic and lexical information. In Proceedings of the HLT/NAACL, 228–235. find our card
  27. M. White, T. Korelsky, C. Cardie, V. Ng, D. Pierce, K. Wagstaff. 2001. Multi-document summarization via information extraction. In Proceedings of the HLT Conference, 263–269. find
  28. A. Wray. 2002. Formulaic Language and the Lexicon. Cambridge University Press, Cambridge. find
  29. J. Wu, S. Khudanpur. 2002. Building a topic-dependent maximum entropy language model for very large corpora. In Proceedings of ICASSP, volume 1, 777–780. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/barzilay-2004-catching-drift-probabilistic-content-models/