Elamur Research⌕
☰
home / papers / 2008 / Bayesian Unsupervised Topic Segmentation

Bayesian Unsupervised Topic Segmentation

Paper · 2008 · Jacob Eisenstein and Regina Barzilay

Summary

The paper tackles unsupervised linear topic segmentation: splitting a document into a sequence of topically coherent segments without labeled data. Earlier unsupervised systems measured lexical cohesion with hand-designed similarity scores, which are hard to combine with other cues and often need tuning. Eisenstein and Barzilay instead cast cohesion as a generative model. Each segment has its own multinomial language model, and the sentences in a segment are drawn from it. A segmentation with high likelihood therefore concentrates each segment's vocabulary. The language models are either point-estimated or integrated out via the Dirichlet Compound Multinomial. Dynamic programming then finds the exact best segmentation. Cue phrases are added as emissions from segment boundaries, drawn from one shared multinomial, and a sampler handles inference for that version. On the ICSI meeting corpus and a new medical-textbook corpus, the system (BayesSeg) beats U&I, MCS and LCSeg on both Pk and WindowDiff. The work matters because it turns an ad hoc heuristic into a principled model that extends naturally to new features and needs no tuning.

Key points

  • Evaluation used the ICSI meeting transcripts and a newly built text corpus taken from a medical textbook. All systems were given the desired number of segments.
  • On the textbook corpus, BayesSeg scored Pk .339 and WD .353, against roughly .368–.385 for the baselines.
  • On the meeting corpus, the best variant (BayesSeg-cue-prop) reached Pk .258 and WD .312. The strongest baseline results were U&I's Pk of .297 and LCSeg's WD of .322. Purver et al. (2006) reported .289 and .329.
  • Cue phrases helped on meetings but not on the textbook.
    • On meetings, the top induced cues included "okay", "so", "um" and "but". Four of the top ten matched the supervised list of Galley et al. (2003).
    • On the textbook, mostly very frequent words such as "the" were selected.
  • The model needs no development-set tuning because its priors are re-estimated. Code is released at groups.csail.mit.edu/rbg/code/bayesseg/.

Interesting ideas

  • I1. Replace a heuristic score with a likelihood. Reframing "cohesion" as the likelihood under per-segment language models makes the objective principled. It also makes the objective extensible, since new evidence becomes just another generated variable. The same move could work for other tasks driven by similarity heuristics, such as chunking or document clustering.
  • I2. Recover old methods as special cases. With a zero prior, the objective becomes a length-weighted sum of segment entropies. U&I's smoothing corresponds to one particular prior choice. Showing that an existing method falls out of a general model is a useful way to justify and unify prior work.
  • I3. Use a shared multinomial to induce markers without labels. Generating boundary words from one distribution shared across documents pushes the model toward a compact set of markers. Ranking the results by chi-squared then gives an interpretable sanity check. This could help discover discourse markers in under-resourced languages.
  • I4. Inject knowledge through the proposal, not the model. Biasing the sampler toward boundaries that contain known cue phrases steers the search while leaving the probabilistic model untouched. This is a cheap way to add linguistic knowledge without hurting the model's validity.
  • I5. Exploit linear structure for exact inference. Restricting the search to segmentation points and marginalizing out the language models allows exact dynamic programming. This avoids the cost of sampling many layers of hidden variables, as LDA-style segmenters must.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States159 works · 65,157 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Jacob Eisenstein and Regina Barzilay. 2008. Bayesian Unsupervised Topic Segmentation. Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{eisenstein2008bayesian,
  title = {{Bayesian Unsupervised Topic Segmentation}},
  author = {Jacob Eisenstein and Regina Barzilay},
  booktitle = {Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing},
  year = {2008},
  doi = {10.3115/1613715.1613760},
  url = {https://doi.org/10.3115/1613715.1613760}
}

References

32 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Doug Beeferman, Adam Berger, and John D. Lafferty. 1999. Statistical models for text segmentation. Machine Learning, 34(1-3):177–210. find
  2. José M. Bernardo and Adrian F. M. Smith. 2000. Bayesian Theory. Wiley. find
  3. Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning. Springer. find
  4. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993–1022.
  5. Freddy Y. Y. Choi. 2000. Advances in domain independent linear text segmentation. In Proceedings of NAACL, pages 26–33.
  6. Micha Elsner and Eugene Charniak. 2008. You Talking to Me? A Corpus and Algorithm for Conversation Disentanglement. In Proceedings of ACL. find
  7. Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of ACL, pages 363–370.
  8. Michel Galley, Kathleen R. McKeown, Eric Fosler-Lussier, and Hongyan Jing. 2003. Discourse segmentation of multi-party conversation. Proceedings of ACL, pages 562–569.
  9. Jean-Luc Gauvain and Chin-Hui Lee. 1994. Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains. IEEE Transactions on Speech and Audio Processing, 2(2):291–298. find
  10. Dmitriy Genzel and Eugene Charniak. 2002. Entropy rate constancy in text. In Proceedings of ACL, pages 199–206.
  11. Sharon Goldwater and Tom Griffiths. 2007. A fully bayesian approach to unsupervised part-of-speech tagging. In Proceedings of ACL, pages 744–751.
  12. Barbara Grosz and Candace Sidner. 1986. Attention, intentions, and the structure of discourse. Computational Linguistics, 12(3):175–204.
  13. Barbara Grosz. 1977. The representation and use of focus in dialogue understanding. Technical Report 151, Artificial Intelligence Center, SRI International. find
  14. M. A. K. Halliday and Ruqaiya Hasan. 1976. Cohesion in English. Longman. find
  15. Marti A. Hearst. 1994. Multi-paragraph segmentation of expository text. In Proceedings of ACL, pages 9–16. find
  16. Julia Hirschberg and Diane Litman. 1993. Empirical studies on the disambiguation of cue phrases. Computational Linguistics, 19(3):501–530.
  17. A. Janin, D. Baron, J. Edwards, D. Ellis, D. Gelbart, N. Morgan, B. Peskin, T. Pfau, E. Shriberg, A. Stolcke, et al. 2003. find
  18. The ICSI Meeting Corpus. Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03). 2003 IEEE International Conference on, 1. find
  19. Norman L. Johnson, Samuel Kotz, and N. Balakrishnan. 1997. Discrete Multivariate Distributions. Wiley. find
  20. Alistair Knott. 1996. A Data-Driven Methodology for Motivating a Set of Coherence Relations. Ph.D. thesis, University of Edinburgh. find
  21. Diane J. Litman and Rebecca J. Passonneau. 1995. Combining multiple knowledge sources for discourse segmentation. In Proceedings of the ACL, pages 108–115. find
  22. Igor Malioutov and Regina Barzilay. 2006. Minimum cut model for spoken lecture segmentation. In Proceedings of ACL, pages 25–32.
  23. Christopher D. Manning and Hinrich Schütze. 1999. Foundations of Statistical Natural Language Processing. The MIT Press. find
  24. Thomas P. Minka. 2003. Estimating a dirichlet distribution. Technical report, Massachusetts Institute of Technology. find
  25. Rebecca Passonneau and Diane Litman. 1993. Intention-based segmentation: Human reliability and correlation with linguistic cues. In Proceedings of ACL, pages 148–155. find
  26. Lev Pevzner and Marti A. Hearst. 2002. A critique and improvement of an evaluation metric for text segmentation. Computational Linguistics, 28(1):19–36.
  27. M. F. Porter. 1980. An algorithm for suffix stripping. Program, 14:130–137. find
  28. M. Purver, T.L. Griffiths, K.P. Körding, and J.B. Tenenbaum. 2006. Unsupervised topic modelling for multi-party spoken discourse. In Proceedings of ACL, pages 17–24. find
  29. Caroline Sporleder and Mirella Lapata. 2006. Broad coverage paragraph segmentation across languages and domains. ACM Transactions on Speech and Language Processing, 3(2):1–35. find
  30. Masao Utiyama and Hitoshi Isahara. 2001. A statistical model for domain-independent text segmentation. In Proceedings of ACL, pages 491–498.
  31. H. Kenneth Walker, W. Dallas Hall, and J. Willis Hurst, editors. 1990. Clinical Methods : The History, Physical, and Laboratory Examinations. Butterworths. find
  32. Greg C. G. Wei and Martin A. Tanner. 1990. A monte carlo implementation of the EM algorithm and the poor man’s data augmentation algorithms. Journal of the American Statistical Association, 85(411), September. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/eisenstein-2008-bayesian-unsupervised-topic-segmentation/