Elamur Research⌕
☰
home / papers / 1997 / Text Tiling: Segmenting Text into Multi-paragraph Subtopic Passages

Text Tiling: Segmenting Text into Multi-paragraph Subtopic Passages

Paper · 1997 · Marti A. Hearst

Summary

Hearst addresses how to split long expository texts, such as magazine articles or reports without headings, into contiguous, non-overlapping blocks of several paragraphs, each covering one subtopic. Paragraph breaks often serve layout more than content, so a coarser unit is needed. The paper models a text as a series of local subtopic discussions nested under one or more overarching main topics. TextTiling detects shifts between subtopics using only lexical repetition, with no thesaurus, knowledge base, or discourse markers. The algorithm tokenizes the text into terms and pseudo-sentences, scores each gap between units by lexical similarity, and places boundaries at the deepest dips in the resulting score curve. On 12 texts, the automatic boundaries agree reasonably well with human judgments, though there is room for improvement. The work matters because it supplies passage-sized units for information retrieval, summarization, hypertext construction, and on-screen display. It has also been applied in several other languages.

Key points

  • The pipeline has three stages: tokenization, gap scoring, and boundary detection at the largest valleys of the similarity plot.
  • Three scoring schemes were designed: blocks, vocabulary introductions, and lexical chains. The article reports evaluation results for blocks and vocabulary introductions only.
  • Evaluation compares the output against human-identified subtopic boundaries in 12 texts. A running example is the 21-paragraph science article "Stargazers."
  • Adding thesaural relations in earlier versions lowered performance. WordNet-based relatedness is proposed as future work.
  • Others used TextTiling for Arabic news without paragraph breaks (Hasnah 1996), Japanese (Nomoto and Nitta 1994), and aligning English/German/French texts (van der Eijk 1994). Karlgren (1996) found relevant documents contain more TextTiles. A C implementation was offered free for research.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

Cite

Cite as Marti A. Hearst. 1997. Text Tiling: Segmenting Text into Multi-paragraph Subtopic Passages. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@article{hearst1997text,
  title = {{Text Tiling: Segmenting Text into Multi-paragraph Subtopic Passages}},
  author = {Marti A. Hearst},
  journal = {Computational Linguistics},
  year = {1997},
  url = {https://aclanthology.org/J97-1003/}
}

References

77 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Brown, Gillian and George Yule. 1983. Discourse Analysis. Cambridge Textbooks in Linguistics Series. Cambridge University Press. find
  2. Callan, James P. 1994. Passage-level evidence in document retrieval. In Proceedings of the 17th Annual International ACM/SIGIR Conference, pages 302-310, Dublin, Ireland. find
  3. Carletta, Jean. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249-254.
  4. Chafe, Wallace L. 1979. The flow of thought and the flow of language. In Talmy Giv6n, editor, Syntax and Semantics: Discourse and Syntax, volume 12. Academic Press, New York, pages 159-182. find
  5. Chen, Francine, Marti A. Hearst, Julian Kupiec, Jan O. Pedersen, and Lynn Wilcox. 1994. Metadata in mixed-media access. SIGMOD Record, 23(4):64-71. find
  6. Chen, Francine R. and Margaret Withgott. 1992. The use of emphasis to automatically summarize a spoken discourse. In ICASSP-92:1992 IEEE International Conference on Acoustics, Speech and Signal Processing, volume 1, pages 229-232. find
  7. Christel, M., T. Kanade, M. Mauldin, R. Reddy, M. Sirbu, S. Stevens, and H. Wactlar. 1995. Informedia digital video library. Communications of the ACM, 38(4):57-58, April. find
  8. Cochran, W, G. 1950. The comparison of percentages in matched samples. Biometrika, 37:256-266. find
  9. Condon, Sherri L. and Claude G. Cech. 1995. Problems for reliable discourse coding systems. In Johanna Moore and Marilyn Walker, editors, Empirical Methods in Discourse: Interpretation & Generation, AAAI Technical Report SS-95-06, Menlo Park, CA. AAA! Press. find
  10. Gale, William A., Kenneth W. Church, and David Yarowsky. 1992a. Estimating upper and lower bounds on the performance of word-sense disambiguation programs. In Proceedings of the 30th Meeting, pages 249-256. Association for Computational Linguistics. find
  11. Gale, William A., Kenneth W. Church, and David Yarowsky. 1992b. One sense per discourse. In Proceedings of the DARPA Speech and Natural Language Workshop. find
  12. Girill, T. R. 1991. Information chunking as an interface design issue for full-text databases. In Martin Dillon, editor, Interfaces for Information Retrieval and Online Systems. Greenwood Press, New York, NY, pages 149-158. find
  13. Grimes, J. 1975. The Thread of Discourse. Mouton, The Hague. find
  14. Grosz, Barbara J. and Candace L. Sidner. 1986. Attention, intention, and the structure of discourse. Computational Linguistics, 12(3):172-204. find
  15. Halliday, M. A. K. and R. Hasan. 1976. Cohesion in English. Longman, London. find
  16. Harman, Donna. 1991. How effective is suffixing? Journal of the American Society for Information Science (JASIS), 42(1):7-15. find
  17. Harman, Donna. 1993. Overview of the first Text REtrieval Conference. In Proceedings of the 16th Annual International ACM/SIG1R Conference, pages 36-48, Pittsburgh, PA. find
  18. Hasnah, Ahmad. 1996. Full Text Processing and Retrieval: Weight Ranking, Text Structuring, and Passage Retrieval for Arabic Documents. Ph.D. thesis, Illinois Institute of Technology. find
  19. Hearst, Marti A. 1993. TextTiling: A quantitative approach to discourse segmentation. Technical Report Sequoia 93/24, Computer Science Division, University of California, Berkeley. find
  20. Hearst, Marti A. 1994a. Context and Structure in Automated Full-Text Information Access. Ph.D. thesis, University of California at Berkeley. (Computer Science Division Technical Report UCB/CSD-94/836). find
  21. Hearst, Marti A. 1994b. Multi-paragraph segmentation of expository text. In Proceedings of the 32nd Meeting, pages 9-16, Las Cruces, NM, June. Association for Computational Linguistics. find
  22. Hearst, Marti A. 1995. TileBars: Visualization of term distribution information in full text information access. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, Denver, CO, May. find
  23. Hearst, Marti A. 1996. Improving full-text precision using simple query constraints. In Proceedings of the Fifth Annual Symposium on Document Analysis and Information Retrieval (SDAIR), Las Vegas, NV. find
  24. Hearst, Marti, Jan Pedersen, Peter Pirolli, Hinrich Sch(ietze, Gregory Grefenstette, and David Hull. 1996. Four TREC-4 Tracks: The Xerox site report. In Donna Harman, editor, Proceedings of the Fourth Text Retrieval Conference TREC-4. National Institute of Standards and Technology Special Publicatio… find
  25. Hearst, Marti A. and Christian Plaunt. 1993. Subtopic structuring for full-length document access. In Proceedings of the 16th Annual International ACM/SIGIR Conference, pages 59-68, Pittsburgh, PA. find
  26. Hersh, William R., Diane L. Elliot, David H. Hickam, Stephanie L. Wolf, and Anna Molnar. 1995. Towards new measures of information retrieval evaluation. In Proceedings of the 18th Annual International ACM/SIGIR Conference, pages 164-170, Seattle, WA. find
  27. Hinds, John. 1979. Organizational patterns in discourse. In Talmy Giv6n, editor, Syntax and Semantics: Discourse and Syntax, volume 12. Academic Press, New York, pages 135-158. find
  28. Hirschberg, Julia and Diane Litman. 1993. Empirical studies on the disambiguation of cue phrases. Computational Linguistics, 19(3):501-530.
  29. Hirschberg, Julia and Christine H. Nakatani. 1996. A prosodic analysis of discourse segments in direction-giving monologues. In Proceedings of the 34th Annual Meeting, pages 286-293, Santa Cruz, CA. Association for Computational Linguistics. find
  30. Hull, David and Gregory Grefenstette. 1995. Stemming algorithms — A case study for detailed evaluation. Journal of the American Society for Information Science (JASIS), 46(9). find
  31. Hwang, Chung Hee and Lenhart K. Schubert. 1992. Tense trees as the 'fine structure' of discourse. In Proceedings of the 30th Meeting, pages 232-240. Association for Computational Linguistics. find
  32. Isard, Amy and Jean Carletta. 1995. Replicability of transaction and action coding in the map task corpus. In Johanna Moore and Marilyn Walker, editors, Empirical Methods in Discourse: Interpretation & Generation, AAAI Technical Report SS-95~06. AAAI Press, Menlo Park, CA. find
  33. Karlgren, Jussi. 1996. Stylistic variation in an information retrieval experiment. In Proceedings of the NeMLaP-2 Conference, Ankara, Turkey, September. find
  34. Karttunen, Lauri, Kimmo Koskenniemi, and Ronald M. Kaplan. 1987. A compiler for two-level phonological rules. In Mary Dalrymple, editor, Tools for Morphological Analysis. Center for the Study of Language and Information, Stanford, CA. find
  35. Kozima, Hideki. 1993. Text segmentation based on similarity between words. In Proceedings of the 31th Annual Meeting (Student Session), pages 286-288, Columbus, OH. Association for Computational Linguistics. find
  36. Kuno, Susumo. 1972. Functional sentence perspective: A case study from Japanese and English. Linguistic Inquiry, 3(3):269-320. find
  37. Kupiec, Julian, Jan Pedersen, and Francine Chen. 1995. A trainable document summarizer. In Proceedings of the 18th Annual International ACM/SIGIR Conference, pages 68-73, Seattle, WA. find
  38. Lakoff, George P. 1972. Structural complexity in fairy tales. The Study of Man, 1:128-150. find
  39. Lewis, David D. and Philip J. Hayes. 1994. Special issue on text categorization. Transactions of Office Information Systems, 12(3). find
  40. Litman, Diane J. and Rebecca J. Passonneau. 1995. Combining multiple knowledge sources for discourse segmentation. In Proceedings of the 33rd Meeting, pages 108-115, June. Association for Computational Linguistics. find
  41. Longacre, R. E. 1979. The paragraph as a grammatical unit. In Talmy Giv6n, editor, Syntax and Semantics: Discourse and Syntax, volume 12. Academic Press, New York, pages 115-134. find
  42. Mann, William C. and Sandra A. Thompson. 1987. Rhetorical structure theory: A theory of text organization. Technical Report ISI/RS 87-190, ISI. find
  43. Marchionini, Gary, Peter Liebscher, and Xia Lin. 1991. Authoring hyperdocuments: Designing for interaction. In Martin Dillon, editor, Interfaces for Information Retrieval and Online Systems. Greenwood Press, New York, NY, pages 119-131. find
  44. Miller, George A., Richard Beckwith, Christiane Fellbaum, Derek Gross, and Katherine J. Miller. 1990. Introduction to WordNet: An on-line lexical database. Journal of Lexicography, 3(4):235-244. find
  45. Mittendorf, Elke and Peter Scha'uble. 1994. Document and passage retrieval based on Hidden Markov Models. In Proceedings of the 17th Annual International ACM/SIGIR Conference, pages 318-327, Dublin, Ireland. find
  46. Moffat, Alistair, Ron Sacks-Davis, Ross Wilkinson, and Justin Zobel. 1994. Retrieval of partial documents. In Donna Harman, editor, Proceedings of the Second Text Retrieval Conference TREC-2. National Institute of Standards and Technology Special Publication 500-215, pages 181-190. find
  47. Mooney, David J., M. Sandra Carberry, and Kathleen E McCoy. 1990. The generation of high-level structure for extended explanations. In Proceedings of the Thirteenth International Conference on Computational Linguistics, volume 2, pages 276-281, Helsinki. find
  48. Moore, Johanna D. and Martha E. Pollack. 1992. A problem for RST: The need for multi-level discourse analysis. Computational Linguistics, 18(4):537-544.
  49. Morris, Jane. 1988. Lexical cohesion, the thesaurus, and the structure of text. Technical Report CSRI-219, Computer Systems Research Institute, University of Toronto. find
  50. Morris, Jane and Graeme Hirst. 1991. Lexical cohesion computed by thesaural relations as an indicator of the structure of text. Computational Linguistics, 17(1):21-48.
  51. Nakatani, Christine H., Barbara J. Grosz, David D. Ahn, and Julia Hirschberg. 1995. Instructions for annotating discourses. Technical Report TR-25-95, Harvard University Center for Research in Computing Technolog~ Cambridge, MA. find
  52. Nomoto, Tadashi and Yoshihiko Nitta. 1994. A grammatico-statistical approach to discourse partitioning. In Proceedings of the Fifteenth International Conference on Computational Linguistics (COLING), pages 1145-1150, Kyoto, Japan, August. find
  53. Paice, Chris D. 1990. Constructing literature abstracts by computer: Techniques and prospects. Information Processing and Management, 26(1):171-186. find
  54. Passonneau, Rebecca J. and Diane J. Litman. 1993. Intention-based segmentation: Human reliability and correlation with linguistic cues. In Proceedings of the 31st Annual Meeting, pages 148-155. find
  55. Phillips, Martin. 1985. Aspects of Text Structure: An Investigation of the Lexical Organisation of Text. North-Holland, Amsterdam. find
  56. Resnik, Philip. 1995. Using information content to evaluate semantic similarity in a taxonomy. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI-95), volume 1, pages 448-453, Montreal, Canada.
  57. Reynar, Jeffrey C. 1994. An automatic method of finding topic boundaries. In Proceedings of the 32nd Annual Meeting (Student Session), pages 331-333, Las Cruces, NM. Association for Computational Linguistics. find
  58. RosG Carolyn Penstein. 1995. Conversation acts, interactional structure, and conversational outcomes. In Johanna Moore and Marilyn Walker, editors, Empirical Methods in Discourse: Interpretation & Generation, AAAI Technical Report SS-95-06. AAAI Press, Menlo Park, CA. find
  59. Rumelhart, David. 1975. Notes on a schema for stories. In Daniel G. Bobrow and Allan Collins, editors, Representation and Understanding. Academic Press, New York, pages 211-236. find
  60. Salton, Gerard. 1989. Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer. find
  61. Salton, Gerard and James Allan. 1993. Selective text utilization and text traversal. In Proceedings of ACM Hypertext "93. find
  62. Salton, Gerard, James Allan, and Chris Buckley. 1993. Approaches to passage retrieval in full text information systems. In Proceedings of the 16th Annual International ACM/SIGIR Conference, pages 49-58, Pittsburgh, PA. find
  63. Salton, Gerard, Amit Singhal, Chris Buckley, and Mandar Mitra. 1996. Automatic text decomposition using text segmentation and text themes. In Proceedings of Hypertext '96, Seventh ACM Conference on Hypertext, pages 53-65, Washington, D.C. find
  64. Schiffrin, Deborah. 1987. Discourse Markers. find
  65. SchCitze, Hinrich. 1993. Word space. In Stephen J. Hanson, Jack D. Cowan, and C. Lee Giles, editors, Advances in Neural Information Processing Systems 5. Morgan Kaufmann, San Mateo, CA. find
  66. Skorochod'ko, E.E 1972. Adaptive method of automatic abstracting and indexing. In C.V. Freiman, editor, Information Processing 71: Proceedings of the IFIP Congress 71, pages 1179-1182. North-Holland Publishing Company. find
  67. Stark, Heather. 1988. What do paragraph markers do? Discourse Processes, 11(3):275-304. find
  68. Stoddard, Sally. 1991. Text and Texture: Patterns of Cohesion. Advances in Discourse Processes, volume XL. Ablex Publishing Corporation, Norwood, NJ. find
  69. Tannen, Deborah. 1989. Talking Voices: Repetition, Dialogue, and Imagery in Conversational Discourse. Studies in Interactional Sociolinguistics 6. Cambridge University Press. find
  70. Tombaugh, J., A. Lickorish, and P. Wright. 1987. Multi-window displays for readers of lengthy texts. International Journal of Man [sic] -Machine Studies, 26:597-615. van der Eijk, Pim. 1994. Comparative discourse analysis of parallel texts. Technical Report cmp-lg/9407022, Digital Equipment Corporat… find
  71. Macrostructures. Lawrence Erlbaum Associates, Hillsdale, N.J. van Dijk, Teun A. 1981. Studies in the Pragmatics of Discourse. Mouton, The Hague. find
  72. Walker, Marilyn A. 1992. Redundancy in collaborative dialogue. In Proceedings of the Fourteenth International Conference on Computational Linguistics (COLING), pages 345-351, Nantes, France, July. find
  73. Walker, Marilyn and Steve Whittaker. 1990. Mixed initiative dialogue: An investigation into discourse segmentation. In Proceedings of the 28th Annual Meeting, pages 70-78. Association for Computational Linguistics. find
  74. Webber, Bonnie Lynn. 1987. The interpretation of tense in discourse. In Proceedings of the 25th Annual Meeting, pages 147-154, Stanford, CA. Association for Computational Linguistics. find
  75. Webber, Bonnie Lynn. 1988. Discourse deixis: Reference to discourse segments. In Proceedings of the 26th Annual Meeting, pages 113-122, Buffalo, NY. Association for Computational Linguistics.
  76. Yarowsky, David. 1992. Word sense disambiguation using statistical models of Roget's categories trained on large corpora. In Proceedings of the Fourteenth International Conference on Computational Linguistics, pages 454-460, Nantes, France, July.
  77. Youmans, Gilbert. 1991. A new tool for discourse analysis: The vocabulary-management profile. Language, 67(4):763-789. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hearst-1997-text-tiling-segmenting-text-into/