Elamur Research⌕
☰
home / papers / 1997 / Adaptive Multilingual Sentence Boundary Disambiguation

Adaptive Multilingual Sentence Boundary Disambiguation

Paper · 1997 · David D. Palmer and Marti A. Hearst

Summary

Splitting text into sentences is a prerequisite for taggers, parsers and sentence aligners. However, periods, question marks and exclamation points are ambiguous: a period can end a sentence, mark an abbreviation or decimal, or do both at once. Rule-based tokenizers built from regular expressions and abbreviation lists can work well, but they need months of manual tuning per corpus. They also fail on all-caps or OCR-damaged text.

Palmer and Hearst introduce Satz, a trainable disambiguator. For each candidate mark, it represents the neighbouring tokens by rough part-of-speech estimates from a small lexicon instead of brittle cues like capitalization. A learned classifier then labels the mark as a boundary or not. Satz trains in under a minute and segments about 10,000 sentences per minute. Its error rate stayed below 1.5% across corpora, and it was ported easily from English to French and German. It offers a cheap, portable alternative to hand-written rules and can also repair difficult cases left by a rule-based system.

Key points

  • Satz needs only a lexicon that can stay under 5,000 words and a training corpus of roughly 300–500 sentences.
  • The "lower bound" baseline labels every candidate mark as a sentence end, and its score depends heavily on genre. It is about 53% for the Wall Street Journal corpus, where 47% of periods are abbreviations, versus about 90% for the Brown corpus.
  • The UNIX STYLE program, which relies on a 48-entry abbreviation list, served as a comparison baseline. Its developers reported a 6.3% error rate (204 errors out of 3,287 candidate marks).
  • Hand-built rule systems required heavy effort:
    • Alembic used 75 abbreviations and over 100 rules developed over more than six staff months, reaching 0.9% error on WSJ.
    • Wasson's finite-automaton recognizer took nine staff months to build.
  • The system was evaluated on English, French and German, with error consistently below 1.5%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Palo Alto Research CenterPalo Alto · United States14 works · 8,123 citations🏢MitreBedford · United States10 works · 2,631 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 worksTokenization & segmentation · 119 works

Cite

Cite as David D. Palmer and Marti A. Hearst. 1997. Adaptive Multilingual Sentence Boundary Disambiguation. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@article{palmer1997adaptive,
  title = {{Adaptive Multilingual Sentence Boundary Disambiguation}},
  author = {David D. Palmer and Marti A. Hearst},
  journal = {Computational Linguistics},
  year = {1997},
  doi = {10.5555/972695.972697},
  url = {https://doi.org/10.5555/972695.972697}
}

References

34 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aberdeen, John, John Burger, David Day, Lynette Hirschman, Patricia Robinson, and Marc Vilain. 1995. MITRE: Description of the Alembic system used for MUC-6. In Proceedings of the Sixth Message Understanding Conference (MUC-6), Columbia, MD, November. find
  2. Bahl, L. R., P. E Brown, P. V. deSouza, and R. L. Mercer. 1989. A tree-based statistical language model for natural language speech recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing, 36(7):1001-1008. find
  3. Bourland, Herv~ and Nelson Morgan. 1994. Connectionist Speech Recognition: A Hybrid Approach. Kluwer Academic Publishers, Norwell, MA. find
  4. Breiman, Leo, Jerome H. Friedman, Richard Olshen, and Charles J. Stone. 1984. Classification and Regression Trees. Wadsworth International Group, Belmont, CA. find
  5. Cherry, L. L. and W. Vesterman. 1991. Writing tools — the STYLE and DICTION programs. In 4.3BSD UNIX System Documentation. University of California, Berkeley. find
  6. Church, Kenneth W. 1988. A stochastic parts program and noun phrase parser for unrestricted text. In Second Conference on Applied Natural Language Processing, pages 136-143, Austin, TX.
  7. Church, Kenneth W. and Mark Y. Liberman. 1991. A status report on the ACL/DCI. In Proceedings of the 7th Annual Conference of the UW Centre for the New OED and Text Research: Using Corpora, pages 84-91, Oxford. find
  8. Cutting, Doug, Julian Kupiec, Jan Pedersen, and Penelope Sibun. 1991. A practical part-of-speech tagger. In The 3rd Conference on Applied Natural Language Processing, Trento, Italy.
  9. Francis, W. Nelson and Henry Kucera. 1982. Frequency Analysis of English Usage. Houghton Mifflin Co., New York. find
  10. Fung, Pascale and Kenneth Ward Church. 1994. K-vec: A new approach to aligning parallel texts. In Proceedings of COLING-94. find
  11. Fung, Pascale and Kathleen McKeown. 1994. Aligning noisy parallel corpora across language groups: Word pair feature matching by dynamic time warping. In Proceedings of the First Conference of the Association for Machine Translation in the Americas AMTA-94, pages 81-88, Columbia, MD. find
  12. Gale, William A. and Kenneth W. Church. 1993. A program for aligning sentences in bilingual corpora. Computational Linguistics, 19(1):75-102.
  13. Hertz, John, Anders Krogh, and Richard G. Palmer. 1991. Introduction to the Theory of Neural Computation. Santa Fe Institute Studies in the Sciences of Complexity. Addison-Wesley Publishing Co., Redwood City, CA. find
  14. Hoffmann, Christiane. 1994. Automatische Disambiguierung von Satzgrenzen in einem maschinenlesbaren deutschen Korpus. Manuscript, University of Trier, Germany. find
  15. Humphrey, T. L. and E-q. Zhou. 1989. Period disambiguation using a neural network. In IJCNN: International Joint Conference on Neural Networks, page 606, Washington, D.C. find
  16. Kay, Martin and Martin ROscheisen. 1993. Text-translation alignment. Computational Linguistics, 19(1):121-142.
  17. Lesk, M. E. and E. Schmidt. 1975. Lex — a lexical analyzer generator. Computing Science Technical Report 39, AT&T Bell Laboratories, Murray Hill, NJ. find
  18. Liberman, Mark Y. and Kenneth W. Church. 1992. Text analysis and word pronunciation in text-to-speech synthesis. In Sadaoki Furui and Man Mohan Sondhi, editors, Advances in Speech Signal Processing. Marcel Dekker, Inc., pages 791-831. find
  19. Lippmann, R. P. 1989. Review of neural networks for speech recognition. Neural Computation, 1:1-38. find
  20. Magerman, David M. 1995. Statistical decision-tree models for parsing. In Proceedings of the 33rd Annual Meeting. Association for Computational Linguistics.
  21. Mtiller, Hans, V. Amerl, and G. Natalis. 1980. Worterkennungsverfahren als Grundlage einer Universalmethode zur automatischen Segmentierung von Texten in S~itze. Ein Verfahren zur maschinellen Satzgrenzenbestimmung im Englischen. Sprache und Datenverarbeitung, 1. find
  22. Nakamura, Masami, Katsuteru Maruyama, Takeshi Kawabata, and Kiyohiro Shikano. 1990. Neural network approach to word category prediction for English texts. In Proceedings of COLING-90, volume 3, pages 213-218.
  23. Nicol, G. T. 1993. Flex — The Lexical Scanner Generator. The Free Software Foundation, Cambridge, MA. find
  24. Nunberg, Geoffrey. 1990. The Linguistics of Punctuation. C.S.L.I. Lecture Notes, Number 18. Center for the Study of Language and Information, Stanford, CA. find
  25. Palmer, David D. 1995. Experiments in multilingual sentence boundary recognition. In Proceedings of Recent Advances in Natural Language Processing, Velingrad, Bulgaria, September 1995. find
  26. Palmer, David D. and Marti A. Hearst. 1995. Adaptive sentence boundary disambiguation. In Proceedings of the Fourth ACL Conference on Applied Natural Language Processing, pages 78-83.
  27. Morgan Kaufmann, October 1994. find
  28. Quinlan, J. Ross. 1986. Induction of decision trees. Machine Learning, 1(1):81-106. find
  29. Quinlan, J. Ross. 1993. c4.5: Programs for Machine Learning. Morgan Kaufman, San Mateo, CA.
  30. Resnik, Philip. 1993. Semantic classes and syntactic ambiguity. In Proceedings of the ARPA Workshop on Human Language Technology (March 1993). find
  31. Riley, Michael D. 1989. Some applications of tree-based modelling to speech and language indexing. In Proceedings of the DARPA Speech and Natural Language Workshop, pages 339-352. Morgan Kaufmann. find
  32. Schmid, Helmut. 1994. Part-of-speech tagging with neural networks. In Proceedings of COLING-94. find
  33. Siegel, Eric V. and Kathleen R. McKeown. 1994. Emergent linguistic rules from inducing decision trees: Disambiguating discourse clue words. In Proceedings of AAA11994, pages 820-826. AAAI Press/MIT Press, July. find
  34. Soderland, Stephen and Wendy Lehnert. 1994. Corpus-driven knowledge acquisition for discourse analysis. In Proceedings of AAA11994, pages 827-832. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/palmer-1997-adaptive-multilingual-sentence-boundary-disambiguation/