Elamur Research
☰
home / papers / 2006 / Unsupervised Multilingual Sentence Boundary Detection

Unsupervised Multilingual Sentence Boundary Detection

What it is about

The authors build a sentence boundary detector that needs no annotated data and no language-specific resources. Most ambiguous periods come from abbreviations, so the system first finds abbreviations from an unannotated corpus. Three criteria decide this, and all three use only the candidate type, not its context: a truncated word with a final period is a tight collocation, abbreviations are usually short, and some contain internal periods. The same collocation evidence also helps with initials and ordinal numbers. The system was developed on English and tested on eleven languages with newspaper corpora. Without any change to the system, the average error rate for sentence boundaries is 1.26%, ranging from 0.35% on German to 2.12% on Estonian, and the average error rate for abbreviation detection is 0.80%.

Ideas

For Mova

The code is en/src/tok.rs, with the metric from en tok-eval. The chapter is languages/en/07-punctuation-capitals-numbers.md. The collocation statistics come from a corpus, and the paper's own corpora are news in other languages, so the numbers are not Mova's. Use the English part only as a check. Measure first on EWT test, with a two-fold check (en folds).

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/J06-4003.pdf

Summary

Periods are ambiguous. Besides ending sentences, they mark abbreviations, initials, ordinal numbers and ellipses, and a single period can play two roles at once. Kiss and Strunk propose Punkt, an unsupervised, language-independent sentence boundary detector. It learns everything it needs from the corpus being segmented, with no POS tags, abbreviation lists or annotated training data. It also does not lean on capitalization, so it can handle single-case text.

The core idea is to identify abbreviations first, treating each as an extremely strong collocation of a truncated word and its period. A type-level classifier scores candidates with a modified Dunning log-likelihood ratio, weighted by word length and the presence of internal periods. A token-level stage then applies further collocational heuristics to find sentence-final abbreviations, initials and ordinal numbers.

The system was developed on English and applied unchanged to ten more languages and several genres. It reaches 98.74% mean accuracy on newspaper text. It slightly trails the best published systems but needs no manual adaptation, which makes it practical for under-resourced languages and new domains.

Key points

  • Testing covered eleven languages in total: English plus Turkish, Swedish, Spanish, Norwegian, Italian, German, French, Estonian, Dutch and Brazilian Portuguese. The mean sentence boundary accuracy on newspaper corpora is 98.74%.
  • Abbreviations can account for up to roughly 30% of the candidate boundaries in running text. Punkt detects them with 99.38% mean accuracy using three context-free cues: a tight bond with the final period, short length and internal periods. With the extra handling for initials and ordinals, overall accuracy is 99.20%.
  • The two-stage architecture first produces intermediate tags <A>, <E> and <S> at the type level. A token-based pass then corrects these tags and adds combined labels such as <A><S>.
  • Compared with Mikheev's system on WSJ (Penn Treebank sections 03-06) and Brown, Punkt's boundary error rates are 1.65% and 1.02%. Its abbreviation error rates are 0.71% and 0.82%, lower than Mikheev's results even when he used an abbreviation list.
  • Abbreviations form a productive, open class, so fixed lists or length caps fail. In dictionary-derived lists, 28% of German and 26% of English abbreviation types are longer than four characters.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.1162/coli.2006.32.4.485

@article{kiss2006unsupervised,
  title = {{Unsupervised Multilingual Sentence Boundary Detection}},
  author = {Kiss, Tibor and Strunk, Jan},
  journal = {Computational Linguistics},
  year = {2006},
  doi = {10.1162/coli.2006.32.4.485},
  url = {https://doi.org/10.1162/coli.2006.32.4.485}
}

References

26 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aluisio, Sandra M., Gisele M. Pinheiro, Marcelo Finger, Maria das Graças V. Nunes, and Stella E. O. Tagnin. 2003. The Lacio-Web Project: Overview and issues in Brazilian Portuguese corpora creation. In Proceedings of Corpus Linguistics, pages 14–21, Lancaster, UK. find
  2. Brill, Eric. 1995. Transformation-based error-driven learning and natural language processing: A case study in part-of-speech tagging. Computational Linguistics, 21(4):543–565. find our card
  3. Dudenredaktion, editor. 2004. Duden. Die deutsche Rechtschreibung (23rd ed.), volume 1 of Der Duden in zwölf Bänden. Dudenverlag, Mannheim. find
  4. Dunning, Ted. 1993. Accurate methods for the statistics of surprise and coincidence. Computational Linguistics, 19(1):61–74. find our card
  5. Evert, Stefan and Brigitte Krenn. 2001. Methods for the qualitative evaluation of lexical association measures. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics, pages 188–195, Toulouse, France. find
  6. Firth, John R. 1957. A synopsis of linguistic theory 1930-55. In Studies in Linguistic Analysis (special volume of the Philological Society). The Philological Society, Oxford, pages 1–32. find
  7. Francis, Nelson W. and Henry Kucera. 1982. Frequency Analysis of English Usage: Lexicon and Grammar. Houghton Mifflin, New York. find
  8. Grefenstette, Gregory. 1999. Tokenization. In Hans van Halteren, editor, Syntactic Wordclass Tagging. Kluwer Academic Publishers, Dordrecht, pages 117–133. find
  9. Grefenstette, Gregory and Pasi Tapanainen. 1994. What is a word, what is a sentence? Problems of tokenization. In Proceedings of the 3rd International Conference on Computational Lexicography, pages 79–87, Budapest, Hungary. find
  10. Kiss, Tibor and Jan Strunk. 2002a. Scaled log likelihood ratios for the detection of abbreviations in text corpora. In Proceedings of COLING 2002, pages 1228–1232, Taipei, Taiwan. find
  11. Kiss, Tibor and Jan Strunk. 2002b. Viewing sentence boundary detection as collocation identification. In Proceedings of KONVENS 2002, pages 75–82, Saarbrücken, Germany. find
  12. Manning, Christopher D. and Hinrich Schütze. 1999. Foundations of Statistical Natural Language Processing. The MIT Press, Cambridge, MA. find
  13. Marcus, Mitchell P., Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. find our card
  14. Mikheev, Andrei. 2000. Tagging sentence boundaries. In Proceedings of ANLP-NAACL 2000, pages 264–271, Seattle, Washington. find
  15. Mikheev, Andrei. 2002. Periods, capitalized words, etc. Computational Linguistics, 28(3):289–318. find
  16. Müller, Hans, V. Amerl, and G. Natalis. 1980. Worterkennungsverfahren als Grundlage einer Universalmethode zur automatischen Segmentierung von Texten in Sätze. Ein Verfahren zur maschinellen Satzgrenzenbestimmung im Englischen. Sprache und Datenverarbeitung, 4(1):46–64. find
  17. Nunberg, Geoffrey. 1990. The Linguistics of Punctuation, volume 18 of CSLI Lecture Notes. CSLI Publications, Stanford, CA. find
  18. Palmer, David D. and Marti A. Hearst. 1997. Adaptive multilingual sentence boundary disambiguation. Computational Linguistics, 23(2):241–267. find
  19. Reynar, Jeffrey C. and Adwait Ratnaparkhi. 1997. A maximum entropy approach to identifying sentence boundaries. In Proceedings of the Fifth ACL Conference on Applied Natural Language Processing, pages 16–19, Washington, DC. find
  20. Riley, Michael D. 1989. Some applications of tree-based modeling to speech and language indexing. In Proceedings of the DARPA Speech and Natural Language Workshop, pages 339–352, Cape Cod, MA. find
  21. Silla Jr., Carlos N. and Celso A. A. Kaestner. 2004. An analysis of sentence boundary detection systems for English and Portuguese documents. In Proceedings of CICLing 2004, pages 135–141, Seoul, Korea. find
  22. Silla Jr., Carlos N., Jaime Dalla Valle Jr., and Celso A. A. Kaestner. 2003. Detecção automática de sentenças com o uso de expressões regulares. In Proceedings of CBComp 2003, pages 548–560, Itajaı́, Brazil. find
  23. Stamatatos, Efstathios, Nikos Fakotakis, and George K. Kokkinakis. 1999. Automatic extraction of rules for sentence boundary disambiguation. In Proceedings of the Workshop on Machine Learning in Human Language Technology, pages 88–92, Chania, Greece. find
  24. Strunk, Jan, Carlos N. Silla Jr., and Celso A. A. Kaestner. 2006. A comparative evaluation of a new unsupervised sentence boundary detection approach on documents in English and Portuguese. In Proceedings of CICLing 2006, pages 132–143, Mexico City, Mexico. van Rijsbergen, Cornelis J. 1979. Informat… find
  25. Walker, Daniel J., David E. Clements, Maki Darwin, and Jan W. Amtrup. 2001. Sentence boundary detection: A comparison of paradigms for improving MT quality. In Proceedings of the MT Summit VIII, Santiago de Compostela, Spain. find
  26. Willmann, Helmut and Heinz Messinger, editors. 1996. Langenscheidts Großwörterbuch Englisch. Der Kleine Muret-Sanders (7th ed.), volume I: English– German. Langenscheidt, Berlin/Munich. find