Elamur Research⌕
☰
home / papers / 2020 / PySBD: Pragmatic Sentence Boundary Disambiguation

PySBD: Pragmatic Sentence Boundary Disambiguation

Summary

Sentence boundary disambiguation (SBD) is often treated as solved. Yet text from PDFs, OCR, HTML, and medical, legal or scientific sources uses punctuation that confuses naive splitters and models tuned on newswire such as WSJ/Penn Treebank. The authors present PySBD, an open-source Python package that ports and extends the Ruby pragmatic_segmenter. It relies entirely on regular expressions.

PySBD works in three stages. First, it temporarily swaps non-terminal punctuation for Unicode placeholders. Next, a simple regex splits the text. Finally, the original characters are restored. Rules are grouped into shared (Common, Standard) and language-specific sets covering abbreviations, lists, exclamations and bracketed punctuation. An optional Cleaner handles noisy input, and a char-span mode returns offsets into the unaltered text. Development was test-driven against per-language Golden Rules Sets (GRS).

PySBD reaches 97.92% on the English GRS and 97.00% on GENIA, outperforming the other tools tested, though it runs slower than most of them. Its main appeal is that the rules are transparent, can be corrected incrementally and can be extended to new languages without any training corpus.

Key points

  • Supported languages: 22 languages, each with its own GRS. The English GRS contains 48 hand-built exemplars from formal and informal domains.
  • English GRS accuracy: PySBD scores 97.92%. Next is blingfire at 75.00%, then stanza (72.92), syntok (68.75), NLTK (56.25), spacy dep (54.17) and spaCy (52.08).
  • GENIA accuracy: PySBD scores 97.00%, ahead of NLTK (87.95) and blingfire (86.95). The authors attribute the remaining errors to PySBD splitting bulleted items that GENIA keeps as single sentences.
  • Speed: On "The Adventures of Sherlock Holmes", PySBD takes about 9.5 s. This is slower than blingfire (~85 ms), NLTK and spaCy, but faster than spacy dep (~27 s) and stanza (~48 s).
  • Multilingual evaluation: PySBD was tested on OPUS-100 test sets for 21 non-English languages, with 2000 sentences each (the Armenian train set was used instead). Scores range from 96.45% (Japanese) to 48.05% (Burmese), with Polish, Kazakh and Armenian also weak. Ukrainian is not among the supported languages, but a new language can be added by writing its GRS, abbreviations and punctuation rules. The authors report that Marathi took about one hour.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

35 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 34 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.nlposs-1.15 · arXiv 2010.09657

@inproceedings{sadvilkar2020pysbd,
  title = {{PySBD: Pragmatic Sentence Boundary Disambiguation}},
  author = {Sadvilkar, Nipun and Neumann, Mark},
  booktitle = {Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)},
  year = {2020},
  doi = {10.18653/v1/2020.nlposs-1.15},
  eprint = {2010.09657},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.nlposs-1.15}
}

References

14 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Steven Bird. 2006. Nltk: The natural language toolkit. ArXiv, cs.CL/0205028. find
  2. Beyond Language Understanding Bling Team, Microsoft. 2020. Blingfire : A lightning fast finite state machine and regular expression manipulation library. https://github.com/microsoft/ BlingFire/tree/master/ldbsrc/sbd . ↗
  3. Kilian Evang, Valerio Basile, Grzegorz Chrupała, and Johan Bos. 2013. Elephant: Sequence labeling for word and sentence segmentation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1422–1426, Seattle, Washington, USA. Association for Computational Li… find
  4. William A. Gale and Kenneth W. Church. 1993. A program for aligning sentences in bilingual corpora. Computational Linguistics, 19(1):75–102. find our card
  5. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. https://github.com/explosion/ spaCy . To appear. ↗
  6. Jin-Dong Kim, Tomoko Ohta, Yuka Tateisi, and Jun’ichi Tsujii. 2003. Genia corpus - a semantically annotated corpus for bio-textmining. Bioinformatics, 19 Suppl 1:i180–2. find
  7. Tibor Kiss and Jan Strunk. 2006. Unsupervised multilingual sentence boundary detection. Computational Linguistics, 32(4):485–525. find our card
  8. Florian Leitner. 2020. Syntok : Text tokenization and sentence segmentation (segtok v2). https://github.com/fnl/syntok . ↗
  9. Andrei Mikheev. 2002. Periods, capitalized words, etc. Computational Linguistics, 28(3):289–318. find
  10. David D. Palmer and Marti A. Hearst. 1997. Adaptive multilingual sentence boundary disambiguation. Computational Linguistics, 23(2):241–267. find
  11. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In ACL. find our card
  12. Jonathon Read, Rebecca Dridan, Stephan Oepen, and Lars Jørgen Solberg. 2012. Sentence boundary detection: A long solved problem? Computational Linguistics, pages 985–994. find our card
  13. Sebastian Ruder. 2020. Why You Should Do NLP Beyond English. http://ruder.io/ nlp-beyond-english . ↗
  14. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In LREC. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sadvilkar-2020-pysbd-pragmatic-sentence-boundary-disambiguation/