Elamur Research⌕
☰
home / papers / 2021 / Indirectly Supervised English Sentence Break Prediction Using Paragraph Break Probability Estimates

Indirectly Supervised English Sentence Break Prediction Using Paragraph Break Probability Estimates

Paper · 2021 · Robert C. Moore

Summary

Detecting where English sentences end usually depends on hand-annotated data. Such data is scarce, and corpora disagree about what counts as a boundary. Moore tests whether a free, abundant signal can replace most of that annotation: the places where paragraphs end in ordinary text, since every paragraph end is also a sentence end. After normalizing and tokenizing a large unannotated corpus, he builds N-gram models that estimate how likely a paragraph break is at each candidate position. He then converts these estimates into sentence break decisions. Only two hyperparameters are tuned on a small annotated sample. This almost unsupervised predictor already reaches high accuracy. Results improve further, both in-domain and out-of-domain, when the paragraph-break probability is mapped to sentence-break log odds and given as a feature to an SVM trained on a larger annotated set. The report also covers how to define and score sentence breaks, breaks with no whitespace after them, and corrections to the reference annotations. Its broader lesson is that structure already present in raw text can serve as indirect supervision.

Key points

  • Annotated corpora used and compared: Penn Treebank WSJ, Satz WSJ and English Web Treebank. Each follows its own conventions for what counts as a sentence break, and the report discusses which breaks it actually tests on.
  • Earlier studies covered in the review include those by Riley (1989), by Palmer and Hearst (1997), by Kiss and Strunk (2006), and by Gillick (2009).
  • Two main systems are compared:
    • a predictor driven almost entirely by paragraph break probability (PBP) estimates;
    • an SVM classifier enriched with a PBP signal, measured against a plain SVM baseline.
  • Evaluation is discussed in depth because sentence break candidates are hard to count. The report examines F1, Fβ, and errors per sentence break.
  • Further topics include:
    • using real paragraph breaks at prediction time (after colons, in itemized lists, at hard line breaks, in verse titles);
    • predicting breaks where the space is missing;
    • how the amount of annotated data affects results;
    • a new annotation standard for breaks after ellipses, with re-evaluation on corrected test sets.

Interesting ideas

  • I1. A layout cue that naturally occurs in raw text (paragraph ends) is a guaranteed subset of the target label (sentence ends), so it can provide training signal at corpus scale with no annotation.
  • I2. The weakly supervised probability is not used as a replacement for the classifier; it is converted into log odds and added as one feature to a supervised model, which then decides how much to trust it.
  • I3. When the set of negative candidates is fuzzy, the report normalizes errors by the number of true breaks and uses Fβ to weight precision and recall unequally, avoiding a misleading accuracy figure.
  • I4. After error analysis, the gold annotations themselves are corrected, and the new convention (for breaks after ellipses) is written down before rescoring.
  • I5. The amount of tuning data to annotate is reduced by stating explicit assumptions about which cases could actually change the hyperparameter choice.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): the sentence splitter in crate en could learn boundary probabilities from paragraph breaks in large raw English text, keeping a small annotated set only for tuning thresholds.
  • I2 (learn): agreement scores from the witness parsers could be passed to downstream learners as calibrated log-odds features instead of being used as hard silver labels.
  • I3 (learn): segmentation and attachment evaluations could report errors per true unit and an explicit β, so that systems with different candidate definitions remain comparable.
  • I4 (tree): treebank dialect conversion could log each disputed convention as an explicit rule and rescore parsers on the corrected gold, so annotation noise is not mistaken for parser error.
  • I5 (learn): LLM-teacher or human checking could be limited to items near the decision boundary of a threshold, which lowers annotation cost when tuning repair rules.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1Research subjects and languages the work is about.English · 1,373 works

Cite

Cite as Robert C. Moore. 2021. Indirectly Supervised English Sentence Break Prediction Using Paragraph Break Probability Estimates. arXiv:2109.12023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2109.12023 · arXiv 2109.12023

@misc{moore2021indirectly,
  title = {{Indirectly Supervised English Sentence Break Prediction Using Paragraph Break Probability Estimates}},
  author = {Robert C. Moore},
  year = {2021},
  doi = {10.48550/arxiv.2109.12023},
  eprint = {2109.12023},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2109.12023}
}

References

21 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. ACL/DCI. 1993. ACL/DCI LDC93T1. Web Download. Philadelphia: Linguistic Data Consortium. find
  2. Yoav Benjamini and Yosef Hochberg. 1995. Controling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57(1):289–300. find
  3. Michele Banko and Eric Brill. 2001. Scaling to very very large corpora for natural language disambiguation. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics, July 9–11, Toulouse, France. 26–33.
  4. Ann Bies, Justin Mott, Colin Warner, and Seth Kulick. 2012. English Web Treebank LDC2012T13. Web Download. Philadelphia: Linguistic Data Consortium. find
  5. Koby Crammer and Yoram Singer. 2001. On the algorithmic implementation of multiclass kernel-based vector machines. Journal of Machine Learning Research, 2:265–292. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. (Submitted on 11 Oct 2018 (v1), last revised 24 May 2019 (this version, v2)) https://arxiv.org/abs/1810.04805 ↗
  7. Dan Gillick. 2009. Sentence boundary detection and the problem with the U.S. In Proceedings of NAACL HLT 2009: Short Papers, May 31–June 5, Boulder, Colorado, USA. 241–244.
  8. Thorsten Joachims. 1999. Making large-scale support vector machines practical. In Advances in Kernel Methods: Support Vector Learning, The MIT Press, Cambridge, Massachusetts, USA. 169–184. find
  9. Tibor Kiss and Jan Strunk. 2006. Unsupervised multilingual sentence boundary detection. Computational Linguistics, 32(4):485–525. ↗
  10. John Laury, Ann Bies, Justin Mott, and Colin Warner. 2011. Sentence splitting guidelines for web data. Distributed with English Gigaword Fifth Edition LDC2011T07. Web Download. Philadelphia: Linguistic Data Consortium. find
  11. Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157. ↗
  12. Mitchell P. Marcus, Beatrice Santorini, and Mary A. Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. ↗
  13. Mitchell P. Marcus, Beatrice Santorini, and Mary A. Marcinkiewicz. 1995. Treebank-2 LDC95T7. Web Download. Philadelphia: Linguistic Data Consortium. find
  14. David D. Palmer and Marti A. Hearst. 1997. Adaptive multilingual sentence boundary disambiguation. Computational Linguistics, 23(2):242–267. ↗
  15. Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2011. English Gigaword Fifth Edition LDC2011T07. Web Download. Philadelphia: Linguistic Data Consortium. find
  16. Jonathan Read, Rebecca Dridan, Stephan Oepen, Lars J. Solberg. 2012. Sentence boundary detection: a long solved problem? In Proceedings of Coling 2012: Posters, December 8–15, Mumbai, India. 985–994.
  17. Jeffrey C. Reynar and Adwait Ratnaparkhi. 1997. A maximum entropy approach to identifying sentence boundaries. In Proceedings of the Fifth Conference on Applied Natural Language Processing, March 31–April 3, Washington, DC, USA. 16–19.
  18. Michael D. Riley. 1989. Some applications of tree-based modelling to speech and language. In HLT ’89 Proceedings of the workshop on Speech and Natural Language, October 15–18, Cape Cod, Massachusetts, USA. 339–352. find
  19. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. (Submitted on 12 Jun 2017 (v1), last revised 6 Dec 2017 (this version, v5)) https://arxiv.org/abs/1706.03762 ↗
  20. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s neural machine translation system: bridging the gap between human and machine translation. (Submitted on 26 Sep 2016 (v1), last revised 8 Oct 2016 (this version, v2)) https://arxiv.org/abs/1609.08144 ↗
  21. Tong Zhang. 2004. Solving large scale linear prediction problems using stochastic gradient descent algorithms. In Proceedings of the 21st International Conference on Machine Learning, July 4–8, Banff, Alberta, Canada, 919–926. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/moore-2021-indirectly-supervised-english-sentence-break/