Elamur Research⌕
☰
home / papers / 2023 / Parsing “Early English Books Online” for Linguistic Search

Parsing “Early English Books Online” for Linguistic Search

Paper · 2023 · Seth Kulick, Neville Ryant and Beatrice Santorini

Summary

Historical treebanks such as PPCEME are too small to show how rare syntactic constructions change over time. The 1.5-billion-word EEBO collection is large enough, but it has no annotation. Earlier work trained a constituency parser on PPCEME and tested it there, but EEBO differs in spelling, illegible characters and sentence segmentation, so those scores may not carry over. The authors use the fact that 42 PPCEME source texts also appear in EEBO. They align the two versions of this shared material token by token with a Smith-Waterman procedure tuned for period spelling. The alignment lets the gold PPCEME annotation act as a reference for the parsed EEBO text. A Berkeley Neural Parser with RoBERTa further pretrained on EEBO is trained on the non-overlapping part of PPCEME. It is then scored on both versions of the overlap for POS accuracy and for retrieval of six linguistically relevant sentence types, using an improved alignment-mediated query evaluation. Retrieval is worse on EEBO, mainly because its sentences are segmented differently. This points to segmentation as the next thing to fix.

Key points

  • Shared material: 42 of PPCEME's 232 source texts have EEBO counterparts, about 42% of PPCEME and roughly 805K tokens. The alignment covers 98.9% of PPCEME tokens and 97.9% of EEBO tokens.
  • Segmentation gap: EEBO is split automatically only on periods, question marks and exclamation marks, while PPCEME was split manually by linguistic criteria. Mean sentence length is 28.68 tokens in EEBO versus 20.44 in PPCEME.
  • Aligner tuning: Smith-Waterman alignment uses a Levenshtein-based similarity.
    • Substitution cost is lowered for the EEBO bullet (0.1) and for u/v alternation (0.2).
    • Systematic pairs such as &/and and the/ye are forced to count as identical.
  • Parser: Berkeley Neural Parser v0.2.0 (Kitaev et al. span-based, CKY decoding, joint POS classifier). ELMo is replaced by roberta-base with two extra pretraining epochs on EEBO.
    • Cross-validated test evalb is 91.77 and POS accuracy is 98.37.
    • On the PPCEME overlap, evalb is 91.64.
  • Planned work: a TAG-style parser, since 233 nonsensical structures appear even on PPCEME. Other plans include a segmenter trained on PPCEME, dependency conversion, and a "fuzzy" evalb.

Interesting ideas

  • I1. When two editions of one text exist, aligning them carries gold labels onto the unannotated version, giving an in-domain test set without new annotation.
  • I2. An edit-distance aligner gets much more reliable when known systematic variants (spelling alternations, unreadable-character markers) are made cheap or treated as equal.
  • I3. Parser quality is judged by whether the trees find the specific constructions users search for, not only by bracket overlap.
  • I4. The authors plan to repair one corpus's segmentation using the alignment, so that segmentation error can be measured apart from parsing error.
  • I5. An alignment-based "fuzzy" evalb could match constituents even when tokens and sentence boundaries differ, with partial credit for imperfect matches.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Align gold UD treebank sentences with noisy versions of the same text (OCR, web, other editions) to measure how much Mova degrades on messy input.
  • I2 (tokens): Use a small table of cheap-substitution pairs to map variant spellings onto standard forms before lemma lookup, and keep the mapping so it can be explained.
  • I3 (learn): Add construction-level probes (e.g. passives, relative clauses, questions) to evaluation, so a parser change is judged by its effect on frames that later stages rely on.
  • I4 (tree): Re-run the parser with gold sentence boundaries forced in, so the score drop caused by segmentation is reported separately from attachment errors.
  • I5 (tree): A similar alignment-aware LAS would let gparse be scored against a gold tree when Mova's tokenization differs from the treebank's.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PennsylvaniaPhiladelphia · United States221 works · 63,234 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksLiterature and Literary Theory · 519 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Seth Kulick, Neville Ryant and Beatrice Santorini. 2023. Parsing “Early English Books Online” for Linguistic Search. Proceedings of the Society for Computation in Linguistics 2023.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{kulick2023parsing,
  title = {{Parsing “Early English Books Online” for Linguistic Search}},
  author = {Seth Kulick, Neville Ryant and Beatrice Santorini},
  booktitle = {Proceedings of the Society for Computation in Linguistics 2023},
  year = {2023},
  doi = {10.7275/kr54-n102},
  url = {https://doi.org/10.7275/kr54-n102}
}

References

25 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In International conference on learning representations. find
  2. Michele Bevilacqua, Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. 2021. Recent trends in word sense disambiguation: A survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. International Joint Conference on Artificial Intelligence, Inc. find
  3. Aaron Ecay. 2015. A multi-step analysis of the evolution of English do-support. Ph.D. thesis, University of Pennsylvania. find
  4. Ryan Gabbard, Seth Kulick, and Mitchell Marcus. 2006. Fully parsing the Penn Treebank. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 184–191, New York, New York. Association for Computational Linguistics.
  5. Charlotte Galves. 2020. Relaxed V-Second in Classical Portuguese. In Rebecca Woods and Sam Wolfe, https://www.oed.com editors, Rethinking Verb-Second, pages 368–395. Oxford University Press. Reenen, editors, Textual parameters in older language, pages 353–391. Benjamins. ↗
  6. Charlotte Galves, Aroldo Leal de Andrade, and Pablo Faria. 2017. Tycho Brahe Parsed Corpus of Historical Portuguese. http://www.tycho.iel.unicamp.br/ ~tycho/corpus/texts/psd.zip. ↗
  7. Seth Kulick, Neville Ryant, and Beatrice Santorini. 2022a. Penn-Helsinki Parsed Corpus of Early Modern English: First parsing results and analysis. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 578–593, Seattle, Washington. Association for Computational Linguistics.
  8. Andres Karjus. 2020. Competition, selection and communicative need in language change: An investigation using corpora, computational modelling and experimentation. Ph.D. thesis, University of Edinburgh. ↗
  9. Jungo Kasai, Robert Frank, Pauli Xu, William Merrill, and Owen Rambow. 2018. End-to-end graph-based TAG parsing with neural networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1181–1194, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  10. Nikita Kitaev, Steven Cao, and Dan Klein. 2019. Multilingual constituency parsing with self-attention and pre-training. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3499–3505, Florence, Italy. Association for Computational Linguistics. ↗
  11. Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2676–2686, Melbourne, Australia. Association for Computational Linguistics. ↗
  12. Anthony Kroch. 1989. Reflexes of grammar in patterns of language change. Language Variation and Change, 1(3):199–244. ↗
  13. Anthony Kroch. 2020. Penn Parsed Corpora of Historical English. LDC2020T16 Web Download. Philadelphia: Linguistic Data Consortium. Contains Penn-Helsinki Parsed Corpus of Middle English, second edition, Penn-Helsinki Parsed Corpus of Early Modern English, and Penn Parsed Corpus of Modern British English. find
  14. Anthony Kroch and Beatrice Santorini. 2021. Penn-BFM Parsed Corpus of Historical French, version 1.0. https://github.com/beatrice57/ mcvf-plus-ppchf. ↗
  15. Anthony Kroch, Beatrice Santorini, and Lauren Delfs. 2004. Penn-Helsinki Parsed Corpus of Early Modern English (PPCEME). CD-ROM, first edition, release 3. http://www.ling.upenn.edu/ppche/ppche-release-2016/PPCEME-RELEASE-3. ↗
  16. Anthony Kroch, Ann Taylor, and Donald Ringe. 2000. The Middle English verb-second constraint: A case study in language contact and language change. In Susan Herring, Lene Schoessler, and Peter van Seth Kulick, Neville Ryant, and Beatrice Santorini. 2022b. Parsing Early Modern English for linguistic search. In Proceedings of the Society for Computation in Linguistics 2022, pages 143–157, online. Association for Computational Linguistics. ↗
  17. Vladimir I Levenshtein et al. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707– 710. find
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  19. France Martineau, Paul Hirschbühler, Anthony Kroch, and Yves Charles Morin. 2021. MCVF Corpus, parsed, version 2.0. https://github.com/ beatrice57/mcvf-plus-ppchf. ↗
  20. Beth Randall. 2010. CorpusSearch 2: a tool for linguistic research. Download site: http://corpussearch.sourceforge.net/CS.html. User guide: https://www. ling.upenn.edu/~beatrice/corpus-ling/CS-users-guide/index.html. ↗
  21. Djamé Seddah, Sandra Kübler, and Reut Tsarfaty. 2014. Introducing the SPMRL 2014 shared task on parsing morphologically-rich languages. In Proceedings of the First Joint Workshop on Statistical Parsing of Morphologically Rich Languages and Syntactic Analysis of Non-Canonical Languages, pages 103–109, Dublin, Ireland. Dublin City University. find
  22. Satoshi Sekine and Michael Collins. 2008. http://nlp.cs.nyu.edu/evalb/. evalb. ↗
  23. Temple F Smith, Michael S Waterman, et al. 1981. Identification of common molecular subsequences. Journal of molecular biology, 147(1):195–197. ↗
  24. Mitchell Stern, Jacob Andreas, and Dan Klein. 2017. A minimal span-based neural constituency parser. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 818–827, Vancouver, Canada. Association for Computational Linguistics.
  25. Ann Taylor, Arja Nurmi, Anthony Warner, Susan Pintzuk, and Terttu Nevalainen. 2006. Parsed Corpus of Early English Correspondence. Distributed by the Oxford Text Archive. Revised corrected version at https://github.com/beatrice57/pceec2. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kulick-2023-parsing-early-english-books-online/