Elamur Research⌕
☰
home / papers / 2026 / Detecting Latin in Historical Books with Large Language Models: A Multimodal Benchmark

Detecting Latin in Historical Books with Large Language Models: A Multimodal Benchmark

Summary

The paper addresses a gap in historical NLP: finding Latin passages embedded in mostly English 18th-century books. These books often have noisy scans, imperfect OCR and complex layouts. The authors frame this as a two-step task. First, a system decides whether a page contains any Latin. Second, it extracts the Latin spans as text. Systems can use the page image, the OCR text, or both.

To support the task, they built a multimodal benchmark of 724 pages drawn from ECCO. Specialists annotated it, and each Latin segment is labelled with one or more of 12 usage categories, such as quotations, footnotes and code-switching. They then evaluated contemporary LLMs and multimodal LLMs in a zero-shot setting. For fair cross-modal comparison, they added an OCR post-correction and normalization step.

The main finding is that detection is highly reliable, but the models seem to depend on surface cues like vocabulary rather than real understanding of Latin. This matters for tracking the decline of Latin at scale, and image-only models could make un-OCRed archives searchable.

Key points

  • Dataset construction: The benchmark contains 724 annotated pages, 594 of which contain Latin. The authors sampled 800 cross-language text-reuse cases from the Reception Reader database over ECCO (about 200,000 books), using one page per book.
  • Annotation process: Three Latin-literate annotators drew bounding boxes in Label Studio, and a historical-text expert then reviewed all annotations. The boxes were mapped to text offsets through ECCO's OCR coordinates.
  • Usage categories: The typology has 12 categories. The most frequent are Direct Quote (258 pages), Independent (196) and Footnote (191). The pages also contain French, German and Greek, which adds difficulty.
  • Results: Image-only models reached roughly 99% page-level F1. Using raw ECCO OCR instead of corrected text cut text-only model scores by more than 14 points.
  • Error analysis and resources: Most errors come from labelling Roman names, anglicized Latin phrases and Latin-derived jargon as Latin. The dataset and code are released at github.com/COMHIS/EACL26-detect-latin, and the authors plan to extend the work to all of ECCO and to BnF collections.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Natural language processing

Topics: Evaluation & benchmarks · Latin · Vision & language

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.245

@inproceedings{wu2026detecting,
  title = {{Detecting Latin in Historical Books with Large Language Models: A Multimodal Benchmark}},
  author = {Wu, Yu and Shu, Ke and Fischer, Jonas and Pivovarova, Lidia and Rosson, David and Mäkelä, Eetu and Tolonen, Mikko},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.245},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.245}
}

References

41 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774. ↗ our card
  2. Gustavo Aguilar, Sudipta Kar, and Thamar Solorio. 2020. LinCE: A centralized benchmark for linguistic code-switching evaluation. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1803–1813, Marseille, France. European Language Resources Association. find our card
  3. Samuel Backer and Louis Hyman. 2025. Bootstrapping AI: Interdisciplinary approaches to assessing OCR quality in english-language historical documents. In Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities, pages 251–256. find
  4. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923. ↗ our card
  5. Utsab Barman, Amitava Das, Joachim Wagner, and Jennifer Foster. 2014. Code mixing: A challenge for language identification in the language of social media. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 13–23, Doha, Qatar. Association for Computational Linguistics. ↗
  6. Emanuela Boros, Maud Ehrmann, Matteo Romanello, Sven Najem-Meyer, and Frédéric Kaplan. 2024. Post-correction of historical text transcripts with large language models: An exploratory study. In Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2024), pages 133–159, St. Julians, Malta. Association for Computational Linguistics. find
  7. Patrick J Burns. 2023. Latincy: Synthetic trained pipelines for Latin NLP. arXiv preprint arXiv:2305.04365. ↗
  8. Patrick J Burns, James A Brofos, Kyle Li, Pramit Chaudhuri, and Joseph P Dexter. 2021. Profiling of intertextuality in Latin literature using word embeddings. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4900–4907. find
  9. OpenCompass Contributors. 2023. OpenCompass: A universal evaluation platform for foundation models. https://github.com/open-compass/ opencompass. ↗
  10. Mitchell A Gordon, Kevin Duh, and Jared Kaplan. 2021. Data and parameter scaling laws for neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5915–5922. find
  11. Svetlana Gorovaia, Gleb Schmidt, and Ivan P. Yamshchikov. 2024. Sui generis: Large language models for authorship attribution and verification in Latin. In Proceedings of the 4th International Conference on Natural Language Processing for Digital Humanities, pages 398–412, Miami, USA. Association for Computational Linguistics. ↗
  12. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081):633–638. find
  13. Gualberto A Guzmán, Joseph Ricard, Jacqueline Serigos, Barbara E Bullock, and Almeida Jacqueline Toribio. 2017. Metrics for modeling code-switching across corpora. In Interspeech, pages 67–71. find
  14. Jakob Hauser, Daniel Kondor, Jenny Reddish, Majid Benam, Enrico Cioni, Federica Villa, James Bennett, Daniel Hoyer, Pieter Francois, Peter Turchin, et al. 2024. Large language models’ expert-level global history knowledge benchmark (HiST-LLM). Advances in Neural Information Processing Systems, 37:32336–32369. find
  15. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. OpenAI o1 system card. arXiv preprint arXiv:2412.16720. ↗ our card
  16. Kyle P. Johnson, Patrick J. Burns, John Stewart, Todd Cook, Clément Besnier, and William J. B. Mattingly. 2021. The Classical Language Toolkit: An NLP framework for pre-modern languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 20–29, Online. Association for Computational Linguistics. ↗
  17. Jenna Kanerva, Cassandra Ledins, Siiri Käpyaho, and Filip Ginter. 2025. OCR error post-correction with LLMs in historical documents: No free lunches. arXiv preprint arXiv:2502.01205. ↗
  18. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. ↗
  19. Hanna-Mari Kristiina Kupari, Erik Henriksson, Veronika Laippala, and Jenna Kanerva. 2024. Improving Latin dependency parsing by combining treebanks and predictions. In Proceedings of the 4th International Conference on Natural Language Processing for Digital Humanities, pages 216–228, Miami, USA. Association for Computational Linguistics. find
  20. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles. find
  21. Chuwei Luo, Yufan Shen, Zhaoqing Zhu, Qi Zheng, Zhi Yu, and Cong Yao. 2024. LayoutLLM: Layout instruction tuning with large language models for document understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15630–15640. find
  22. Jani Marjanen, Tuuli Tahko, Leo Lahti, and Mikko Tolonen. 2025. Book printing in Latin and vernacular languages in northern Europe, 1500–1800. In The Hermeneutics of Bibliographic Data and Cultural Metadata, pages 27–66. National Library of Norway. find
  23. Valerio Perrone, Simon Hengchen, Marco Palma, Alessandro Vatri, Jim Q Smith, and Barbara McGillivray. 2021. Lexical semantic change for Ancient Greek and Latin. Computational approaches to semantic change, 6. ↗
  24. David Rosson, Eetu Mäkelä, Ville Vaara, Ananth Mahadevan, Yann Ryan, and Mikko Tolonen. 2023. Reception reader: Exploring text reuse in early modern British publications. Journal of Open Humanities Data, 9(1). ↗
  25. Sarah Schulz and Mareike Keller. 2016. Code-switching ubique est - language identification and part-of-speech tagging for historical mixed text. In Proceedings of the 10th SIGHUM Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 43–51, Berlin, Germany. Association for Computational Linguistics. find
  26. Rachele Sprugnoli, Federica Iurescia, and Marco Passarotti. 2024. Overview of the EvaLatin 2024 evaluation campaign. In Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA) @ LREC-COLING-2024, pages 190–197, Torino, Italia. ELRA and ICCL. find
  27. Peter M. Stahl. 2021. Lingua: The most accurate natural language detection library for Python. https://github.com/pemistahl/lingua-py. Python bindings for the Lingua language detection library. ↗
  28. Igor Sterner and Simone Teufel. 2023. TongueSwitcher: Fine-grained identification of German-English code-switching. In Proceedings of the 6th Workshop on Computational Approaches to Linguistic Code-Switching, pages 1–13, Singapore. Association for Computational Linguistics. find
  29. Milan Straka and Jana Straková. 2020. UDPipe at EvaLatin 2020: Contextualized embeddings and treebank embeddings. In Proceedings of LT4HALA 2020 - 1st Workshop on Language Technologies for Historical and Ancient Languages, pages 124–129, Marseille, France. European Language Resources Association (ELRA). ↗
  30. Elina Stüssi and Phillip Ströbel. 2024. Part-of-speech tagging of 16th-century Latin with GPT. In Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2024), pages 196–206, St. Julians, Malta. Association for Computational Linguistics. find
  31. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. ↗
  32. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. ↗ our card
  33. Yunting Xie, Matti La Mela, and Fredrik Tell. 2025. Multimodal LLM-assisted information extraction from historical documents: The case of Swedish patent cards (1945-1975) and ChatGPT. In The 9th Digital Humanities in the Nordic and Baltic Countries Conference (DHNB 2025), March 5–7, 2025, Tartu, Estonia, pages 1–15. University of Oslo Library. ↗
  34. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567. find our card
  35. Yuan Zhang, Jason Riesa, Daniel Gillick, Anton Bakalov, Jason Baldridge, and David Weiss. 2018. A fast, compact, accurate model for language identification of codemixed text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 328–337, Brussels, Belgium. Association for Computational Linguistics. find
  36. Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. 2025. InternVL3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479. ↗ our card
  37. Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. 2021. Document collection visual question answering. In International Conference on Document Analysis and Recognition, pages 778–792. Springer. find
  38. Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. 2020-2025. Label Studio: Data labeling software. Open source software available from https://github.com/HumanSignal/label-studio. ↗
  39. Mikko Tolonen, Eetu Mäkelä, and Leo Lahti. 2022. The anatomy of Eighteenth Century Collections Online (ECCO). Eighteenth-century studies, 56(1):95–123. ↗
  40. Martin Volk, Dominic Philipp Fischer, Lukas Fischer, Patricia Scheurer, and Phillip Benjamin Ströbel. 2024. LLM-based machine translation and summarization for Latin. In Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA) @ LREC-COLING-2024, pages 122–128, Torino, Italia. ELRA and ICCL. find
  41. Martin Volk, Lukas Fischer, Patricia Scheurer, Bernard Silvan Schroffenegger, Raphael Schwitter, Phillip Ströbel, and Benjamin Suter. 2022. Nunc pro-fana tractemus. detecting code-switching in a large corpus of 16th century letters. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2901–2908, Marseille, France. European Language Resources Association. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2026-detecting-latin-historical-books-large/