Elamur Research⌕
☰
home / papers / 2026 / POS Tagging with Generative LLMs for Historical Germanic Low-Resource Languages: An Evaluation Against Fine-Tuned BERT

POS Tagging with Generative LLMs for Historical Germanic Low-Resource Languages: An Evaluation Against Fine-Tuned BERT

Paper · 2026 · Irene Miani, Gregory Darwin and Sara Stymne

Summary

The paper asks whether instruction-tuned generative LLMs can tag parts of speech in historical Germanic languages, which have few resources and often no dependable NLP tools. The authors test Qwen2.5-3B-Instruct and Apertus-8B-Instruct-2509 in zero- and few-shot prompting on Old English, Old High German, Old Saxon and Old Norse, covering both poetry and prose. They compare these models with fine-tuned mBERT taggers. Modern English and Modern German serve as a reference point. All corpora were mapped to the UD UPOS tag set.

Fine-tuned encoders beat the prompted LLMs for every language and both genres. The LLMs also scored much lower on historical languages than on modern ones, which suggests these older varieties are poorly represented in pretraining data. Error analysis found that the LLMs often returned structurally inconsistent output, so their predictions needed extra post-processing before scoring. The authors conclude that, for now, task-specific fine-tuning is a more reliable route than prompting for basic annotation of historical low-resource text.

Key points

  • Historical data sources:
    • Old English: YCOEP (poetry) and YCOE (prose, about 1.5M words).
    • Old High German: Deutsch Diachron Digital – Referenzkorpus Altdeutsch v1.2 (about 650,000 words).
    • Old Saxon: HeliPaD (5,968 lines of Heliand) plus Old Saxon texts from the Referenzkorpus.
    • Old Norse: Menotec on INESS, with Edda Regius (3,665 lines) for poetry and about 10,318 lines of prose.
  • Modern reference data:
    • English prose: UD_English-LinES literary sub-corpora.
    • German prose: UD German LIT, used because no closer modern prose treebank was available.
    • Poetry for both languages: the PoeTree dataset, mostly sampled from 1900 onward.
  • Tag set: all tags were converted to UPOS using the mapping scheme of Miani et al. (2026).
  • Main result: fine-tuned mBERT outperformed both generative models in every setting. Zero-shot per-tag F1 for the LLMs was frequently very low on historical languages.
  • Code: publicly available at github.com/irenemiani/historical-germanic.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Uppsala UniversityUppsala · Sweden64 works · 13,164 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksPart-of-speech tagging · 126 worksLow-resource languages · 267 works

Cite

Cite as Irene Miani, Gregory Darwin and Sara Stymne. 2026. POS Tagging with Generative LLMs for Historical Germanic Low-Resource Languages: An Evaluation Against Fine-Tuned BERT. Proceedings of the Fourth Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA 2026) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2r9n7btigbd6

@inproceedings{miani2026pos,
  title = {{POS Tagging with Generative LLMs for Historical Germanic Low-Resource Languages: An Evaluation Against Fine-Tuned BERT}},
  author = {Irene Miani, Gregory Darwin and Sara Stymne},
  booktitle = {Proceedings of the Fourth Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA 2026) @ LREC 2026},
  year = {2026},
  doi = {10.63317/2r9n7btigbd6},
  url = {https://doi.org/10.63317/2r9n7btigbd6}
}

References

27 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lars Ahrenberg. 2015. Converting an English-Swedish parallel treebank to Universal Dependencies. In Proceedings of the Third International Conference on Dependency Linguistics (DepLing 2015), pages 10–19, Uppsala, Sweden. Association for Computational Linguistics.
  2. Project Apertus, Alejandro Hernández-Cano, Alexander Hägele, Allen Hao Huang, Angelika Romanou, Antoni-Joan Solergibert, Barna Pasztor, Bettina Messmer, Dhia Garbaya, Eduard Frank Ďurech, Ido Hakimi, Juan García Giraldo, Mete Ismayilzada, Negar Foroutan, Skander Moalla, Tiancheng Chen, Vinko Sabolčec, Yixuan Xu, Michael Aerni, Badr AlKhamissi, Inés Altemir Mariñas, Mohammad Hossein Amani, Matin Ansaripour, Ilia Badanin, Harold Benoit, Emanuela Boros, Nicholas Browning, Fabian Bösch, Maximilian Böther, Niklas Canova, Camille Challier, Clement Charmillot, Jonathan Coles, Jan Deriu, Arnout Devos, Lukas Drescher, Daniil Dzenhaliou, Maud Ehrmann, Dongyang Fan, Simin Fan, Silin Gao, Miguel Gila, María Grandury, Diba Hashemi, Alexander Hoyle, Jiaming Jiang, Mark Klein, Andrei Kucharavy, Anastasiia Kucherenko, Frederike Lübeck, Roman Machacek, Theofilos Manitaras, Andreas Marfurt, Kyle Matoba, Simon Matrenok, Henrique Mendonça, Fawzi Roberto Mohamed, Syrielle Montariol, Luca Mouchel, Sven Najem-Meyer, Jingwei Ni, Gennaro Oliva, Matteo Pagliardini, Elia Palme, Andrei Panferov, Léo Paoletti, Marco Passerini, Ivan Pavlov, Auguste Poiroux, Kaustubh Ponkshe, Nathan Ranchin, Javi Rando, Mathieu Sauser, Jakhongir Saydaliev, Muhammad Ali Sayfiddinov, Marian Schneider, Stefano Schuppli, Marco Scialanga, Andrei Semenov, Kumar Shridhar, Raghav Singhal, Anna Sotnikova, Alexander Sternfeld, Ayush Kumar Tarun, Paul Teiletche, Jannis Vamvas, Xiaozhe Yao, Hao Zhao, Alexander Ilic, Ana Klimovic, Andreas Krause, Caglar Gulcehre, David Rosenthal, Elliott Ash, Florian Tramèr, Joost Vande-Vondele, Livio Veraldi, Martin Rajman, Thomas Schulthess, Torsten Hoefler, Antoine Bosselut, Martin Jaggi, and Imanol Schlag. 2025. Apertus: Democratizing open and compliant llms for global language environments.
  3. Kristin Bech and Kristine Eide. 2014. The ISWOC corpus. Department of Literature, Area Studies and European Languages, University of Oslo. find
  4. Werner Besch and Norbert Richard Wolf. 2009. Geschichte der deutschen Sprache. Erich Schmidt, Berlin. find
  5. Terra Blevins, Hila Gonen, and Luke Zettlemoyer. 2023. Prompting language models for linguistic structure. arXiv preprint arXiv:2211.07830. ↗
  6. Chao-Yang Chang, Yan-Ming Lin, Chih-Chung Kuo, Yen-Chun Lai, Chao-Shih Huang, Yuan-Fu Liao, and Tsun-guan Thiann. 2024. A preliminary study on Taiwanese POS taggers: Leveraging Chinese in the absence of Taiwanese POS annotation datasets. In 2024 27th Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardisation of Speech Databases and Assessment Techniques (O-COCOSDA), pages 1–6. find
  7. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2023. A survey on evaluation of large language models. arXiv preprint arXiv:2307.03109. ↗
  8. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308.
  9. Zhao Fang, Liang-Chun Wu, Xuening Kong, and Spencer Dean Stewart. 2025. A comparative analysis of word segmentation, part-of-speech tagging, and named entity recognition for historical Chinese sources, 1900–1950. arXiv preprint arXiv:2503.19844. ↗
  10. Brendan Ferreri Hanberry. 2015. Application of a POS tagger to a novel chronological division of Early Modern German text. Master’s thesis, University of North Carolina at Chapel Hill. find
  11. Odd Einar Haugen and Fartein Th. Øverland. 2014. Guidelines for morphological and syntactic annotation of Old Norwegian texts. Bergen Language and Linguistics Studies (BeLLS), 4(2). find
  12. Kyle P. Johnson, Patrick J. Burns, John Stewart, Todd Cook, Clément Besnier, and William J. B. Mattingly. 2021. The Classical Language Toolkit: An NLP framework for pre-modern languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 20–29, Online. Association for Computational Linguistics. ↗
  13. Mariya Koleva, Melissa Farasyn, Bart Desmet, Anne Breitbarth, and Veronique Hoste. 2017. An automatic part-of-speech tagger for Middle Low German. International Journal of Corpus Linguistics, 22(1):108–141. ↗
  14. Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning. arXiv preprint arXiv:2304.05613. ↗
  15. Mateus Machado and Evandro Ruiz. 2024. Evaluating large language models for the tasks of PoS tagging within the Universal Dependency framework. In Proceedings of the 16th International Conference on Computational Processing of Portuguese – Vol. 1, pages 454–460, Santiago de Compostela, Galicia/Spain. Association for Computational Linguistics.
  16. Irene Miani, Sara Stymne, and Gregory R. Darwin. 2025. Cross-genre learning for Old English poetry POS tagging. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 708–724, Vienna, Austria. Association for Computational Linguistics.
  17. Irene Miani, Sara Stymne, and Gregory R. Darwin. 2026. Cross-lingual and cross-domain transfer learning for POS tagging in historical Germanic low-resource languages. In Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026), pages 542–558, Rabat, Morocco. Association for Computational Linguistics.
  18. Pavel Plecháč, Andris Šel , a, Héctor Bermúdez Sabel, Katharina Bobenhausen, Soňa Cinková, I. L. Dale, Émilie Delente, Marco De Sisto, Thomas Haider, Benjamin Hammerich, Péter Horváth, Randi Kvinnsland, Niko Kočnik, Radek Kolár, Kirill Korchagin, Anna Martynenko, Alexander Mittmann, Bence Nagy, Begoña Navarro Colorado, and Diana Sitchinava. 2025. PoeTree: Poetry corpora in Czech, English, French, German, Hungarian, Italian, Norwegian, Portuguese, Russian, Slovenian, and Spanish (1.0.0) [data set]. find
  19. Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip S. Yu. 2025. Large language models meet NLP: A survey. arXiv preprint arXiv:2405.12819. ↗
  20. Qwen Team. 2024. Qwen2.5: A party of foundation models. Qwen Blog. find
  21. Victoria Rosén, Koenraad De Smedt, Paul Meurer, and Helge Dyvik. 2012. An open infrastructure for advanced treebanking. In META-RESEARCH Workshop on Advanced Treebanking at LREC 2012, pages 22–29, Istanbul, Turkey. find
  22. Silke Scheible, Richard J. Whitt, Martin Durrell, and Paul Bennett. 2011. Evaluating an ‘off-the-shelf’ POS-tagger on Early Modern German text. In Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 19–23, Portland, OR, USA. Association for Computational Linguistics. find
  23. Matthias Schöffel, Esteban Garcés Arias, Marinus Wiedner, Paula Ruppert, Meimingwei Li, Christian Heumann, and Matthias Aßenmacher. 2025a. Unveiling factors for enhanced POS tagging: A study of low-resource medieval romance languages. arXiv preprint arXiv:2506.17715. ↗
  24. Matthias Schöffel, Marinus Wiedner, Esteban Garces Arias, Paula Ruppert, Christian Heumann, and Matthias Aßenmacher. 2025b. Modern models, medieval texts: A POS tagging study of Old Occitan. In Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities, pages 334–349, Albuquerque, USA. Association for Computational Linguistics. ↗
  25. Elina Stüssi and Phillip Ströbel. 2024. Part-of-speech tagging of 16th-century Latin with GPT. In Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2024), pages 196–206, St. Julians, Malta. Association for Computational Linguistics. ↗
  26. Rob van der Goot, Ahmet Üstün, Alan Ramponi, Ibrahim Sharaf, and Barbara Plank. 2021. Massive choice, ample tasks (MaChAmp): A toolkit for multi-task learning in NLP. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 176–197, Online. Association for Computational Linguistics. ↗
  27. Klotz Martin Donhauser Karin Gippert Jost Lühr Rosemarie Zeige Lars Erik, Schnelle Gohar. 2025. Deutsch Diachron Digital – Referenzkorpus Altdeutsch (Version 1.2). Humboldt-Universität zu Berlin. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/miani-2026-pos-tagging-generative-llms-historical/