Elamur Research⌕
☰
home / papers / 2026 / Validator-Guided Hard Negative Mining for Masked Language Modeling in Low-Resource Ancient Languages

Validator-Guided Hard Negative Mining for Masked Language Modeling in Low-Resource Ancient Languages

Paper · 2026 · Andrei Voinea

Summary

Pre-trained multilingual encoders perform poorly on Sumerian. It is a language isolate absent from mBERT's pre-training, so cross-lingual transfer cannot help restore damaged cuneiform text. The author tests whether explicit linguistic knowledge drawn from a small annotated sample can make up for this. The data are the 370 POS-annotated Sumerian tablets in the Cuneiform Digital Library Initiative (CDLI), split by tablet into 4,545 training, 568 validation and 253 test lines. From these, a three-tier validator is built that scores candidate tokens on subword, word and POS patterns. It serves two purposes:

  • It prunes the prediction space at inference.
  • It selects ambiguous positions, where many grammatically acceptable tokens fall outside mBERT's top-5, for targeted MLM fine-tuning of bert-base-multilingual-cased.

Each component helps substantially on its own, and together they raise hit@10 from 18.0% to 86.7%. On tablets roughly 600 years older, however, the validator's rules hurt the fine-tuned model, while the fine-tuning gains persist. The study shows that inexpensive rule-based constraints can strongly boost neural models for ancient languages within one period, but they must be rebuilt for each era.

Key points

  • Data: Only 370 of 143,078 CDLI Sumerian tablets (~0.3%) carry POS tags. They yield 5,366 lines after mBERT WordPiece tokenization, split 85-10-5 at the tablet level to avoid leakage.
  • Validator: A cascade of subword, word and POS filters with equal weights and a 0.01 confidence threshold. It shrinks candidates from ~30,000 to fewer than 20 per position while retaining 94.2% recall on validation data.
  • Hit@10 results:
    • Vanilla mBERT: 18.0%
    • Validator only: 72.8%
    • Hard-negative fine-tuning: 78.3%
    • Combined: 86.7% (+68.7 pp)
    • Validator-guided position selection beats uniform masking by 4.4 pp (86.7% vs 82.3%).
  • Temporal shift (~600 years earlier): The fine-tuned model loses 18.5% relative accuracy. The validator's contribution flips from +8.4 pp to −4.5 pp.
  • Limitations: The study tests only mBERT (no XLM-R) and uses a small test set (253 lines, 472 predictions per seed, 5 seeds, std ≤ 2.0%). Evaluation is intrinsic only, with no end-to-end tablet restoration.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Babeș-Bolyai University.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksMultilingual & cross-lingual NLP · 828 worksLow-resource languages · 267 works

Cite

Cite as Andrei Voinea. 2026. Validator-Guided Hard Negative Mining for Masked Language Modeling in Low-Resource Ancient Languages. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-srw.69

@inproceedings{voinea2026validator,
  title = {{Validator-Guided Hard Negative Mining for Masked Language Modeling in Low-Resource Ancient Languages}},
  author = {Andrei Voinea},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-srw.69},
  url = {https://doi.org/10.18653/v1/2026.acl-srw.69}
}

References

20 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yannis Assael, Thea Sommerschield, Brendan Shillingford, Mahyar Bordbar, John Pavlopoulos, Marita Chatzipanagiotou, Ion Androutsopoulos, Jonathan Prag, and Nando de Freitas. 2022. Restoring and attributing ancient texts using deep neural networks. Nature, 603(7900):280–283. find
  2. CDLI Contributors. 2026. Cuneiform digital library initiative. Accessed: 2026-02-19. find
  3. Christian Chiarcos, Émilie Pagé-Perron, Ilya Khait, Niko Schenk, and Lucas Reckling. 2018. Annotating a low-resource language with LLOD technology: Sumerian morphology and syntax. Information, 9(11). find
  4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  6. Ethan Fetaya, Yonatan Lifshitz, Elad Aaron, and Shai Gordin. 2020. Restoration of fragmentary babylonian texts using recurrent neural networks. Proceedings of the National Academy of Sciences, 117(37):22743–22751.
  7. Shai Gordin, Aleksi Sahala, Shahar Spencer, and Stav Klein. 2025. find
  8. EvaCun 2025 shared task: Lemmatization and token prediction in Akkadian and Sumerian using LLMs. In Proceedings of the Second Workshop on Ancient Language Processing, pages 242– 250, The Albuquerque Convention Center, Laguna. Association for Computational Linguistics.
  9. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Onli…
  10. Koren Lazar, Benny Saret, Asaf Yehudai, Wayne Horowitz, Nathan Wasserman, and Gabriel Stanovsky. 2021. Filling the gaps in Ancient Akkadian texts: A masked language modelling approach. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4682–4691, Online…
  11. Gabriel de Souza P Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. 2024. Nv-retriever: Improving text embedding models with effective hard-negative mining. arXiv preprint arXiv:2407.15831. — [arXiv:2407.15831](https://arxiv.org/abs/2407.15831) find
  12. Émilie Pagé-Perron, Maria Sukhareva, Ilya Khait, and Christian Chiarcos. 2017. Machine translation and automated analysis of the Sumerian language. In Proceedings of the Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 10–16,… find
  13. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, and 2 others. 20… — PyTorch: An Imperative Style, High-Performance Deep Learning Library
  14. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An adapter-based framework for multi-task cross-lingual transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673, Online. Association for Computational L…
  15. Ravneet Punia, Niko Schenk, Christian Chiarcos, and Émilie Pagé-Perron. 2020. Towards the first machine translation system for Sumerian transliterations. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3454–3460, Barcelona, Spain (Online). International Commit… find
  16. Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Ass…
  17. Marie-Louise Thomsen. 1984. The Sumerian Language: An Introduction to its History and Grammatical Structure. Akademisk Forlag. find
  18. Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. 2020. Extending multilingual BERT to low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2649–2656, Online. Association for Computational Linguistics. find
  19. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transf…
  20. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120–130, Online. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/voinea-2026-validator-guided-hard-negative-mining/