Elamur Research⌕
☰
home / papers / 2026 / A Recipe for Adapting Multilingual Embedders to OCR-Error Robustness and Historical Texts

A Recipe for Adapting Multilingual Embedders to OCR-Error Robustness and Historical Texts

Summary

Off-the-shelf multilingual embedding models work well on modern text but lose accuracy on digitized historical documents, where OCR mistakes and older spellings disrupt tokenization. The problem is worst for low-resource languages such as Luxembourgish, whose archives span about 150 years of shifting orthography.

The authors merge two earlier adaptation ideas into a sequential recipe for M-GTE:

  • Step 1: The model is trained on cross-lingual pairs linking historical Luxembourgish with machine-translated German, French and English, plus modern Luxembourgish pairs.
  • Step 2: The model undergoes monolingual denoising. Roughly 5% of characters are randomly edited, and the model learns to match noisy and clean versions of sentences, summaries and full articles in six languages.

Training both objectives at once proved unstable, which motivated the two-stage design. Evaluation covers three settings: clean cross-lingual ranking, artificially OCR-degraded copies of those sets, and historical Luxembourgish bitext mining with genuine OCR errors. The adapted models resist noise better and perform better on real historical data, while clean-text quality stays roughly unchanged. Models, code and data are openly released, enabling cross-lingual search over European heritage collections.

Key points

  • Released models (AGPL-3.0, github.com/impresso/ocr-robust-multilingual-embeddings):
    • gte-multilingual-base-ocr-noise-robust, a generalist model.
    • gte-multilingual-base-histlux-ocr-noise-robust, a historical Luxembourgish specialist that accepts a slight drop on modern text.
  • Task A ("Mix-it-all" setup from Michail et al., 2025c), 120k pairs in total:
    • 20k historical Luxembourgish sentences, each paired with DE/FR/EN translations, giving 60k pairs.
    • 40k modern LB–FR and 20k modern LB–EN pairs taken from the LuxEmbedder training data.
  • Task B data:
    • TED: 10k DE and 10k FR pairs.
    • Impresso historical newspaper articles (DE/FR), filtered to at least 98% in-vocabulary tokens.
    • MLSum articles (about 550 tokens) and summaries (about 20 tokens) in DE, ES, FR, RU and TR, with 2,000 records per language.
  • Training setup: MultipleNegativesRankingLoss in SentenceTransformers with batch size 8. Each batch contains a single language and source corpus. Both tasks together use 180k training pairs.
  • Evaluation conditions:
    • Control: clean tasks, e.g., CLSD German–French.
    • Realistic: OCR-degraded versions of the control sets.
    • Organic: HistLUX historical Luxembourgish bitext mining with real OCR errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/29rfx5wcyz3z

@inproceedings{michail2026recipe,
  title = {{A Recipe for Adapting Multilingual Embedders to OCR-Error Robustness and Historical Texts}},
  author = {Michail, Andrianos and Psychias, Stylianos and Opitz, Juri and Clematide, Simon},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/29rfx5wcyz3z},
  url = {https://doi.org/10.63317/29rfx5wcyz3z}
}

References

25 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondřej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina España-Bonet, Angela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christophe… find our card
  2. Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (W… find our card
  3. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity — multilingual and cross-lingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canad… find
  4. Andreas Chari, Sean MacAvaney, and Iadh Ounis. 2025. Improving low-resource retrieval effectiveness using zero-shot linguistic similarity transfer. In Advances in Information Retrieval, pages 290– 306, Cham. Springer Nature Switzerland. find
  5. Maud Ehrmann, Matteo Romanello, Simon Clematide, Phillip Benjamin Ströbel, and Raphaël Barman. 2020. Language resources for historical newspapers: the impresso collection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 958–968, Marseille, France. European Language… find
  6. Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, et al. 2025. MMTEB: Massive multilingual text embedding benchmark. In The Thirteenth International Conference on Learning Representations. find
  7. Özay Ezerceli, Gizem Gümüşçekiçci, Tuğba Erkoç, and Berke Özenç. 2025. Turkembed4retrieval: Turkish embedding model for retrieval task. In 2025 33rd Signal Processing and Communications Applications Conference (SIU), pages 1–4. find
  8. Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun hsuan Sung, Laszlo Lukacs, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. find
  9. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781. find our card
  10. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Proc… find
  11. Cedric Lothritz, Bertrand Lebichot, Kevin Allix, Lisa Veiber, Tegawende Bissyande, Jacques Klein, Andrey Boytsov, Clément Lefebvre, and Anne Goujon. 2022. LuxemBERT: Simple and practical data augmentation in language model pre-training for Luxembourgish. In Proceedings of the Thirteenth Language Res… find
  12. Andrianos Michail, Simon Clematide, and Rico Sennrich. 2025a. Examining multilingual embedding models cross-lingually through LLM-generated adversarial examples. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 2161–2170, Suzhou, China. Association for Computational Li… find
  13. Andrianos Michail, Juri Opitz, Yining Wang, Robin Meister, Rico Sennrich, and Simon Clematide. 2025b. Cheap character noise for OCR-robust multilingual embeddings. In Findings of the Association for Computational Linguistics: ACL 2025, pages 11705–11716, Vienna, Austria. Association for Computationa… find
  14. Andrianos Michail, Corina Raclé, Juri Opitz, and Simon Clematide. 2025c. Adapting multilingual embedding models to historical Luxembourgish. In Proceedings of the 9th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 202… find
  15. Stephen Mutuvi, Antoine Doucet, Moses Odeo, and Adam Jatowt. 2018. Evaluating the impact of ocr errors on topic modeling. In Maturity and Innovation in Digital Libraries, pages 3–14, Cham. Springer International Publishing. find
  16. Fred Philippy, Siwen Guo, Jacques Klein, and Tegawende Bissyande. 2025. LuxEmbedder: A cross-lingual approach to enhanced Luxembourgish sentence embeddings. In Proceedings of the 31st International Conference on Computational Linguistics, pages 11369–11379, Abu Dhabi, UAE. Association for Computatio… find
  17. Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua… find our card
  18. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3982–3992. Association for Computational Linguistics. find
  19. Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2020. MLSUM: The multilingual summarization corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8051–8067, Online. Association for Computational L… find
  20. Georgios Sidiropoulos and Evangelos Kanoulas. 2022. Analysing the robustness of dual encoders for dense retrieval against misspellings. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22, page 2132–2136, New York, NY, USA. As… find
  21. Kazem Taghva, Julie Borsack, and Allen Condit. 1996. Evaluation of model-based retrieval effectiveness with ocr text. ACM Trans. Inf. Syst., 14(1):64–93. find
  22. Panuthep Tasawong, Wuttikorn Ponwitayarat, Peerat Limkonchotiwat, Can Udomcharoenchaikit, Ekapol Chuangsuwanich, and Sarana Nutanong. 2023. Typo-robust representation learning for dense retrieval. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: S… find
  23. Shaoxiong Zhan, Hai Lin, Hongming Tan, Xiaodong Cai, Hai-Tao Zheng, Xin Su, Zifei Shan, Ruitong Liu, and Hong-Gee Kim. 2025. Lexsembridge: Fine-grained dense representation enhancement through token-aware embedding augmentation. find
  24. Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. 2024. mGTE: Generalized long-context text representation and reranking models for multilingual text retrieval. In Proceedings of the 2024… find our card
  25. Elaine Zosa, Stephen Mutuvi, Mark Granroth-Wilding, and Antoine Doucet. 2021. Evaluating the robustness of embedding-based topic models to ocr noise. In Towards Open and Trustworthy Digital Societies: 23rd International Conference on Asia-Pacific Digital Libraries, ICADL 2021, Virtual Event, Decembe… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/michail-2026-recipe-adapting-multilingual-embedders-ocr/