Elamur Research⌕
☰
home / papers / 2026 / LëtzCross: A Cross-Lingual Page-Level Benchmark for Multimodal Retrieval over Luxembourgish Documents

LëtzCross: A Cross-Lingual Page-Level Benchmark for Multimodal Retrieval over Luxembourgish Documents

Paper · 2026 · Omar El Bachyr, Fred Philippy, Laura Maria Bernardy et al.

Summary

Page-image retrievers in the ColPali family handle visually rich PDFs well, but it was unclear how they behave when queries and documents are in different languages and one of those languages is low-resource. The authors built LëtzCross to test this. It indexes 908 Luxembourgish PDF pages as images and pairs them with 579 questions, each available in English, French, German and Luxembourgish. The source PDFs come from the Luxembourgish split of FinePDFs. Most questions were generated by an LLM from text-heavy pages, checked automatically and then reviewed by people. A smaller set was written by hand to target charts, tables and diagrams. English questions were machine-translated into the other three languages.

In this comparison, late-interaction page-image retrievers beat OCR-based text-only pipelines for every query language. Fine-tuning on one query language helped the others too. A multilingual fine-tuning mix that includes Luxembourgish gave the best Luxembourgish-query results. The work matters because it supplies a first benchmark for multimodal RAG over a language with few NLP resources.

Key points

  • Benchmark size: 579 QA pairs over 908 pages, split into 530 auto-generated text-focused pairs and 49 manually written visually grounded pairs.
    • Average query length is 19.5 tokens.
    • Average page length is 722.5 tokens.
  • Generation and filtering:
    • GPT-5-mini wrote QA pairs from page images.
    • Gemini-2.5-Flash then answered each question independently.
    • 90.28% of pairs were judged semantically equivalent and 71.50% matched the gold answer exactly.
    • Inconsistent pairs were dropped.
  • Translation: Gemini-3-Flash produced the German and French queries. Gemini-3-Pro produced Luxembourgish, chosen after the authors compared trial translations by hand.
  • Fine-tuning backbone: colnomic-embed-multimodal-3b, scored by nDCG@10 averaged over four query languages.
    • Baseline: 78.66
    • FT (FR): 80.96
    • EN+FR+DE: 79.93
    • EN+FR+DE+LB: 80.94
  • Effect of adding Luxembourgish data: Luxembourgish-query retrieval rose from 77.48 to 80.02, which is +0.57 over FT (FR).

Interesting ideas

  • I1. Generated QA pairs are kept only when a second, independent model reads the source and reaches the same answer, judged by both exact and meaning-level matching; this is a cheap way to clean synthetic evaluation data.
  • I2. Easy, plentiful items are produced automatically, while a small set of hard items is written by a human to probe a different capability, and that subset is reported separately.
  • I3. Reviewers see only the question, never the answer, and rewrite it to sound like a natural information need; this separates fluency fixes from answer checking.
  • I4. Results are given as mean ± std over three runs, and gaps smaller than the spread are explicitly called ties rather than wins.
  • I5. All query languages are aligned translations of one source query, so differences in scores reflect the query language rather than differences in content; the authors also note the translation artifacts this can introduce.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver QA items for the readers could be kept only when two different LLM teachers agree on the answer, with the exact-vs-semantic agreement rates logged as a quality signal.
  • I2 (answer): School-test and fable-QA suites could pair a large auto-generated pool with a small handcrafted slice targeting tricky phenomena (negation, event order), scored on its own.
  • I3 (answer): When building reader tests, questions could be polished blind to their answers so the wording does not leak the answer or reward surface matching.
  • I4 (learn): Parser and reader comparisons, including negative controls, should report run-to-run variance and treat sub-noise differences as equal before accepting a new repair rule or model.
  • I5 (back): Meta-world translation could be tested on aligned multi-language versions of the same question, checking whether each one maps to an identical meaning graph.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓King Fahd University of Petroleum and MineralsDhahran · Saudi Arabia54 works · 4,666 citations🎓University of LuxembourgLuxembourg · Luxembourg28 works · 337 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksInformation Systems · 1,730 works

🧭 Topics 4Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 worksMultilingual & cross-lingual NLP · 882 worksVision & language · 811 worksInformation retrieval & search · 552 works

Cite

Cite as Omar El Bachyr, Fred Philippy, Laura Maria Bernardy, Saad Ezzini, Jacques Klein and Tegawende Bissyande. 2026. LëtzCross: A Cross-Lingual Page-Level Benchmark for Multimodal Retrieval over Luxembourgish Documents. arXiv:2608.21714.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2608.21714 · arXiv 2608.21714

@misc{bachyr2026letzcross,
  title = {{LëtzCross: A Cross-Lingual Page-Level Benchmark for Multimodal Retrieval over Luxembourgish Documents}},
  author = {Omar {El Bachyr}, Fred Philippy, Laura Maria Bernardy, Saad Ezzini, Jacques Klein and Tegawende Bissyande},
  year = {2026},
  doi = {10.48550/arxiv.2608.21714},
  eprint = {2608.21714},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2608.21714}
}

References

26 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Voyage AI. 2026. find
  2. Laura Bernardy. 2022. A Luxembourgish GPT-2 Approach Based on Transfer Learning. Master’s thesis, University of Trier. find
  3. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. find
  4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 8440–8451.
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186. ↗
  6. Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. 2025. Colpali: Efficient document retrieval with vision language models. In International Conference on Learning Representations, volume 2025, pages 61424–61449. ↗
  7. Daniela Gierschek. 2022. Detection of Sentiment in Luxembourgish User Comments. Ph.D. thesis, University of Luxembourg. find
  8. Peter Gilles, Nina Hosseini Kivanani, and Léopold Edem Ayité Hillah. 2023. Lux-asr: Building an asr system for the luxembourgish language. In Proceedings of the 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE. find
  9. Ping Guo, Yubing Ren, Yue Hu, Yanan Cao, Yunpeng Li, and Heyan Huang. 2024. Steering large language models for cross-lingual information retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 585–596. find
  10. Michael Günther, Saba Sturua, Mohammad Kalim Akram, Isabelle Mohr, Andrei Ungureanu, Sedigheh Eslami, Scott Martens, Bo Wang, Nan Wang, and Han Xiao. 2025. jina-embeddings-v4: Universal embeddings for multimodal multilingual retrieval. ↗
  11. Chao-Wei Huang, Chen-An Li, Tsu-Yuan Hsu, Chen-Yu Hsu, and Yun-Nung Chen. 2024. Unsupervised multilingual dense retrieval via generative pseudo labeling. In Findings of the Association for Computational Linguistics: EACL 2024, pages 736–746. find
  12. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 6769–6781. ↗
  13. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 39– 48. ↗
  14. Hynek Kydlı́ček, Guilherme Penedo, and Leandro von Werra. 2025. Finepdfs. https://huggingface.co/datasets/HuggingFaceFW/finepdfs . ↗
  15. António Loison, Quentin Macé, Antoine Edy, Victor Xing, Tom Balough, Gabriel Moreira, Bo Liu, Manuel Faysse, Céline Hudelot, and Gautier Viaud. 2026. Vidore v3: A comprehensive evaluation of retrieval augmented generation in complex real-world scenarios. ↗
  16. Cedric Lothritz, Bertrand Lebichot, Kevin Allix, Lisa Veiber, Tegawende Bissyande, Jacques Klein, Andrey Boytsov, Clément Lefebvre, and Anne Goujon. 2022. LuxemBERT: Simple and Practical Data Augmentation in Language Model Pre-Training for Luxembourgish. In Proceedings of LREC. find
  17. Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. 2024. Unifying multimodal retrieval via document screenshot embedding. arXiv preprint arXiv:2406.11251. ↗
  18. Nomic AI. 2025. Colnomic embed multimodal 3b. https://huggingface.co/nomic-ai/ colnomic-embed-multimodal-3b . Hugging Face model card. ↗
  19. Fred Philippy, Shohreh Haddadan, and Siwen Guo. 2024. Forget NLI, Use a Dictionary: Zero-Shot Topic Classification for Low-Resource Languages with Application to Luxembourgish. In Proceedings of SIGUL (LREC-COLING). find
  20. Alistair Plum, Caroline Döhmer, Emilia Milano, Anne-Marie Lutgen, and Christoph Purschke. 2024. LuxBank: The First Universal Dependency Treebank for Luxembourgish. In Proceedings of TLT. ↗
  21. Alistair Plum, Tharindu Ranasinghe, and Christoph Purschke. 2025. Text Generation Models for Luxembourgish with Limited Data: A Balanced Multilingual Strategy. In Proceedings of VarDial (COLING). ↗
  22. Christoph Purschke. 2020. Attitudes toward multilingualism in luxembourg. a comparative analysis of online news comments and crowdsourced questionnaire data. Frontiers in Artificial Intelligence, 3. ↗
  23. Tharindu Ranasinghe, Alistair Plum, Christoph Purschke, and Marcos Zampieri. 2023. Publish or hold? automatic comment moderation in Luxembourgish news articles. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 968–978, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria. find
  24. Joshgun Sirajzade, Daniela Gierschek, and Christoph Schommer. 2020. An Annotation Framework for Luxembourgish Sentiment Analysis. In Proceedings of SLTU-CCURL (LREC). find
  25. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672. ↗
  26. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bachyr-2026-l-etzcross-cross-lingual-page/