Elamur Research⌕
☰
home / papers / 2026 / The Multilingual Curse at the Retrieval Layer: Evidence from Amharic

The Multilingual Curse at the Retrieval Layer: Evidence from Amharic

Paper · 2026 · Yosef Worku Alemneh, Kidist Amde Mekonnen and Maarten de Rijke

Summary

The paper asks whether multilingual retrievers that score well zero-shot actually transfer to an underrepresented, morphologically rich language. Amharic serves as the test case. It is written in Ge'ez script and has root-and-pattern morphology. The authors extend an earlier Amharic benchmark into the Amharic Passage Retrieval Dataset V2, with 68K weakly aligned query–passage pairs drawn from four sources. Under one shared protocol they compare:

  • zero-shot multilingual models,
  • multilingual models fine-tuned on Amharic,
  • monolingual Amharic retrievers across four paradigms: dense bi-encoder, late interaction, learned sparse, and cross-encoder re-ranking.

BM25 serves as the lexical reference. The best zero-shot multilingual system trails the best monolingual first-stage retriever by 23% relative MRR@10. Amharic fine-tuning gives large gains, but the best fine-tuned multilingual model, though much larger, still falls short of the monolingual one. The authors conclude that aggregate multilingual benchmarks cannot certify coverage of a given language. Retrieval, as the evidence layer of RAG, must be measured and adapted in the target language itself. Data, code and models are released.

Key points

  • Dataset V2 combines AMNEWS, XL-SUM, Amharic Wikipedia and AmQA. It has 68,000 pairs after MD5 deduplication and a fixed 90/10 train–test split, with one source-aligned positive per query.
  • Monolingual retrievers come in Medium and Base sizes, built on roberta-medium-amharic and roberta-base-amharic. The paradigms are a dense bi-encoder, ColBERT-style late interaction, a SPLADE-style learned sparse model and a cross-encoder re-ranker.
  • Fine-tuning the multilingual models raises MRR@10 by 32–60% relative:
    • Gemma goes from 0.448 to 0.718.
    • Harrier goes from 0.576 (prompted zero-shot) to 0.760.
  • Harrier still stays below ColBERT-Base-Amharic despite having roughly 2.5× the parameters.
  • The top score is MRR@10 = 0.830, from cross-encoder re-ranking of the top-50 candidates of an Amharic dense bi-encoder. Metrics are Recall@5/10, MRR@10 and NDCG@10.

Interesting ideas

  • I1. After adaptation, the authors judge a model against the strongest in-language alternative at similar cost, not only against its own zero-shot score; improvement shows capacity, while parity needs this stronger comparison.
  • I2. They argue a pipeline should report the quality of its evidence-selecting layer separately, because later stages cannot recover what that layer never surfaced.
  • I3. Positives are taken from document structure (headline→body, question→answer paragraph), with an explicit caveat that judgments are incomplete; Recall@k is therefore read only as a hit measure and absolute scores as conservative.
  • I4. Hash-based deduplication runs before a fixed train/test split, keeping repeated items from leaking across the split.
  • I5. A simple lexical baseline stays in every table and proves competitive, revealing when a learned representation is poorly aligned to the language.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when gparse is fine-tuned on new silver data, compare it with the best specialised parser of comparable size, not just its pre-tuning version.
  • I2 (answer): fable-QA and word-problem evaluations could log whether the needed entity or event was present in the text world before scoring the answer, separating world-building failures from reasoning failures.
  • I3 (learn): structurally derived silver pairs, such as title→story or question→supporting sentence, could feed evaluation, with metrics labelled as lower bounds rather than exact accuracy.
  • I4 (learn): apply the same deduplication to silver treebanks and test sets before splitting, so parser scores are not inflated by repeated sentences.
  • I5 (learn): keep a trivial or rule-based baseline as a negative control in each evaluation, so a learned component that fails to beat it is flagged immediately.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands151 works · 174,506 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 3,066 worksMultilingual & cross-lingual NLP · 882 worksInformation retrieval & search · 552 works

Cite

Cite as Yosef Worku Alemneh, Kidist Amde Mekonnen and Maarten de Rijke. 2026. The Multilingual Curse at the Retrieval Layer: Evidence from Amharic. Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.mellm-1.19

@inproceedings{alemneh2026multilingual,
  title = {{The Multilingual Curse at the Retrieval Layer: Evidence from Amharic}},
  author = {Yosef Worku Alemneh, Kidist Amde Mekonnen and Maarten {de Rijke}},
  booktitle = {Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.mellm-1.19},
  url = {https://doi.org/10.18653/v1/2026.mellm-1.19}
}

References

30 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tilahun Abedissa, Ricardo Usbeck, and Yaregal Assabie. 2023. AmQA: Amharic question answering dataset. arXiv preprint arXiv:2303.03290. ↗
  2. Yosef Worku Alemneh. 2024. Amharic bert and roberta models. Hugging Face model collection. find
  3. Akari Asai, Jungo Kasai, Jonathan Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2021. XOR QA: Cross-lingual open-retrieval question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 547–564, Online. Association for Computational Linguistics. ↗
  4. Israel Abebe Azime and Nebil Mohammed. 2021. An Amharic news text classification dataset. arXiv preprint arXiv:2103.05639. ↗
  5. Shaik Johny Basha, Duggineni Veeraiah, Boddu Venkat Charan, Wiltrud Sahithi Joyce Yeddu, and Devalla Ganesh Babu. 2023. Detection and comparative analysis of handwritten words of Amharic language to english using CNN-based frameworks. In 2023 International Conference on Inventive Computation Technologies (ICICT), pages 422–427. IEEE. find
  6. Luiz Bonifacio, Vitor Jeronymo, Hugo Queiroz Abonizio, Israel Campiotti, Marzieh Fadaee, Roberto Lotufo, and Rodrigo Nogueira. 2021. mMARCO: A multilingual version of the MS MARCO passage ranking dataset. arXiv preprint arXiv:2108.13897. ↗
  7. Antoine Chaffin and Raphaël Sourty. 2025. PyLate: Flexible training and retrieval for late interaction models. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, CIKM 2025, Seoul, Republic of Korea, November 10-14, 2025, pages 6334–6339. ACM. find
  8. Nadezhda Chirkova, David Rau, Hervé Déjean, Thibault Formal, Stéphane Clinchant, and Vassilina Nikoulina. 2024. Retrieval-augmented generation in multilingual settings. In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pages 177–188, Bangkok, Thailand. Association for Computational Linguistics. find
  9. David M. Eberhard, Gary F. Simons, and Charles D. Fennig. 2024. Ethnologue: Languages of the World, 27th edition. SIL International, Dallas, Texas. Tahmid Hasan, Abhik Bhattacharjee, Md. find
  10. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693–4703. ACL. find
  11. Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. arXiv preprint arXiv:1705.00652. ↗
  12. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the 29th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, page 1693–1701, Cambridge, MA, USA. MIT Press. ↗
  13. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, page 39–48, New York, NY, USA. Association for Computing Machinery. ↗
  14. Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. 2022. Matryoshka representation learning. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc. find
  15. Kidist Amde Mekonnen, Yosef Worku Alemneh, and Maarten de Rijke. 2025. Optimized text embedding models and benchmarks for Amharic passage retrieval. In Findings of the Association for Computational Linguistics: ACL 2025, pages 10428–10445, Vienna, Austria. Association for Computational Linguistics.
  16. Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2022. From distillation to hard negative sampling: Making sparse neural IR models more effective. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22, page 2353–2359, New York, NY, USA. Association for Computing Machinery. ↗
  17. Microsoft. 2026. Harrier-OSS-v1-270M. Hugging Face model card. find
  18. Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. SPLADE: Sparse lexical and expansion model for first stage ranking. In Proceedings Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. arXiv preprint arXiv:1901.04085. ↗
  19. Thong Nguyen, Yibin Lei, Jia-Huei Ju, Eugene Yang, and Andrew Yates. 2026. Milco: Learned sparse retrieval across languages via a multilingual connector. In International Conference on Learning Representations. ↗
  20. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics. ↗
  21. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? On the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3118–3135, Online. Association for Computational Linguistics.
  22. Nandan Thakur, Luiz Bonifacio, Crystina Zhang, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Boxing Chen, Mehdi Rezagholizadeh, and Jimmy Lin. 2024. “Knowing when you don’t know”: A multilingual relevance assessment dataset for robust retrieval-augmented generation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12508–12526, Miami, Florida, USA. Association for Computational Linguistics. find
  23. Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. 2024. mGTE: Generalized long-context text representation and reranking models for multilingual text retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1393–1412, Miami, Florida, US. Association for Computational Linguistics. ↗
  24. Xinyu Zhang, Kelechi Ogueji, Xueguang Ma, and Jimmy Lin. 2023a. Toward best practices for training multilingual dense retrieval models. ACM Trans. Inf. Syst., 42(2). ↗
  25. Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. 2023b. MIRACL: A multilingual retrieval dataset covering 18 diverse languages. Transactions of the Association for Computational Linguistics, 11:1114–1131. ↗
  26. Henrique Schechter Vera, Sahil Dua, Biao Zhang, Daniel Salz, Ryan Mullins, Sindhu Raghuram Panyam, Sara Smoot, Iftekhar Naim, Joe Zou, Feiyang Chen, et al. 2025. EmbeddingGemma: Powerful and lightweight text representations. arXiv preprint arXiv:2509.20354. ↗
  27. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual E5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672. ↗
  28. Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou. 2020. MIND: A large-scale dataset for news recommendation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3597–3606, Online. Association for Computational Linguistics.
  29. Tilahun Yeshambel, Josiane Mothe, and Yaregal Assabie. 2020. 2AIRTC: The Amharic adhoc information retrieval test collection. In Experimental IR Meets Multilinguality, Multimodality, and Interaction: 11th International Conference of the CLEF Association, CLEF 2020, Thessaloniki, Greece, September 22–25, 2020, Proceedings, page 55–66, Berlin, Heidelberg. Springer-Verlag. find
  30. Puxuan Yu, Luke Merrick, Gaurav Nuti, and Daniel Campos. 2024. Arctic-Embed 2.0: Multilingual retrieval without compromise. arXiv preprint arXiv:2412.04506. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alemneh-2026-multilingual-curse-retrieval-layer-evidence/