Elamur Research⌕
☰
home / papers / 2025 / Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval

Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval

Paper · 2025 · Kidist Amde Mekonnen, Yosef Worku Alemneh and Maarten de Rijke

Summary

Amharic is a low-resource Semitic language with root-and-pattern morphology and the Ge'ez abugida script. Neither lexical matching nor large multilingual embedders handle it well, and the authors trace much of the trouble to poor subword segmentation. They fine-tune existing Amharic BERT and RoBERTa checkpoints, pre-trained on about 300M tokens, into dense passage retrievers. They also train a ColBERT-style late-interaction model. Training and evaluation use a new resource, the Amharic Passage Retrieval Dataset. It is built from AM-NEWS by pairing each news headline, used as a query, with its own article, which gives MS MARCO-style pairs with heuristic relevance labels. All systems are compared against sparse retrieval (BM25) and strong multilingual dense models. The 110M-parameter RoBERTa-Base-Amharic-Embed beats the much larger Arctic Embed 2.0. A 42M variant stays competitive, and the ColBERT model scores best overall. The work is the first systematic sparse-versus-dense benchmark for Amharic. It shows that compact, language-specific models can outperform big multilingual ones, and it releases data, code and models publicly.

Key points

  • RoBERTa-Base-Amharic-Embed (110M) beats Arctic Embed 2.0 (568M), the strongest multilingual baseline. Its MRR@10 is 17.6% higher in relative terms, and Recall@10 rises by 9.86%.
  • RoBERTa-Medium-Amharic-Embed (42M) remains competitive while being more than 13× smaller than Arctic Embed 2.0.
  • The ColBERT-based late-interaction retriever reaches the top MRR@10 of all evaluated systems (0.843).
  • Two benchmarks are used, each with caveats:
    • The Amharic Passage Retrieval Dataset (AM-NEWS headlines → articles) is on Hugging Face but has no gold human judgments.
    • The TREC-style 2AIRTC collection has incomplete relevance labels, which makes recall-based scores unreliable.
  • Tokenization quality is analysed with subword fertility, i.e. how many pieces each word is split into. Over-segmentation of templatic words is identified as a main cause of weak multilingual performance (e.g. Multilingual E5, Arctic Embed 2.0).

Interesting ideas

  • I1. An existing document structure can be turned into free supervision: each news headline becomes a pseudo-query whose only relevant passage is its own article, so no annotation is needed.
  • I2. Subword fertility works as a simple diagnostic that ties tokenizer behaviour to downstream failures, instead of blaming the model as a whole.
  • I3. A small model adapted to one language beat a model five times larger, which suggests that targeted adaptation can matter more than scale.
  • I4. Late interaction scores relevance through token-to-token matches, which is more fine-grained than a single pooled vector, and here it gave the best ranking.
  • I5. The authors evaluate on two collections and openly state that incomplete judgments make recall unreliable, so label gaps are treated as a property of the benchmark.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Titles, morals or first sentences of tales could serve as cheap pseudo-questions, letting the answer stage be trained or checked against texts we already have.
  • I2 (tokens): Fertility could be tracked per POS or per morphological class in the en tokenizer, flagging words whose splits cut across lemma/affix boundaries before those errors reach the tree.
  • I3 (learn): This supports training compact, specialised parsers and readers on silver data, rather than relying on a large general LLM teacher at inference time.
  • I4 (answer): Question words could be explicitly aligned to passage tokens or frame slots, and those alignment links kept as an explainable trace of why a passage was chosen.
  • I5 (learn): Evaluation reports should mark which test sets have partial gold, so that apparent misses on unjudged items are not counted as model errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands144 works · 173,906 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 4Research subjects and languages the work is about.Semitic languages · 2,908 worksEvaluation & benchmarks · 1,220 worksMultilingual & cross-lingual NLP · 828 worksInformation retrieval & search · 473 works

Cite

Cite as Kidist Amde Mekonnen, Yosef Worku Alemneh and Maarten de Rijke. 2025. Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval. Findings of the Association for Computational Linguistics: ACL 2025.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.findings-acl.543

@inproceedings{mekonnen2025optimized,
  title = {{Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval}},
  author = {Kidist Amde Mekonnen, Yosef Worku Alemneh and Maarten {de Rijke}},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2025},
  year = {2025},
  doi = {10.18653/v1/2025.findings-acl.543},
  url = {https://doi.org/10.18653/v1/2025.findings-acl.543}
}

References

42 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Alcides Alcoba Inciarte. 2023. find
  2. SERENGETI: Massively multilingual language models for Africa. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1498– 1537, Toronto, Canada. Association for Computational Linguistics. find
  3. Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, Stefan Kesselheim, and Nicolas Flores-Herr. 2024. Tokenizer choice for LLM training: Negligible or crucial? In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3907–3924, Mexico City, Mexico. Association for Computational Linguistics. ↗
  4. Israel Abebe Azime, Mitiku Yohannes Fuge, Atnafu Lambebo Tonja, Tadesse Destaw Belay, Aman Kassahun Wassie, Eyasu Shiferaw Jada, Yonas Chanie, Walelign Tewabe Sewunetie, and Seid Muhie Yimam. 2024a. Enhancing Amharic-LLaMA: Integrating task specific and generative datasets. arXiv preprint arXiv:2402.08015. ↗
  5. Israel Abebe Azime and Nebil Mohammed. 2021. An Amharic news text classification dataset. arXiv preprint arXiv:2103.05639. ↗
  6. Israel Abebe Azime, Atnafu Lambebo Tonja, Tadesse Destaw Belay, Mitiku Yohannes Fuge, Aman Kassahun Wassie, Eyasu Shiferaw Jada, Yonas Chanie, Walelign Tewabe Sewunetie, and Seid Muhie Yimam. 2024b. Walia-LLM: Enhancing Amharic-LLaMA by integrating task-specific and generative datasets. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 432–444, Miami, Florida, USA. Association for Computational Linguistics. ↗
  7. Tadesse Destaw Belay, Abinew Ayele, and Seid Muhie Yimam. 2021. The development of pre-processing tools and pre-trained embedding models for Amharic. In Proceedings of the Fifth Workshop on Widening Natural Language Processing, pages 25–28, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  8. Daniel Fernando Campos, Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng, and Bhaskar Mitra. 2016. MS MARCO: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268. ↗
  9. Antoine Chaffin and Raphaël Sourty. 2024. Pylate: Flexible training and retrieval for late interaction models. find
  10. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870–1879, Vancouver, Canada. Association for Computational Linguistics. ↗
  11. Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yiqun Liu, Yixing Fan, and Xueqi Cheng. 2023. A unified generative retriever for knowledge-intensive language tasks via prompt learning. In SIGIR. ACM. find
  12. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  13. Zhuyun Dai and Jamie Callan. 2020. Context-aware term weighting for first stage passage retrieval. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 1533–1536. find
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  15. Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2021a. Splade v2: Sparse lexical and expansion model for information retrieval. arXiv preprint arXiv:2109.10086. ↗
  16. Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021b. Splade: Sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2288–2292. ↗
  17. Andargachew Mekonnen Gezmu, Binyam Ephrem Seyoum, Michael Gasser, and Andreas Nürnberger. 2018. Contemporary Amharic corpus: Automatically morpho-syntactically tagged Amharic corpus. In Proceedings of the First Workshop on Linguistic Resources for Natural Language Processing, pages 65–70, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  18. Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. find
  19. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. In TMLR. ↗
  20. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547. ↗
  21. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics. ↗
  22. Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and effective passage search via contextualized late interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, page 39–48, New York, NY, USA. Association for Computing Machinery. ↗
  23. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466. ↗
  24. Donald Metzler, Yi Tay, and Dara Bahri. 2021. Rethinking search. ACM SIGIR Forum, 55:1 – 27. find
  25. Hellina Hailu Nigatu and Inioluwa Deborah Raji. 2024. “i searched for a religious song in amharic and got sexual content instead”: Investigating online harm in low-resourced languages on youtube. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’24), Rio de Janeiro, Brazil. ACM. find
  26. Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. arXiv preprint arXiv:1901.04085. ↗
  27. Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019. Multi-stage document ranking with BERT. arXiv preprint arXiv:1910.14424. ↗
  28. Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin. 2021. Small data? No problem! Exploring the viability of pretrained multilingual language models for low-resourced languages. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 116–126, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  29. Stephen Della Pietra, Mark Epstein, Salim Roukos, and Todd Ward. 1997. Fertility models for statistical natural language understanding. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics, ACL ’98/EACL ’98, page 168–173, USA. Association for Computational Linguistics. find
  30. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001, Florence, Italy. Association for Computational Linguistics. ↗
  31. Ronak Pradeep, Kai Hui, Jai Gupta, Adam Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, and Vinh Tran. 2023. How does generative retrieval scale to millions of passages? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1305–1321, Singapore. Association for Computational Linguistics. find
  32. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  33. Stephen E. Robertson and Steve Walker. 1997. On relevance weights with little relevance information. In Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, page 16–24, New York, NY, USA. Association for Computing Machinery. ↗
  34. Stephen E. Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval. NOW Publishers. ↗
  35. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3118–3135, Online. Association for Computational Linguistics.
  36. Walelign Sewunetie, Atnafu Tonja, Tadesse Belay, Hellina Hailu Nigatu, Gashaw Gebremeskel, Zewdie Mossie, Hussien Seid, and Seid Yimam. 2024. Gender bias evaluation in machine translation for Amharic, Tigrigna, and Afaan Oromoo. In Proceedings of the 2nd International Workshop on Gender-Inclusive Translation Technologies, pages 1–11, Sheffield, United Kingdom. European Association for Machine Translation (EAMT).
  37. Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, and Daniel Khashabi. 2024. The language barrier: Dissecting safety challenges of LLMs in multilingual contexts. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2668– 2680, Bangkok, Thailand. Association for Computational Linguistics.
  38. Yi Tay, Vinh Quang Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. 2022. Transformer memory as a differentiable search index. In NeurIPS.
  39. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05355. ↗
  40. Cagri Toraman, Eyup Halit Yilmaz, Furkan Şahinüc, and Oguzhan Ozcelik. 2023. Impact of tokenization on language models: An analysis for Turkish. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 22(4). ↗
  41. Ahmet Üstün, Gosse Bouma, and Gertjan van Noord. 2019. Cross-lingual word embeddings for morphologically rich languages. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 1222– 1228, Varna, Bulgaria. INCOMA Ltd. find
  42. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual E5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mekonnen-2025-optimized-text-embedding-models-benchmarks/