Elamur Research⌕
☰
home / papers / 2026 / AmharicIR+Instr: A Two-Dataset Resource for Neural Retrieval and Instruction Tuning

AmharicIR+Instr: A Two-Dataset Resource for Neural Retrieval and Instruction Tuning

Paper · 2026 · Tilahun Yeshambel, Moncef Garouani and Josiane Mothe

Summary

Amharic is a morphologically rich Semitic language with more than 80 million speakers. It has few curated benchmarks for neural retrieval and almost no quality-checked instruction data. Existing Amharic retrieval resources are small or weakly supervised. For example, one earlier benchmark pairs news headlines with their articles and produces negatives only on the fly during training. The authors release two datasets to fill this gap. The first holds 1,091 query–positive–negative triplets with explicitly stored negatives. The second holds 6,285 prompt–response pairs for instruction tuning. Both come from one pipeline that mixes expert-written queries, web-harvested material and LLM-generated content, with native speakers checking or correcting every item. The data ship in CSV/JSON/JSONL with fixed splits and dataset statements. As a baseline, the authors fine-tune three pretrained Amharic retrievers on the triplets: sparse, dense and late-interaction. The dense RoBERTa model performs best. The resource gives Amharic a reproducible testbed for retrieval, generation and their combination (RAG), and the construction recipe is meant to transfer to other low-resource languages.

Key points

  • The retrieval set has 1,091 queries and 2,182 documents (one positive and one negative per query), with qrels. It is split 654 / 218 / 219 for train / validation / test.

  • The instruction set has 6,285 Amharic prompt–response pairs across several domains and instruction types. Three LLMs produced the initial drafts, which people then revised by hand so that each pair is grammatically correct, on-topic, fluent and believable in its factual content.

  • The baselines are SPLADE-RoBERTa-Amharic-Medium, RoBERTa-Amharic-Embed-Medium and ColBERT-Amharic-Medium, pretrained models from the Hugging Face account rasyosef, each fine-tuned on the triplets.

  • Test results:

    ModelNDCG@1NDCG@10Recall@100
    RoBERTa.744.845.986
    ColBERT.666.787.981
    SPLADE.598.769.977

  • Precision falls steeply beyond K=3, down to .009 at K=100. The authors explain this by each query having very few relevant documents.

Interesting ideas

  • I1. Combining three sources of examples (expert-authored, web-harvested, LLM-synthesized) and putting all of them through the same native-speaker check gives volume without giving up trust, and the provenance of each item stays known.
  • I2. Storing explicit, verified negatives inside the dataset, rather than relying on in-batch or sampled ones, makes contrastive training and evaluation reproducible and easy to inspect.
  • I3. Building a retrieval set and an instruction set side by side in one language and setting allows retrieval-augmented generation to be studied on shared material.
  • I4. The paper reads its metric curves against the qrels design: precision collapsing at higher K reflects having one relevant document per query, not model weakness.
  • I5. Shipping a separate validation split along with documented collection, quality-control and limitation statements fixes a gap in the earlier benchmark it cites, which had only train and test.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver training data could carry a provenance tag (hand-written, corpus-mined, LLM-teacher) with one shared human or witness check, so later evaluation can be broken down by source tier.
  • I2 (answer): QA sets on fables could include, for each question, a vetted distractor sentence that looks relevant but is not. This gives a fixed negative control showing whether the reader grounds its answer or just matches words.
  • I3 (answer): School-test and tale QA could pair each question with its supporting passage from the text, so evidence selection and answer production are scored separately and each failure can be explained.
  • I4 (learn): Evaluation reports should state the ceiling each metric can reach given how gold annotations are structured, so that apparent drops are not mistaken for regressions.
  • I5 (tree): Converted or silver treebanks could always be released with a held-out dev split and a short datasheet, so parser tuning never touches the test set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Addis Ababa UniversityAddis Ababa · Ethiopia11 works · 34 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksInformation Systems · 523 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 1,344 worksSemitic languages · 1,485 worksInformation retrieval & search · 146 works

Cite

Cite as Tilahun Yeshambel, Moncef Garouani and Josiane Mothe. 2026. AmharicIR+Instr: A Two-Dataset Resource for Neural Retrieval and Instruction Tuning. arXiv:2602.09914.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2602.09914 · arXiv 2602.09914

@misc{yeshambel2026amharicir,
  title = {{AmharicIR+Instr: A Two-Dataset Resource for Neural Retrieval and Instruction Tuning}},
  author = {Tilahun Yeshambel, Moncef Garouani and Josiane Mothe},
  year = {2026},
  doi = {10.48550/arxiv.2602.09914},
  eprint = {2602.09914},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2602.09914}
}

References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kindie Alebachew, Durga Prasad Sharma, and Messay Samuel. 2025. Navigating the NLP and text-based conversational AI: a survey for problem formulation in Amharic texts. Discover Data 3 (2025). doi:10.1007/s44248-025-00077-9 ↗
  2. Israel Abebe Azime and Nebil Mohammed. 2021. An Amharic News Text Classification Dataset. arXiv preprint arXiv:2103.05639 (2021). arXiv:2103.05639 [cs.CL] https://arxiv.org/abs/2103.05639 ↗
  3. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. MS MARCO: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 (2016). ↗
  4. Gashaw Demlew and Dessalew Yohannes. 2022. Resolving Amharic Lexical Ambiguity using Neural Word Embedding. In 2022 International Conference on Information and Communication Technology for Development for Africa (ICT4DA). 55–60. doi:10.1109/ICT4DA56482.2022.9971344 ↗
  5. DM Eberhard, GF Simons, and Fennig CD. 2023. CD (Eds.)," Ethnologue: Languages of the World. Twenty-sixth edition", Dallas, Texas: SIL International, Turkish Language Ethnologue (2023). find
  6. Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2021. SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval. arXiv:2109.10086 [cs.IR] https://arxiv.org/abs/2109.10086 ↗
  7. Roksana Goworek, Olivia Macmillan-Scott, and Eda B. Özyiğit. 2025. What Drives Cross-lingual Ranking? Retrieval Approaches with Multilingual Language Models. (2025). arXiv:2511.19324 [cs.IR] doi:10.48550/arXiv.2511.19324 ↗
  8. Xiyang Hu, Xinchi Chen, Peng Qi, Deguang Kong, Kunlun Liu, William Yang Wang, and Zhiheng Huang. 2023. Language Agnostic Multilingual Information Retrieval with Contrastive Learning. In Findings of the Association for Computational Linguistics: ACL 2023, Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 9133–9146. doi:10.18653/v1/2023.findings-acl.581 ↗
  9. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 6769–6781. doi:10.18653/v1/2020.emnlp-main.550 ↗
  10. Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, China) (SIGIR ’20). Association for Computing Machinery, New York, NY, USA, 39–48. doi:10.1145/3397271.3401075 ↗
  11. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: A Benchmark for Question Answering Research. Transactions of the Association for Computational Linguistics 7 (2019), 452–466. doi:10.1162/tacl_a_00276 ↗
  12. Carlos Lassance, Simon Lupart, Hervé Déjean, Stéphane Clinchant, and Nicola Tonellotto. 2023. A Static Pruning Study on Sparse Neural Retrievers. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (Taipei, Taiwan) (SIGIR ’23). Association for Computing Machinery, New York, NY, USA, 1771–1775. doi:10.1145/3539618.3591941 ↗
  13. Robert Litschko, Ivan Vulić, Simone Paolo Ponzetto, and Goran Glavaš. 2022. On cross-lingual retrieval with multilingual text encoders. Information Retrieval Journal 25 (2022), 149–183. doi:10.1007/s10791-022-09406-x ↗
  14. Kidist Amde Mekonnen, Yosef Worku Alemneh, and Maarten de Rijke. 2025. Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval. In Findings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 10428–10445. doi:10.18653/v1/2025.findings-acl.543 ↗
  15. Kidist Amde Mekonnen, Yosef Worku Alemneh, and Maarten de Rijke. 2025. Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval. In Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics, 10428–10445. doi:10.18653/v1/2025.findings-acl.543 ↗
  16. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. In NeurIPS 2021 Datasets and Benchmarks Track. doi:10.48550/arXiv.2104.08663 ↗
  17. Atnafu Lambebo Tonja, Tadesse Destaw Belay, Israel Abebe Azime, Abinew Ali Ayele, Moges Ahmed Mehamed, Olga Kolesnikova, and Seid Muhie Yimam. 2023. Natural Language Processing in Ethiopian Languages: Current State, Challenges, and Opportunities. In Proceedings of the Fourth workshop on Resources for African Indigenous Languages (RAIL 2023), Rooweither Mabuya, Don Mthobela, Mmasibidi Setaka, and Menno Van Zaanen (Eds.). Association for Computational Linguistics, Dubrovnik, Croatia, 126–139. doi:10.18653/v1/2023.rail-1.14 ↗
  18. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations Poster. https://openreview.net/forum?id=zeFrfgyZln ↗
  19. Tilahun Yeshambel, Moncef Garouani, Serge Molina, and Josiane Mothe. 2025. Dense Retrieval for Low Resource languages-the Case of Amharic Language. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. 3098–3100. ↗
  20. Tilahun Yeshambel, Josiane Mothe, and Yaregal Assabie. 2020. Amharic document representation for adhoc retrieval. In Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management. find
  21. Dessalew Yohannes, Yenewondim Biadgie Sinshaw, Surafiel Habib Asefa, and Yaregal Assabie. 2025. Amharic document clustering using semantic information from neural word embedding and encyclopedic knowledge. Scientific African 28 (2025), e02657. ↗
  22. Xinyu Zhang, Kelechi Ogueji, Xueguang Ma, and Jimmy Lin. 2023. Toward best practices for training multilingual dense retrieval models. ACM Transactions on Information Systems 42, 2 (2023), 1–33. ↗
  23. Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. 2024. Dense text retrieval based on pretrained language models: A survey. ACM Transactions on Information Systems 42, 4 (2024), 1–60. Received 10 February 2026 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yeshambel-2026-amharicir-instr-two-dataset-resource/