Elamur Research⌕
☰
home / papers / 2026 / AmharicStoryQA: A Multicultural Story Question Answering Benchmark in Amharic

AmharicStoryQA: A Multicultural Story Question Answering Benchmark in Amharic

Summary

Multilingual benchmarks usually equate one language with one culture. This paper argues that cultural differences inside a single language community also shape how well LLMs understand text. The authors built AmharicStoryQA from parallel Amharic–English folktales drawn from nine Ethiopian regions, with SNNP further divided into zones. GPT-4.1 generated question–answer sets with structured distractors. Three annotators rated the questions, translation quality was screened with SSA-COMET, and the data was split randomly into train and test. The benchmark covers both multiple-choice questions and open-ended generation over long stories.

The authors tested seven open-source LLMs zero-shot and found three things:

  • Amharic story comprehension is weak.
  • In English, accuracy varies noticeably by region.
  • Prompt language matters.

LoRA fine-tuning of Llama 3.1 8B Instruct on the training split helped substantially, especially for Amharic multiple-choice questions, and made results more even across regions. The work matters because evaluation at the level of the whole language can hide region-specific failures. That risk is greatest for low-resource languages.

Key points

  • The dataset has 571 training and 649 test items from folktales across nine regions, in Amharic and English, with MCQA and generation formats. The contributions section gives 244 stories, while the methodology section gives 224.
  • GPT-4.1 produced five questions per story in a single structured-output call. Each question has one correct answer and three typed distractors:
    • one biased by world knowledge,
    • one unrelated,
    • one factually false.
  • Three human raters judged each question on correctness/faithfulness, linguistic quality, and challenge level. Agreement was measured with Gwet's AC1, chosen for robustness to class imbalance.
  • Zero-shot, English results swing by region by up to about ±10 chrF or roughly 20% accuracy, though Gemma-3-27b-it MCQA stays stable. Amharic shows less regional spread because scores are uniformly low.
  • LoRA SFT on Llama 3.1 8B Instruct gave the following gains:
    • English generation rose from 15.3 to 28.9 chrF.
    • Amharic MCQA with an Amharic prompt rose from about 0.38 to about 0.79.
    • Training on stories in one language improved evaluation in the other language.

Interesting ideas

  • I1. Each wrong option has a defined role (world-knowledge bait, irrelevant, false), so errors reveal which failure mode a model has rather than just a wrong answer. — For Mova (answer): Fable/tale QA could tag distractors by type and report whether the reader picks the "common sense instead of the text" option, exposing places where the world of the text is overridden by background knowledge.
  • I2. The world-knowledge-biased distractor works as a built-in negative control: a system that truly reads the story should reject an answer that is plausible in general but unsupported by the narrative. — For Mova (learn): Evaluation could include such controls automatically, checking that answers are derived from the text graph and not from the compiled ontology alone.
  • I3. Results are broken down by the cultural or regional origin of texts within one language, which shows that a single aggregate score hides uneven competence. — For Mova (world): Tale QA results could be split by story source or domain (pastoral, trade, court) to find gaps in verb classes or ontology coverage for culture-specific activities like herding or cattle theft.
  • I4. Generating all questions for a story in one structured call avoids repeated ideas across questions and parsing errors that separate generations can cause. — For Mova (learn): LLM teacher tiers producing silver QA or frame annotations could request a full per-text batch as a typed schema, giving diverse, machine-checkable items.
  • I5. Story density in a shared embedding space (KDE over PCA of MiniLM embeddings) gives a cheap visual check of how distinct subcorpora really are before drawing conclusions about them. — For Mova (learn): A similar check could verify that test fables differ from training ones before claiming generalization.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2602.02774 · arXiv 2602.02774

@misc{azime2026amharicstoryqa,
  title = {{AmharicStoryQA: A Multicultural Story Question Answering Benchmark in Amharic}},
  author = {Azime, Israel Abebe and Angamo, Abenezer Kebede and Tamiru, Hana Mekonen and Marilign, Dagnachew Mekonnen and Slusallek, Philipp and Yimam, Seid Muhie and Klakow, Dietrich},
  year = {2026},
  doi = {10.48550/arxiv.2602.02774},
  eprint = {2602.02774},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2602.02774}
}

References

20 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba Oluwadara Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Ijeoma Chukwuneke, Happy Buzaaba, Blessing Kudzaishe Sibanda, Godson Koffi Kalipe, Jonathan Mukiibi, Salomon Kabongo Kabenamualu, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela, and 8 others. 2025. IrokoBench: A new benchmark for African languages in the age of large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2732–2757, Albuquerque, NM, USA. Association for Computational Linguistics. find
  2. Jesujoba Oluwadara Alabi, Israel Abebe Azime, Miaoran Zhang, Cristina España-Bonet, Rachel Bawden, Dawei Zhu, David Ifeoluwa Adelani, Clement Oyeleke Odoje, Idris Akinade, Iffat Maab, Davis David, Shamsuddeen Hassan Muhammad, Neo Putini, David O. Ademuyiwa, Andrew Caines, and Dietrich Klakow. 2025a. AFRIDOC-MT: Document-level MT corpus for African languages. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 27758–27794, Suzhou, China. Association for Computational Linguistics. find
  3. Jesujoba Oluwadara Alabi, Michael A. Hedderich, David Ifeoluwa Adelani, and Dietrich Klakow. 2025b. Charting the landscape of African NLP: Mapping progress and shaping the road ahead. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 27795–27829, Suzhou, China. Association for Computational Linguistics. find
  4. Israel Abebe Azime, Tadesse Destaw Belay, Dietrich Klakow, Philipp Slusallek, and Anshuman Chhabra. 2025a. Bridging the Culture Gap: A Framework for LLM-Driven Socio-Cultural Localization of Math Word Problems in Low-Resource Languages. arXiv preprint arXiv:2508.14913. ↗ our card
  5. Israel Abebe Azime, Atnafu Lambebo Tonja, Tadesse Destaw Belay, Yonas Chanie, Bontu Fufa Balcha, Negasi Haile Abadi, Henok Biadglign Ademtew, Mulubrhan Abebe Nerea, Debela Desalegn Yadeta, Derartu Dagne Geremew, Assefa Ats-biha Tesfu, Philipp Slusallek, Thamar Solorio, and Dietrich Klakow. 2025b. ProverbEval: Exploring LLM evaluation challenges for low-resource language understanding. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 6250–6266, Albuquerque, NM, USA. Association for Computational Linguistics. find
  6. Israel Abebe Azime, Atnafu Lambebo Tonja, Tadesse Destaw Belay, Mitiku Yohannes Fuge, Aman Kassahun Wassie, Eyasu Shiferaw Jada, Yonas Chanie, Walelign Tewabe Sewunetie, and Seid Muhie Yimam. 2024. Walia-LLM: Enhancing Amharic-LLaMA by integrating task-specific and generative datasets. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 432–444, Miami, FL, USA. Association for Computational Linguistics. find
  7. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 749–775, Bangkok, Thailand. Association for Computational Linguistics. find our card
  8. Tadesse Destaw Belay, Seid Muhie Yimam, Abinew Ayele, and Chris Biemann. 2022. Question answering classification for Amharic social media community based questions. In Proceedings of the 1st Annual Meeting of the ELRA/ISCA Special Interest Group on Under-Resourced Languages, pages 137–145, Marseille, France. European Language Resources Association. find
  9. Thomas Berkane, Marie-Laure Charpignon, and Maimuna S. Majumder. 2025. LLM-based web data collection for research dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 12610–12622, Suzhou, China. Association for Computational Linguistics. find
  10. Chen Cecilia Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2024. Are multilingual LLMs culturally-diverse reasoners? an investigation into multicultural proverbs and sayings. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2016–2039, Mexico City, Mexico. Association for Computational Linguistics. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others. 2024. A framework for few-shot language model evaluation. find
  11. Shreya Havaldar, Sunny Rai, Bhumika Singhal, Langchen Liu, Sharath Chandra Guntuku, and Lyle Ungar. 2023. Multilingual language models are not multicultural: A case study in emotion. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 202–214, Toronto, Canada. Association for Computational Linguistics. find
  12. Senyu Li, Jiayi Wang, Felermino D. M. A. Ali, Colin Cherry, Daniel Deutsch, Eleftheria Briakou, Rui Sousa-Silva, Henrique Lopes Cardoso, Pontus Stenetorp, and David Ifeoluwa Adelani. 2025. SSA-COMET: Do LLMs outperform learned metrics in evaluating MT for under-resourced African languages? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 12979–12998, Suzhou, China. Association for Computational Linguistics. find our card
  13. Olga Loginova, Oleksandr Bezrukov, Ravi Shekhar, and Alexey Kravets. 2025. Addressing Blind Guessing: Calibration of Selection Bias in Multiple-Choice Question Answering by Video Language Models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3216–3246, Vienna, Austria. Association for Computational Linguistics. find
  14. Hellina Hailu Nigatu, Atnafu Lambebo Tonja, Benjamin Rosman, Thamar Solorio, and Monojit Choudhury. 2024. The Zeno’s Paradox of ‘Low-Resource’ Languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17753–17774, Miami, Florida, USA. Association for Computational Linguistics. find
  15. Odunayo Ogundepo, Tajuddeen R. Gwadabe, Clara E. Rivera, Jonathan H. Clark, Sebastian Ruder, David Ifeoluwa Adelani, Bonaventure F. P. Dossou, Abdou Aziz Diop, Claytone Sikasote, Gilles Hacheme, Happy Buzaaba, Ignatius Ezeani, Rooweither Mabuya, Salomey Osei, Chris Emezue, Albert Njoroge Kahira, Shamsuddeen Hassan Muhammad, Akintunde Oladipo, Abraham Toluwase Owodunni, and 33 others. 2023. AfriQA: Cross-lingual open-retrieval question answering for African languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14957– 14972, Singapore. Association for Computational Linguistics. find
  16. Jessica Ojo, Odunayo Ogundepo, Akintunde Oladipo, Kelechi Ogueji, Jimmy Lin, Pontus Stenetorp, and David Ifeoluwa Adelani. 2025. AfroBench: How good are large language models on African languages? In Findings of the Association for Computational Linguistics: ACL 2025, pages 19048–19095, Vienna, Austria. Association for Computational Linguistics. David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco, David Ifeoluwa Adelani, David Le Meur, Emilio Villa-Cueva, Fajri Koto, Fauzan Farooqui, and 56 others. 2024. CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark. Preprint, arXiv:2406.05967. ↗
  17. Dipankar Srirag, Nihar Ranjan Sahoo, and Aditya Joshi. 2025. Evaluating Dialect Robustness of Language Models via Conversation Understanding. In Proceedings of the Second Workshop on Scaling Up Multilingual & Multi-Cultural Evaluation, pages 24–38, Abu Dhabi, UAE. Association for Computational Linguistics. find
  18. Tilahun Abedissa Taffa, Ricardo Usbeck, and Yaregal Assabie. 2024. Low Resource Question Answering: An Amharic Benchmarking Dataset. In Proceedings of the Fifth Workshop on Resources for African Indigenous Languages @ LREC-COLING 2024, pages 124–132, Torino, Italia. ELRA and ICCL. find
  19. Sizhe Yuen, Ting Su, Ziyang Wang, Yali Du, and Adam J Sobey. 2025. Automatic Dataset Generation for Knowledge Intensive Question Answering Tasks. arXiv preprint arXiv:2505.14212. ↗
  20. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand. Association for Computational Linguistics. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/azime-2026-amharicstoryqa-multicultural-story-question-answering/