Elamur Research⌕
☰
home / papers / 2024 / Low Resource Question Answering: An Amharic Benchmarking Dataset

Low Resource Question Answering: An Amharic Benchmarking Dataset

Paper · 2024 · Tilahun Abedissa Taffa, Ricardo Usbeck and Yaregal Assabie

Summary

Amharic had no public question answering dataset, so neither extractive readers nor open-domain QA could be trained or compared. The authors built Amh-QuAD, which they describe as the first openly released Amharic QA benchmark. It is factoid, extractive and open-domain. Crowd workers read 378 Amharic Wikipedia articles and wrote 2,628 question–answer pairs, each answer being a span of the source text. Annotation followed the Haystack guideline. The authors chose crowdsourcing over translating SQuAD because no well-tested open English-to-Amharic machine translation tool exists. They fine-tuned an XLM-R model on the training split as a reader and combined it with a retriever into a baseline system called AmhQA. The reader reaches F1 81.34 when given the correct context and 80.3 when it must work on retrieved passages. The dataset is published on GitHub. It gives Amharic, and Ge'ez-script NLP more broadly, a shared yardstick for future QA research.

Key points

  • Amh-QuAD contains 2,628 crowdsourced question–answer pairs drawn from 378 Amharic Wikipedia contexts, stored as context–question–answer triplets in the SQuAD style.
  • The reader is a fine-tuned XLM-R model. It scores F1 81.34 in reading comprehension and 80.3 inside the full retriever-reader pipeline.
  • Annotators followed the Haystack guideline. They were asked to flag articles with offensive content and to write as many questions per context as they could.
  • The paper lists the ways Amharic forms questions:
    • interrogative pronouns: ምን, መቼ, ማን, የት, የትኛው;
    • preposition + pronoun compounds, such as ለ-ምን "why";
    • imperative verbs: explain, list, compare.
  • The Tigrinya dataset TigQuAD was judged unsuitable for Amharic despite the shared Semitic family. Earlier Amharic QA systems (TETEYEQ, Abedissa & Libsie 2019, Elema 2022) released no data.

Interesting ideas

  • I1. Scoring the reader twice, once with the gold context and once with retrieved passages, separates reading errors from retrieval errors in a single benchmark.
  • I2. Every answer must be a literal span of the context, so each gold answer comes with its own evidence and is easy to check.
  • I3. Questions are recognised not only by wh-words but also by imperative verbs such as "explain", "list" and "compare".
  • I4. Some interrogatives are built from a preposition plus a pronoun (for-what = why), so question meaning is spread across morphemes or words.
  • I5. When no reliable translation tool existed, the authors chose fewer human-written items over a larger machine-translated set.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Fable and tale QA could be scored both with the gold sentence supplied and with Mova locating it in the full text, which shows whether misses come from the world model or from the search for evidence.
  • I2 (answer): Mova's readers could be required to link each answer to the text span or world entity it came from, keeping answers explainable and verifiable.
  • I3 (frames): The speech-act layer in prag could classify imperatives like "List the animals that…" or "Compare X and Y" as information requests with an expected answer type.
  • I4 (tokens): The tagger and lemmatiser could treat English multiword wh-forms like "what for", "how come" and "in which" as single interrogative units with a fixed answer type.
  • I5 (learn): For gold evaluation sets, Mova should prefer small human-verified data over silver produced by an untested tool, and keep the two clearly separated.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksQuestion answering · 441 worksLow-resource languages · 267 works

Cite

Cite as Tilahun Abedissa Taffa, Ricardo Usbeck and Yaregal Assabie. 2024. Low Resource Question Answering: An Amharic Benchmarking Dataset. Proceedings of the Fifth Workshop on Resources for African Indigenous Languages @ LREC-COLING 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{taffa2024low,
  title = {{Low Resource Question Answering: An Amharic Benchmarking Dataset}},
  author = {Tilahun Abedissa Taffa, Ricardo Usbeck and Yaregal Assabie},
  booktitle = {Proceedings of the Fifth Workshop on Resources for African Indigenous Languages @ LREC-COLING 2024},
  year = {2024},
  doi = {10.63317/283hsucqvqy9},
  url = {https://doi.org/10.63317/283hsucqvqy9}
}

References

30 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tilahun Abedissa and Mulugeta Libsie. 2019. Amharic Question Answering for Biography, Definition, and Description Questions. In Fisseha Mekuria, Ethiopia Nigussie, and Tesfa Tegegne, editors, Information and Communication Technology for Development for Africa, volume 1026, pages 301–310. Springer International Publishing, Cham. Series Title: Communications in Computer and Information Science. find
  2. Getahun Amare. 2013. Amharic Grammar in Simple Way. Addis Ababa, Addis Ababa University Press. find
  3. B Barla Cambazoglu, Mark Sanderson, Falk Scholer, and Bruce Croft. 2020. A Review of Public Datasets in Question Answering Research. ACM SIGIR Forum, 54(2):23. ↗
  4. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870–1879, Vancouver, Canada. Association for Computational Linguistics. ↗
  5. Danqi Chen and Wen-tau Yih. 2020. Open-Domain Question Answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 34–37, Online. Association for Computational Linguistics.
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics. ↗
  7. Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. 2019. ”a span-extraction dataset for Chinese machine reading comprehension”. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5883–5889, Hong Kong, China. Association for Computational Linguistics.
  8. Martin d’Hoffschmidt, Wacim Belblidia, Quentin Heinrich, Tom Brendlé, and Maxime Vidal. 2020. FQuAD: French question answering dataset. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1193–1208, Online. Association for Computational Linguistics. find
  9. Phong Nguyen-Thuan Do, Nhat Duy Nguyen, Tin Van Huynh, Kiet Van Nguyen, Anh Gia-Tuan Nguyen, and Ngan Luu-Thuy Nguyen. 2021. Sentence Extraction-Based Machine Reading Comprehension for Vietnamese. In Knowledge Science, Engineering and Management: 14th International Conference, KSEM 2021, Tokyo, Japan, August 14–16, 2021, Proceedings, Part II, page 511–523, Berlin, Heidelberg. Springer-Verlag. ↗
  10. Daria Dzendzik, Jennifer Foster, and Carl Vogel. 2021. English Machine Reading Comprehension Datasets: A Survey. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8784–8804, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  11. Abenezer Mengistu Elema. 2022. Developing Amharic Question Answering Model Over Unstructured Data Source Using Deep Learning Approach. In 2022 International Conference on Information and Communication Technology for Development for Africa (ICT4DA), pages 108–113. find
  12. Alexander Fabbri, Patrick Ng, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang. 2020. Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4508–4513, Online. Association for Computational Linguistics. find
  13. Tekabe Legesse Feleke. 2017. The similarity and mutual intelligibility between Amharic and Tigrigna varieties. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), pages 47–54, Valencia, Spain. Association for Computational Linguistics.
  14. Fitsum Gaim, Wonsuk Yang, Hancheol Park, and Jong Park. 2023. Question-answering in a low-resourced language: Benchmark dataset and models for Tigrinya. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11857–11870, Toronto, Canada. Association for Computational Linguistics.
  15. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2.
  16. Md Tahmid Rahman Laskar, Jimmy Xiangji Huang, and Enamul Hoque. 2020. Contextualized embeddings based transformer encoder for sentence similarity modeling in answer selection task. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5505–5514, Marseille, France. European Language Resources Association. find
  17. Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020a. MLQA: Evaluating Cross-lingual Extractive Question Answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315–7330, Online. Association for Computational Linguistics.
  18. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020b. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA. Curran Associates Inc.
  19. Timo Möller, Julian Risch, and Malte Pietsch. 2021. GermanQuAD and GermanDPR: Improving non-English question answering and passage retrieval. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 42–50, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  20. Hussein Mozannar, Elie Maamary, Karl El Hajal, and Hazem Hajj. 2019. Neural Arabic question answering. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 108–118, Florence, Italy. Association for Computational Linguistics.
  21. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1). ↗
  22. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv:1606.05250 [cs]. ArXiv: 1606.05250. ↗
  23. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How Much Knowledge Can You Pack Into the Parameters of a Language Model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online. Association for Computational Linguistics. ↗
  24. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4):333–389. ↗
  25. Anna Rogers, Matt Gardner, and Isabelle Augenstein. 2023. QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension. ACM Comput. Surv., 55(10). ↗
  26. Fandy Setyo Utomo, Nanna Suryana, and Mohd Sanusi Aami. 2017. Question Answering System: A Review on Question Analysis, Document Processing, and Answer Extraction Techniques. Journal of Theoretical & Applied Information Technology, 95(14). find
  27. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120–130, Online. Association for Computational Linguistics.
  28. Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. End-to-end open-domain question answering with BERTserini. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 72–77, Minneapolis, Minnesota. Association for Computational Linguistics.
  29. Baye Yimam. 2009. Amharic Grammar. Addis Ababa, Addis Ababa University Press. find
  30. Seid Muhie Yimam and Mulugeta Libsie. 2009. TETEYEQ: Amharic question answering for factoid questions. IE-IR-LRL, 3(4):17. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/taffa-2024-low-resource-question-answering-amharic/