Elamur Research⌕
☰
home / papers / 2023 / AmQA: Amharic Question Answering Dataset

AmQA: Amharic Question Answering Dataset

Summary

Amharic, Ethiopia's official language and a widely spoken Semitic language, had no public benchmark for question answering. This work builds the first one, AmQA. Clean, sufficiently long articles were taken from the Amharic Wikipedia dump and split into chunks along their sub-topics. Native-speaker crowd workers with university degrees then wrote questions whose answers are contiguous spans of the text. The authors reviewed every pair, rewrote vague questions, dropped unsuitable ones, and exported the result through the Haystack annotation tool in SQuAD-style JSON.

Baselines use XLM-R models fine-tuned on SQuAD 2.0. They were tested in two ways: reading comprehension with the gold context, and a retriever-reader pipeline that pairs BM25 with an XLM-R Large reader. XLM-R Large reached about 72 F1 when given the context, roughly matching its average on MLQA. The pipeline scored lower, but a light answer normalizer narrowed much of that gap. AmQA thus gives low-resource and multilingual QA a testbed for both extractive and open-domain evaluation.

Key points

  • The dataset has 2628 question-answer pairs over 378 Amharic Wikipedia articles, released at github.com/semantic-systems/amharic-qa.
  • Average lengths are 172.07 words per context, 9.22 per question and 2.66 per answer, so answers are very short spans.
  • In a sample of 300 questions, expected answer types were:
    • Location 18.72%, Number 18.72%, Time 18.06%
    • Person 14.38%, Description 13.71%, Other 10.7%
    • List 3.01%, Organization 2.67%
  • Reading-comprehension results:
    • XLM-R Base: EM 47.49 / F1 64.69
    • XLM-R Large: EM 50.76 / F1 71.74, against an MLQA reference average of 70.7 F1
  • Retriever-reader results (BM25 + deepset xlm-roberta-large-squad2 via Farm Haystack): EM 42.4 / F1 64.3, rising to EM 50.3 / F1 69.58 with the pre-processor.

Interesting ideas

  • I1. The authors dropped whole source categories (proverbs, recipes) because their natural questions need reasoning or step lists rather than spans, which keeps the task format consistent. — For Mova (answer): Our fable and tale QA sets could tag each question by whether the answer is a span, a reasoning step or a list, and score each group separately.
  • I2. Questions that lean on the asker's own context, such as "our country", or that leave subject or object unstated, were rewritten to be self-contained so they have exactly one correct reading. — For Mova (frames): Unanchored possessives and missing arguments could be flagged during frame building, so the system asks for clarification or resolves them against the text before answering.
  • I3. Many model answers contained the gold answer but were wrapped in affixes, punctuation or stray spaces, and a small normalizer recovered a large part of the lost score. — For Mova (tokens): In English we could compare answers by lemma and strip determiners and punctuation, so evaluation measures understanding rather than surface form.
  • I4. The expected answer type was derived from the wh-word plus the shape of the answer phrase, then estimated from a random sample instead of labelling everything. — For Mova (world): A wh-word-to-entity-type map (who→person, where→location, how many→measure) would narrow which world entities can be candidate answers.
  • I5. Running the same reader with gold contexts and with retrieval separates retrieval errors from reading errors, and comparing against MLQA shows whether the reader is reasonable. — For Mova (learn): Our evaluations could include a gold-context run next to the full pipeline, so each drop in score is blamed on the right stage.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

9 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 6 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.03290 · arXiv 2303.03290

@misc{abedissa2023amqa,
  title = {{AmQA: Amharic Question Answering Dataset}},
  author = {Abedissa, Tilahun and Usbeck, Ricardo and Assabie, Yaregal},
  year = {2023},
  doi = {10.48550/arxiv.2303.03290},
  eprint = {2303.03290},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.03290}
}

References

22 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abate, S. T., Melese, M., Tachbelie, M. Y., Meshesha, M., Atinafu, S., Mulugeta, W., Assabie, Y., Abera, H., Ephrem, B., Abebe, T., Tsegaye, W., Lemma, A., Andargie, T., & Shifaw, S. (2018). Parallel Corpora for bi-lingual English-Ethiopian Languages Statistical Machine Translation. Proceedings of the 27th International Conference on Computational Linguistics, 3102–3111. https://aclanthology.org/C18-1262 d’Hoffschmidt, M., Belblidia, W., Heinrich, Q., Brendlé, T., & Vidal, M. (2020). FQuAD: French Question Answering Dataset. Findings of the Association for Computational Linguistics: EMNLP 2020, 1193–1208. https://doi.org/10.18653/v1/2020.findings-emnlp.107 ↗
  2. Abedissa, T., & Libsie, M. (2019). Amharic Question Answering for Biography, Definition, and Description Questions. In F. Mekuria, E. Nigussie, & T. Tegegne (Eds.), Information and Communication Technology for Development for Africa (Vol. 1026, pp. 301–310). Springer International Publishing. https://doi.org/10.1007/978-3-030-26630-1_26 ↗
  3. Do, P. N.-T., Nguyen, N. D., Van Huynh, T., Van Nguyen, K., Nguyen, A. G.-T., & Nguyen, N. L.-T. (2021). Sentence Extraction-Based Machine Reading Comprehension for Vietnamese. ArXiv:2105.09043 [Cs]. http://arxiv.org/abs/2105.09043 ↗
  4. Dzendzik, D., Foster, J., & Vogel, C. (2021). English Machine Reading Comprehension Datasets: A Survey. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 8784–8804. https://aclanthology.org/2021.emnlp-main.693 (arXiv:2104.12741). https://doi.org/10.48550/arXiv.2104.12741 arXiv. ↗ our card
  5. Mozannar, H., Maamary, E., El Hajal, K., & Hajj, H. (2019). Neural Arabic Question Answering. Proceedings of the Fourth Arabic Natural Language Processing Workshop, 108–118. https://doi.org/10.18653/v1/W19-4612 ↗
  6. Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., & Auli, M. (2019). ELI5: Long Form Question Answering. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3558–3567. https://doi.org/10.18653/v1/P19-1346 ↗ our card
  7. Rajpurkar, P., Jia, R., & Liang, P. (2018). Know What You Don’t Know: Unanswerable Questions for SQuAD. ArXiv:1806.03822 [Cs]. http://arxiv.org/abs/1806.03822 ↗ our card
  8. Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. ArXiv:1606.05250 [Cs]. http://arxiv.org/abs/1606.05250 ↗ our card
  9. Gezmu, A. M., Seyoum, B. E., Gasser, M., & Nürnberger, A. (2018). Contemporary Amharic Corpus: Automatically Morpho-Syntactically Tagged Amharic Corpus. Proceedings of the First Workshop on Linguistic Resources for Natural Language Processing, 65–70. https://aclanthology.org/W18-3809 ↗
  10. Rogers, A., Gardner, M., & Augenstein, I. (2021). QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension. ArXiv:2107.12708 [Cs]. http://arxiv.org/abs/2107.12708 ↗
  11. Gupta, D., Kumari, S., Ekbal, A., & Bhattacharyya, P. (2018). MMQA: A Multi-domain Multi-lingual Question-Answering Framework for English and Hindi. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). LREC 2018, Miyazaki, Japan. https://aclanthology.org/L18-1440 ↗
  12. Usbeck, R., Röder, M., Hoffmann, M., Conrads, F., Huthmann, J., Ngonga-Ngomo, A.-C., Demmler, C., & Unger, C. (2016). Benchmarking Question Answering Systems. 11. find
  13. Yeshambel, T., Mothe, J., & Assabie, Y. (2020). 2AIRTC: The Amharic Adhoc Information Retrieval Test Collection. In A. Arampatzis, E. Kanoulas, T. Tsikrika, S. Vrochidis, H. Joho, C. Lioma, C. Eickhoff, A. Névéol, L. Cappellato, & N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction (pp. 55–66). Springer International Publishing. https://doi.org/10.1007/978-3-030-58219-7_5 ↗
  14. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., & Petrov, S. (2019). Natural Questions: A Benchmark for Question Answering Research. Transactions of the Association for Computational Linguistics, 7, 452–466. https://doi.org/10.1162/tacl_a_00276 ↗ our card
  15. Yimam, S. M., Alemayehu, H. M., Ayele, A., & Biemann, C. (2020). Exploring Amharic Sentiment Analysis from Social Media Texts: Building Annotation Tools and Classification Models. Proceedings of the 28th International Conference on Computational Linguistics, 1048–1060. https://doi.org/10.18653/v1/2020.coling-main.91 ↗
  16. Lewis, P., Oguz, B., Rinott, R., Riedel, S., & Schwenk, H. (2020). MLQA: Evaluating Cross-lingual Extractive Question Answering. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 7315–7330. https://doi.org/10.18653/v1/2020.acl-main.653 ↗ our card
  17. Liu, J., Lin, Y., Liu, Z., & Sun, M. (2019). XQA: A Cross-lingual Open-domain Question Answering Dataset. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2358–2368. https://doi.org/10.18653/v1/P19-1227 ↗
  18. Yimam, S. M., & Libsie, M. (2009). TETEYEQ (ተጠየቅ): AMHARIC QUESTION ANSWERING SYSTEM. Addis Ababa University. find
  19. ባየ ይማም (2009) ፡፡ የአማርኛ ሰዋስው፣ አዲስ አበባ፣ አዲስ አበባ ዩኒቨርሲቲ ማተሚያ ቤት (Baye Yimam, (2009). Amharic Grammar, Addis Ababa, Addis Ababa University Press) find
  20. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv:1907.11692 [Cs]. http://arxiv.org/abs/1907.11692 ↗
  21. ጌታሁን አማረ (2013) ፡፡ የአማርኛ ሰዋስው በቀላል አቀራረብ፣ አዲስ አበባ፣ አዲስ አበባ ዩኒቨርሲቲ ማተሚያ ቤት (Getahun Amare, (20013). Amharic Grammar with Simple Presentation, Addis Ababa, Addis Ababa University Press) find
  22. Möller, T., Risch, J., & Pietsch, M. (2021). GermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage Retrieval find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abedissa-2023-amqa-amharic-question-answering-dataset/