Elamur Research⌕
☰
home / papers / 2025 / AmaSQuAD: A Benchmark for Amharic Extractive Question Answering

AmaSQuAD: A Benchmark for Amharic Extractive Question Answering

Summary

Amharic is a Semitic language with tens of millions of speakers, but it has very little training data for extractive question answering. The largest existing set, AmQA, holds only about 2.6k crowdsourced question–answer pairs. The authors build AmaSQuAD by machine-translating the SQuAD 2.0 train and dev splits into Amharic with Google Translate, called through the Deep Translator library.

The difficult step is re-anchoring each answer. When the answer is translated on its own, its Amharic string often fails to appear verbatim in the translated passage, or it appears more than once. To pick the right span, they score candidates with cosine similarity over embeddings from an Amharic-tuned BERT, combined with Longest Common Subsequence. They also favour candidates that sit close to the answer's original position in the English passage.

Fine-tuning XLM-R on this synthetic data beats the untuned model on the AmaSQuAD dev set, raising EM from 36.55 to 44.41 and F1 from 50.01 to 57.55. On the human-built AmQA set the gains are small. The released dataset and pipeline offer a reusable recipe for porting span-based QA benchmarks to low-resource languages.

Key points

  • AmaSQuAD is a translated version of SQuAD 2.0, including its unanswerable questions, which AmQA lacks. It is published on Hugging Face as nebhailema/AmaSquad.
  • Answer-span alignment combines two signals with a positional preference:
    • cosine similarity from Davlan/bert-base-multilingual-cased-finetuned-amharic embeddings;
    • LCS string matching;
    • a preference for spans near the answer's original location in the English passage.
  • XLM-R on the AmaSQuAD dev set, before vs. after fine-tuning: EM 36.55% → 44.41% (+7.86), F1 50.01% → 57.55% (+7.54).
  • XLM-R on the human-curated AmQA set: EM 52.50% → 52.66% (+0.16), F1 67.80% → 68.80% (+1.00). The authors present this as the first fine-tuned baseline reported against AmQA.
  • Prior comparable pipelines include TAR, which used SQuAD 1.1 → Spanish with EFMARAL alignment and mBERT, and ParSQuAD, which used SQuAD 2.0 → Persian with sentence-level mapping and both manual and automatic variants.

Interesting ideas

  • I1. After translating text and answer separately, recover the answer span with a hybrid matcher: a semantic score (embedding cosine) plus a surface score (LCS), since neither alone handles inflection and paraphrase well. — For Mova (answer): when a reader derives an answer from the world graph, this kind of fuzzy two-signal matching could point the answer back to an exact, explainable span of the source text.
  • I2. Break ties among multiple candidate occurrences with a positional prior: prefer the span whose location matches where the answer sat in the original document. This is a cheap, interpretable disambiguator for projecting annotations across versions of a text. — For Mova (learn): when projecting silver labels from an original text onto a paraphrase or translation, a position-aware choice among duplicate matches would reduce mislabeled spans before witness-parser agreement.
  • I3. Report results on both the synthetic dev set and an independent human-made set. The large in-distribution gain shrinks to about one F1 point on AmQA, which exposes how optimistic synthetic-only evaluation can be. — For Mova (learn): every gain measured on self-generated or LLM-generated data should be paired with a hand-built held-out control, so that improvements specific to the data generator are caught.
  • I4. Keep unanswerable questions when porting a benchmark, so the model learns to abstain rather than always extracting something. — For Mova (answer): fable and school-test QA sets could include questions whose answer is absent from the world of the text, checking that readers say "not stated" instead of guessing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.02047 · arXiv 2502.02047

@misc{hailemariam2025amasquad,
  title = {{AmaSQuAD: A Benchmark for Amharic Extractive Question Answering}},
  author = {Hailemariam, Nebiyou Daniel and Guda, Blessed and Tefferi, Tsegazeab},
  year = {2025},
  doi = {10.48550/arxiv.2502.02047},
  eprint = {2502.02047},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.02047}
}

References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Hirpassa and G. Lehal, “Improving part-of-speech tagging in amharic language using deep neural network,” Heliyon, vol. 9, no. 7, 2023. find
  2. S. J. Basha, D. Veeraiah, B. V. Charan, W. S. J. Yeddu, and D. G. Babu, “Detection and comparative analysis of handwritten words of amharic language to english using cnn-based frameworks,” in 2023 International Conference on Inventive Computation Technologies (ICICT). IEEE, 2023, pp. 422–427. find
  3. F. Zhu, W. Lei, C. Wang, J. Zheng, S. Poria, and T.-S. Chua, “Retrieving and reading: A comprehensive survey on open-domain question answering,” arXiv preprint arXiv:2101.00774, 2021. ↗
  4. T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng, “Ms marco: A human-generated machine reading comprehension dataset,” 2016. find our card
  5. K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom, “Teaching machines to read and comprehend,” Advances in neural information processing systems, vol. 28, 2015. find our card
  6. P. Rajpurkar, R. Jia, and P. Liang, “Know what you don’t know: Unanswerable questions for squad,” arXiv preprint arXiv:1806.03822, 2018. ↗ our card
  7. T. Abedissa, R. Usbeck, and Y. Assabie, “Amqa: amharic question answering dataset,” arXiv preprint arXiv:2303.03290, 2023. ↗ our card
  8. A. Conneau, “Unsupervised cross-lingual representation learning at scale,” arXiv preprint arXiv:1911.02116, 2019. ↗
  9. P. Lewis, B. Oğuz, R. Rinott, S. Riedel, and H. Schwenk, “Mlqa: Evaluating cross-lingual extractive question answering,” arXiv preprint arXiv:1910.07475, 2019. ↗ our card
  10. C. P. Carrino, M. R. Costa-Jussà, and J. A. Fonollosa, “Automatic spanish translation of the squad dataset for multilingual question answering,” arXiv preprint arXiv:1912.05200, 2019. ↗
  11. R. O. stling, “efmaral,” https://github.com/robertostling/efmaral, Oct. 2023, accessed: Dec. 10, 2023. ↗
  12. T. Abedissa and M. Libsie, “Amharic question answering for biography, definition, and description questions,” in Information and Communication Technology for Development for Africa: Second International Conference, ICT4DA 2019, Bahir Dar, Ethiopia, May 28-30, 2019, Revised Selected Papers 2. Springer, 2019, pp. 301–310. find
  13. N. Abadani, J. Mozafari, A. Fatemi, M. A. Nematbakhsh, and A. Kazemi, “Parsquad: machine translated squad dataset for persian question answering,” in 2021 7th International Conference on Web Research (ICWR). IEEE, 2021, pp. 163–168. find
  14. C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “Boolq: Exploring the surprising difficulty of natural yes/no questions,” arXiv preprint arXiv:1905.10044, 2019. ↗ our card
  15. “squad v2 · datasets at hugging face,” https://huggingface.co/datasets/ squad v2, accessed: Dec. 19, 2023. ↗
  16. “Davlan/bert-base-multilingual-cased-finetuned-amharic,” https://huggingface.co/Davlan/bert-base-multilingual-cased-finetuned-amharic, accessed: Dec. 10, 2023. ↗
  17. T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein, Introduction to algorithms. MIT press, 2022. find
  18. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio, Eds. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186. [Online]. Available: https://aclanthology.org/N19-1423 ↗
  19. Y. Liu, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, vol. 364, 2019. ↗
  20. T. Pires, E. Schlinger, and D. Garrette, “How multilingual is multilingual BERT?” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. Màrquez, Eds. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 4996–5001. [Online]. Available: https://aclanthology.org/P19-1493 ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hailemariam-2025-amasquad-benchmark-amharic-extractive-question/