Elamur Research⌕
☰
home / papers / 2023 / Question-Answering in a Low-resourced Language: Benchmark Dataset and Models for Tigrinya

Question-Answering in a Low-resourced Language: Benchmark Dataset and Models for Tigrinya

Paper · 2023 · Fitsum Gaim, Wonsuk Yang, Hancheol Park et al.

Summary

Tigrinya is a Semitic language of Eritrea and northern Ethiopia with more than 10 million speakers. It had no public dataset for extractive question answering. The authors built TiQuAD, a SQuAD-style benchmark. Native speakers wrote questions about paragraphs drawn from the professionally edited Haddas Ertra newspaper and marked the answer spans. The development and test questions received extra reference answers. The released data has 10.6K question-answer pairs (6.5K distinct questions) over 572 paragraphs from 290 articles.

Four models were fine-tuned: TiRoBERTa and TiELECTRA (monolingual) and XLM-R and AfriBERTa (multilingual). The authors compared four training setups:

  • native data only
  • zero-shot transfer from English SQuAD1.1
  • a machine-translated Tigrinya SQuAD
  • mixtures of all three sources

The strongest system was XLM-R trained jointly on English, native and translated data. It reached 76.58 F1 and 68.06 EM, while human performance was estimated at 92.1 F1. The gap leaves room for future work. The paper also shows that a modest native resource, built on a small budget, is far more useful than English-only transfer.

Key points

  • TiQuAD splits: the data is divided by article. Train has 205 articles, 408 paragraphs and 4,452 questions; dev has 43 articles, 76 paragraphs and 934 questions; test has 42 articles, 96 paragraphs and 1,122 questions. Annotators earned 150 ERN (about 10 USD) per hour.
  • Question types: "Which" questions make up 31.37% and "How many/much" questions 26.23%. "Why" and "How" questions are each only about 1%.
  • Zero-shot transfer: AfriBERTa and XLM-R trained only on English SQuAD1.1 score 32–34% F1 on the TiQuAD test set. Models trained on the small native set do far better.
  • Multilingual best: combining English, native and translated data gives XLM-R BASE a gain of over three F1 points, reaching 68.06 EM and 76.58 F1. Humans score 81.3 EM and 92.1 F1.
  • Data size and limitations: learning curves built in 10% data steps show no saturation. Known limitations are lexical overlap between questions and passages, few information-seeking questions, and no unanswerable questions.

Interesting ideas

  • I1. Splitting the data by source article rather than by question keeps near-duplicate contexts out of both train and test, which gives a fairer estimate of generalisation.
  • I2. Collecting several extra reference answers for evaluation questions, and measuring human agreement as a ceiling, makes scores interpretable and tolerant of reasonable span variation.
  • I3. Retraining on data subsets grown in 10% steps shows whether more annotation would still help before money is spent on it.
  • I4. Profiling questions by their wh-word showed that counting and selection questions dominate a news-based benchmark, while causal questions are rare.
  • I5. The authors admit that annotators who write questions after reading the passage tend to reuse its words, so lexical overlap can inflate scores.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Split fables and school texts by document when building evaluation sets, so a reader is never tested on a story it has already seen in training.
  • I2 (answer): Store multiple acceptable answers for each fable or test question, and report Mova's score against a human-agreement ceiling instead of a single gold string.
  • I3 (learn): Plot parser and reader accuracy against the amount of silver data at each step, to decide whether to add data or to fix rules.
  • I4 (world): A large share of "how many" questions argues for strong entity counting and quantity tracking in the text world. "Why" coverage needs separately sourced tests, since it rarely appears naturally.
  • I5 (answer): Add negative-control questions paraphrased away from the passage wording, to verify that answers come from the meaning graph rather than surface word matching.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Korea Advanced Institute of Science and TechnologyDaejeon · South Korea51 works · 3,167 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 4Research subjects and languages the work is about.Semitic languages · 2,908 worksEvaluation & benchmarks · 1,220 worksQuestion answering · 441 worksLow-resource languages · 267 works

Cite

Cite as Fitsum Gaim, Wonsuk Yang, Hancheol Park and Jong Park. 2023. Question-Answering in a Low-resourced Language: Benchmark Dataset and Models for Tigrinya. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.661

@inproceedings{gaim2023question,
  title = {{Question-Answering in a Low-resourced Language: Benchmark Dataset and Models for Tigrinya}},
  author = {Fitsum Gaim, Wonsuk Yang, Hancheol Park and Jong Park},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.661},
  url = {https://doi.org/10.18653/v1/2023.acl-long.661}
}

References

42 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics.
  2. Akari Asai, Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2018. Multilingual extractive reading comprehension by runtime machine translation. CoRR, abs/1809.03275. ↗
  3. Casimiro Pio Carrino, Marta Ruiz Costa-jussà, and José A. R. Fonollosa. 2020. Automatic spanish translation of squad dataset for multi-lingual question answering. In LREC. find
  4. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470. ↗
  5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  6. Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. 2019. A span-extraction dataset for Chinese machine reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5883–5889, Hong Kong, China. Association for Computational Linguistics.
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  8. Martin d’Hoffschmidt, Wacim Belblidia, Quentin Heinrich, Tom Brendlé, and Maxime Vidal. 2020. FQuAD: French question answering dataset. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1193–1208, Online. Association for Computational Linguistics. find
  9. Pavel Efimov, Leonid Boytsov, and Pavel Braslavski. 2020. SberQuAD–Russian reading comprehension dataset: Description and analysis. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 3–15. Springer. find
  10. Fitsum Gaim. 2017. Applying morphological segmentation to machine translation of low-resourced and morphologically complex languages: The case of tigrinya. Master’s thesis, School of Computing, Korea Advanced Institute of Science and Technology (KAIST), July. find
  11. Fitsum Gaim, Wonsuk Yang, and Jong C. Park. 2021. Monolingual pre-trained language models for tigrinya. In 5th Widening NLP (WiNLP2021) workshop, co-located with the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  12. Fitsum Gaim, Wonsuk Yang, and Jong C. Park. 2022. Geezswitch: Language identification in typologically related low-resourced east african languages. In Proceedings of the 13th Language Resources and Evaluation Conference.
  13. Michael Gasser. 2011. Hornmorpho: a system for morphological processing of amharic, oromo, and tigrinya. In Conference on Human Language Technology for Development, Alexandria, Egypt. find
  14. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  15. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4411–4421. PMLR. find
  16. Divyansh Kaushik and Zachary C. Lipton. 2018. How much reading does reading comprehension require? a critical investigation of popular benchmarks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5010– 5015, Brussels, Belgium. Association for Computational Linguistics.
  17. Arefeh Kazemi, Jamshid Mozafari, and Mohammad Ali Nematbakhsh. 2022. Persianquad: The native question answering dataset for the persian language. IEEE Access, 10:26045–26057. ↗
  18. Omri Keren and Omer Levy. 2021. ParaShoot: A Hebrew question answering dataset. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 106–112, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  19. Lidia Kidane, Sachin Kumar, and Yulia Tsvetkov. 2021. An exploration of data augmentation techniques for improving english to tigrinya translation. ArXiv, abs/2103.16789. ↗
  20. Omer Osman and Yoshiki Mikami. 2012. Stemming tigrinya words for information retrieval. In Proceedings of COLING 2012: Demonstration Papers, pages 345–352.
  21. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466. ↗
  22. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789. Association for Computational Linguistics.
  23. Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315– 7330, Online. Association for Computational Linguistics. ↗
  24. Seungyoung Lim, Myungji Kim, and Jooyoul Lee. 2019. Korquad1.0: Korean qa dataset for machine reading comprehension. arXiv. ↗
  25. Shayne Longpre, Yi Lu, and Joachim Daiber. 2021. MKQA: A linguistically diverse benchmark for multilingual open domain question answering. Transactions of the Association for Computational Linguistics, 9:1389–1406. ↗
  26. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR). find
  27. Timo Möller, Julian Risch, and Malte Pietsch. 2021. GermanQuAD and GermanDPR: Improving non-English question answering and passage retrieval. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 42–50, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  28. Hussein Mozannar, Elie Maamary, Karl El Hajal, and Hazem Hajj. 2019. Neural Arabic question answering. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 108–118, Florence, Italy. Association for Computational Linguistics.
  29. Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin. 2021. Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 116–126, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  30. Alp Öktem, Mirko Plitt, and Grace Tang. 2020. Tigrinya neural machine translation with transfer learning for humanitarian response. arXiv preprint arXiv:2003.11523. ↗
  31. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. ↗
  32. Anna Rogers, Matt Gardner, and Isabelle Augenstein. 2022. Qa dataset explosion: A taxonomy of nlp resources for question answering and reading comprehension. ACM Comput. Surv. Just Accepted. ↗
  33. Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021. XTREME-R: Towards more challenging and nuanced multilingual evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10215–10245, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  34. Wissam Siblini, Charlotte Pasqual, Axel Lavielle, and Cyril Cauchois. 2019. Multilingual question answering from formatted text applied to conversational agents. ArXiv, abs/1910.04659. ↗
  35. Yemane Tedla and Kazuhide Yamamoto. 2016. The effect of shallow segmentation on english-tigrinya statistical machine translation. In 2016 International Conference on Asian Language Processing (IALP), pages 79–82. IEEE. find
  36. Yemane Tedla and Kazuhide Yamamoto. 2018. Morphological segmentation with lstm neural networks for tigrinya. In Intenational Journal on Natural Language Computing (JNLC), volume 7. ↗
  37. Yemane Tedla, Kazuhide Yamamoto, and A. Maras-inghe. 2016. Tigrinya part-of-speech tagging with morphological patterns and the new nagaoka tigrinya corpus. International Journal of Computer Applications, 146:33–41. ↗
  38. Abrhalei Tela, Abraham Woubie, and Ville Hauta-maki. 2020. Transferring monolingual model to low-resource language: The case of tigrinya.
  39. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗
  40. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics. ↗
  41. Hailemariam Mehari Yohannes and Toshiyuki Amagasa. 2022. Named-entity recognition for a low-resource language using pre-trained language model. In Proceedings of the 37th SIGAPP Symposium on Applied Computing, SAC ’22, page 837–844, New York, NY, USA. Association for Computing Machinery. find
  42. ACL 2023 Responsible NLP Checklist find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gaim-2023-question-answering-low-resourced-language/