Elamur Research⌕
☰
home / papers / 2023 / ArTrivia: Harvesting Arabic Wikipedia to Build A New Arabic Question Answering Dataset

ArTrivia: Harvesting Arabic Wikipedia to Build A New Arabic Question Answering Dataset

Paper · 2023 · Sultan Alrowili and K Vijay-Shanker

Summary

Arabic extractive QA datasets suffer from three problems: narrow topical coverage (few source articles), skewed question patterns from crowd workers (about a third of Arabic TyDi questions simply ask what or who an entity is), and small size due to crowd-sourcing costs. Machine translation and zero-shot LLMs have not filled the gap for Arabic. The authors build ArTrivia, a dataset of 10,045 question–answer–passage triplets across 18 topics, including underrepresented areas such as classical Arabic poetry. It uses neither crowd workers nor translation. Question–answer pairs are harvested from structured parts of Arabic Wikipedia (tables, WikiData relations, lists, and short entity descriptions turned into questions with ChatGPT). A pipeline then attaches a supporting passage to each pair, and the development set is manually checked. Experiments with AraELECTRA show that ArTrivia is out-of-distribution and harder for models trained on TyDi. Combining both training sets gives near-best results on both dev sets. The paper also argues that answer normalization is an underrated factor in dataset quality.

Key points

  • ArTrivia contains 10,045 questions, 7,982 passages and 7,594 articles, which is about 1.3 questions per article. TyDi QA has 1.7, and ARCD, AQAD and ArabicSQuAD each draw on fewer than 300 articles.
  • The pairs come from four sources:
    • Wiki Tables and WikiData, which together yielded over 12.7K raw pairs using a fixed question template per relation.
    • Wikipedia lists, which gave 591 long-answer pairs.
    • Entity descriptions, which ChatGPT rephrased as questions.
  • Entities are selected by popularity, measured as the number of language editions an article exists in.
  • With AraELECTRA tested on the ArTrivia dev set, TyDi-trained models reach 79.0 EM / 84.9 F1. Training on ArTrivia gives 89.4 / 93.3, and training on both gives 89.5 / 93.0.
  • On the TyDi dev set, ArTrivia-trained models score low (60.7 F1). Most of that gap comes from "who/what is X" questions, which have long answers. Removing them (Dev-Short) narrows the F1 gap to TyDi training to 4.4 (79.9 vs 84.3).

Interesting ideas

  • I1. Each table column pair or knowledge-base relation becomes a reusable question template (e.g., "capital of A?" → B), producing many cheap, typed QA items without annotators.
  • I2. An entity's importance is estimated by how many language editions of Wikipedia cover it, a simple, language-neutral popularity signal.
  • I3. The authors audit the dataset's mix of question types and remove one dominant, easy category (entity definitions) to build a "short" evaluation slice that exposes the real gap between training sets.
  • I4. In error analysis, models often picked the first entity in the passage that matched the question word's type (the first person for "who", the first date for "when"), not the correct one.
  • I5. Normalizing gold answers is treated as a major lever for dataset quality, not a cosmetic step.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Mova could turn facts from its compiled Wikidata/WordNet knowledge into templated test questions to check whether its readers can retrieve and justify typed answers.
  • I2 (world): this count could decide which Wikidata entities get compiled first into the ontology and knowledge graph, so common referents are covered before obscure ones.
  • I3 (learn): evaluation reports could include slices with dominant easy question types removed, so improvements are not driven by trivial patterns.
  • I4 (learn): this suggests a negative control in which a same-type distractor entity is placed before the true answer, testing whether the frame-based reader truly uses roles rather than position.
  • I5 (answer): school-test and fable-QA scoring should normalize gold and predicted answers (articles, number formats, name variants) before judging, so real reasoning errors are not confused with surface mismatches.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of DelawareNewark · United States20 works · 1,056 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksQuestion answering · 441 works

Cite

Cite as Sultan Alrowili and K Vijay-Shanker. 2023. ArTrivia: Harvesting Arabic Wikipedia to Build A New Arabic Question Answering Dataset. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.17

@inproceedings{alrowili2023artrivia,
  title = {{ArTrivia: Harvesting Arabic Wikipedia to Build A New Arabic Question Answering Dataset}},
  author = {Sultan Alrowili and K Vijay-Shanker},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.17},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.17}
}

References

18 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cohen Adam. 2023. thefuzz. https://github.com/ seatgeek/thefuzz. ↗
  2. Sultan Alrowili and Vijay Shanker. 2021. ArabicTransformer: Efficient large Arabic language model with funnel transformer and ELECTRA objective. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1255–1261, Punta Cana, Dominican Republic. Association for Computational Linguistics. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. 2023. Palm 2 technical report. ↗
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. AraELECTRA: Pre-training text discriminators for Arabic language understanding. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 191–195, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  5. Adel Atef, Bassam Mattar, Sandra Sherif, Eman Elrefai, and Marwan Torki. 2020. Aqad: 17,000+ arabic questions for machine comprehension of text. find
  6. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470. ↗
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  8. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30). ↗
  9. Fan Huang, Haewoon Kwak, and Jisun An. 2023. Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech. In Companion Proceedings of the ACM Web Conference 2023, WWW ’23 Companion, page 294–297, New York, NY, USA. Association for Computing Machinery. find
  10. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.
  11. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics. ↗
  12. Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Gptaraeval: A comprehensive evaluation of chatgpt on arabic nlp.
  13. Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning.
  14. Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, page 2356–2362, New York, NY, USA. Association for Computing Machinery. find
  15. Hussein Mozannar, Elie Maamary, Karl El Hajal, and Hazem Hajj. 2019. Neural Arabic question answering. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 108–118, Florence, Italy. Association for Computational Linguistics.
  16. OpenAI. 2023. https://chat.openai.com. ↗
  17. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. ↗
  18. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4):333–389. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alrowili-2023-artrivia-harvesting-arabic-wikipedia-build/