Elamur Research⌕
☰
home / papers / 2019 / Neural Arabic Question Answering

Neural Arabic Question Answering

Paper · 2019 · Hussein Mozannar, Elie Maamary, Karl El Hajal et al.

Summary

Arabic question answering has trailed English, mainly because there were no sizeable span-annotated datasets. Earlier Arabic systems mostly used classical retrieval and were tested on very small question sets. The authors target open-domain factual QA, where every answer must be a text span from Arabic Wikipedia.

They build two resources:

  • ARCD: 1,395 questions written by screened Arabic-speaking crowdworkers about popular Wikipedia articles.
  • Arabic-SQuAD: roughly 48k question–answer items produced by running part of SQuAD through Google's neural MT.

Their pipeline, SOQAL, has three stages. A hierarchical TF-IDF retriever narrows down Wikipedia. A fine-tuned multilingual BERT reader extracts answers from the retrieved paragraphs (QANet is also tested). A ranking step merges retriever and reader confidences.

Trained on translated plus native data, the BERT reader reaches 61.3 F1 on ARCD, and the full open-domain system scores 27.6 F1. The work matters because it releases public benchmarks and code. It also shows that machine-translated training data can largely replace expensive native annotation for a lower-resource language.

Key points

  • ARCD: 1,395 questions on 465 paragraphs from 155 articles. The articles were sampled from the 1000 most viewed Arabic Wikipedia pages of 2018 and annotated via Amazon Mechanical Turk.
  • Arabic-SQuAD: the first 231 SQuAD training articles were translated with the Google Translate NMT API, yielding 48,344 questions on 10,364 paragraphs. The authors note that around half of the answers may be damaged by translation.
  • Retrieval on ARCD: Google Search (k = 10) scores 75.6%, and a bigram TF-IDF retriever with k = 350 comes close at 73.5%. The hierarchical variant (bigram k = 1000 followed by a 4-gram stage) scores 65.3%. A fastText embedding retriever performed poorly.
  • Reader results on ARCD-Test: BERT (BERT-Base multilingual) trained only on Arabic-SQuAD gets 53.0 F1, and trained only on ARCD-Train gets 50.1 F1. Training on both gives 34.2 EM, 61.3 F1 and 90.0 SM. Non-learning baselines barely beat random guessing on EM and F1.
  • SOQAL end-to-end: the top-1 answer scores 12.8 EM, 27.6 F1 and 29.8 SM. Considering the top-5 answers raises F1 to 42.5. All data and code are published on GitHub (husseinmozannar/SOQAL).

Interesting ideas

  • I1. Copy/paste controls in the annotation interface: workers could not paste into question fields but had to paste into answer fields. This pushes them to phrase questions themselves while guaranteeing answers are exact spans, a cheap trick for any span-extraction dataset built by crowdsourcing.
  • I2. Proficiency gate before annotation: workers first had to pass a small reading-comprehension test. This simple filter for annotator language skill transfers to any language where crowd platforms attract many non-native participants.
  • I3. Sentence match (SM) as a diagnostic metric: SM checks only whether the prediction falls in the same sentence as the gold answer. Comparing it with F1 shows whether errors come from locating the right region or from cutting the span boundaries. In the open-domain setting, F1 and SM were nearly equal, which pointed to paragraph selection as the real bottleneck.
  • I4. Translated data first, native data second: training on noisy machine-translated SQuAD worked about as well as training on the small native set alone. Combining the two added about 8 F1 points. This is a practical recipe for low-resource languages, including Ukrainian.
  • I5. Coarse-to-fine lexical retrieval: a cheap bigram TF-IDF pass first retrieves many documents, then a richer 4-gram pass reranks them. This balances feature richness against the number of documents processed, and raw Wikipedia text is used instead of a knowledge base, so it ports easily to new domains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓American University of BeirutBeirut · Lebanon17 works · 1,847 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksInformation Systems · 523 works

🧭 Topics 2Research subjects and languages the work is about.Question answering · 242 worksSemitic languages · 1,484 works

Cite

Cite as Hussein Mozannar, Elie Maamary, Karl El Hajal and Hazem Hajj. 2019. Neural Arabic Question Answering. Proceedings of the Fourth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-4612

@inproceedings{mozannar2019neural,
  title = {{Neural Arabic Question Answering}},
  author = {Hussein Mozannar, Elie Maamary, Karl {El Hajal} and Hazem Hajj},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Workshop},
  year = {2019},
  doi = {10.18653/v1/w19-4612},
  url = {https://doi.org/10.18653/v1/w19-4612}
}

References

31 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Karim Bouzouba Abouenour Lahsen and Paolo Rosso. 2010. An evaluated semantic query expansion and structure-based approach for enhancing arabic question/answering. In International Journal on Information and Communication Technologies 3, no. 3, pages 37–51. find
  2. Mohammed Akour, Sameer Abufardeh, Kenneth Magel, and Qasemm Al-Radaideh. 2011. Qarabpro: A rule based question answering system for reading comprehension tests in arabic. American Journal of Applied Sciences, 8(6):652. ↗
  3. Ahmad Al Sallab, Hazem Hajj, Gilbert Badaro, Ramy Baly, Wassim El Hajj, and Khaled Bashir Shaban. 2015. Deep learning models for sentiment analysis in arabic. In Proceedings of the second workshop on Arabic natural language processing, pages 9–17. Open Domain QA We test our open domain approach SOQAL on ARCD-Test. For our retriever we combine our hierarchical TF-IDF retriever with the Google Cus-Aqil M Azmi and Nouf A Alshenaifi. 2016. Answering arabic why-questions: Baseline vs. rst-based approach. ACM Transactions on Information Systems (TOIS), 35(1):6. https://github.com/google-research/ bert 6 https://github.com/NLPLearn/QANet ↗
  4. Gilbert Badaro, Obeida El Jundi, Alaa Khaddaj, Alaa Maarouf, Raslan Kain, Hazem Hajj, and Wassim El-Hajj. 2018. Ema at semeval-2018 task 1: Emotion mining for arabic. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 236–244. find
  5. Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. 2015. Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6(2):167–195. ↗
  6. Yonatan Belinkov, Alberto Barrón-Cedeño, and Hamdy Mubarak. 2015. Answer selection in arabic community question answering: A feature-rich approach. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 183– 190. find
  7. Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. 2018. Denoising distantly supervised open-domain question answering. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1736–1745.
  8. Abdelouahid Lyhyaoui Benajiba Yassine, Paolo Rosso. 2007. Implementation of the arabiqa question answering system’s components. In Proc. Workshop on Arabic Natural Language Processing, 2nd Information Communication Technologies Int. Symposium, ICTIS-2007, Fez, Morroco, April, pages 3–5. find
  9. Preslav Nakov, Doris Hoogeveen, Lluı́s Màrquez, Alessandro Moschitti, Hamdy Mubarak, Timothy Baldwin, and Karin Verspoor. 2017. Semeval-2017 task 3: Community question answering. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 27–48.
  10. Steven Bird. 2006. Nltk: The natural language toolkit. In COLING ACL 2006, page 69.
  11. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014).
  12. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1870–1879.
  13. Eduard H. Hovy Pamela Forner lvaro Rodrigo Richard FE Sutcliffe Corina Forascu Peas, Anselmo and Caroline Sporleder. 2011. Overview of qa4mre at clef 2011: Question answering for machine reading evaluation. In CLEF (Notebook Papers/Labs/Workshop), pages 1–20. find
  14. Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. 2018. Multi-step retriever-reader interaction for scalable open-domain question answering. ↗
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  16. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. ↗
  17. Walaa Saber Ismail and Masun Nabhan Homsi. 2018. Dawqas: A dataset for arabic why question answering system. In Procedia computer science 142, pages 123–131. find
  18. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 193–203.
  19. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551. ↗
  20. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603. ↗
  21. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651. ↗
  22. Mohamed Shaheen and Ahmed Magdy Ezzeldin. 2014. Arabic question answering: systems, resources, tools, and future trends. Arabian Journal for Science and Engineering, 39(6):4541–4564. ↗
  23. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics. ↗
  24. Omar Trigui, Lamia Hadrich Belguith, and Paolo Rosso. 2010. Defarabicqa: Arabic definition question answering system. In Workshop on Language Resources and Human Language Technologies for Semitic Languages, 7th LREC, Valletta, Malta, pages 40–45. find
  25. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. Newsqa: A machine comprehension dataset. ACL 2017, page 191. ↗
  26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
  27. Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. 2018a. R 3: Reinforced ranker-reader for open-domain question answering. In Thirty-Second AAAI Conference on Artificial Intelligence. find
  28. Zhen Wang, Jiachen Liu, Xinyan Xiao, Yajuan Lyu, and Tian Wu. 2018b. Joint training of candidate extraction and answer selection for reading comprehension. arXiv preprint arXiv:1805.06145. ↗
  29. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. ↗
  30. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013–2018.
  31. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018. Qanet: Combining local convolution with global self-attention for reading comprehension. arXiv preprint arXiv:1804.09541. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mozannar-2019-neural-arabic-question-answering/