Elamur Research⌕
☰
home / papers / 2017 / A Preliminary Study for Building an Arabic Corpus of Pair Questions-Texts from the Web: AQA-Webcorp

A Preliminary Study for Building an Arabic Corpus of Pair Questions-Texts from the Web: AQA-Webcorp

Summary

The paper addresses the shortage of Arabic resources for question answering, in particular collections that link a natural-language question to texts likely to contain its answer. The authors propose a method for assembling such a resource, called AQA-Webcorp, directly from the Web. Each factual question is sent to Google as a query. A Java program collects the returned list of HTML pages, and a cleaning step turns them into plain text. The output is a database of texts plus a corpus of question–text pairs, intended as an experimental base for an Arabic QA system that analyses the texts to find an answer. The paper also surveys earlier work on building corpora from the Web, for Arabic and other languages, and reports preliminary results of the collection procedure. Its contribution is a simple, reproducible pipeline for gathering question-specific evidence in a language where ready-made QA corpora are scarce.

Key points

  • AQA-Webcorp consists of factual Arabic questions, each paired with Web texts retrieved through Google that may contain the answer.
  • The pipeline has three steps: query the search engine, extract the list of HTML pages, then clean them into a text database (html → txt).
  • The survey covers several earlier resources:
    • BootCaT, which builds corpora iteratively from Google queries and term extraction.
    • The tool of Meftouh et al., which gathers Arabic sites to train language models.
    • The Web Arabic corpus of Ghoul, with 207,356 sentences and 7,653,881 words across culture, economy, religion and sports, annotated with TreeTagger.
    • arTenTen, with 5.8 billion words, cleaned with JusText and Onion and tagged with MADA 3.2.
  • Earlier Arabic QA corpora were small. For example, Trigui et al. used 50 organization-definition questions and about 2000 retrieved snippets.
  • According to the cited June 2015 statistics, Arabic ranked fourth among Internet user languages.

Interesting ideas

  • I1. Using the question itself as the retrieval query yields a corpus organised around questions, where each text is candidate evidence rather than a random document. — For Mova (answer): QA test sets on fables and school texts could be built the same way, pairing each question with the passages where the answer should be found, so failures can be traced to reading rather than to retrieval.
  • I2. Raw web pages are kept apart from a cleaned text database, so cleaning is a separate, inspectable stage between harvesting and use. — For Mova (learn): silver-data pipelines could store the raw source and its normalised text side by side, so any parser disagreement can be checked against what was actually extracted.
  • I3. Following Sinclair and Rastier, the paper treats a corpus as a set of texts chosen by explicit external criteria, not a "bag of words". — For Mova (learn): each evaluation corpus could record its selection criteria, which makes negative controls and domain comparisons meaningful.
  • I4. The survey cites Elghamry's harvesting of hypernym–hyponym pairs from the Web with a lexico-syntactic "certain x such as y1…yn" pattern. — For Mova (world): such patterns, read over UD trees rather than raw strings, could propose candidate ontology links to be checked against WordNet before entering the knowledge graph.
  • I5. The survey also describes BootCaT's loop, in which terms extracted from retrieved texts become seeds for the next round of queries. — For Mova (learn): a similar loop could grow domain-specific training text, with the outputs of the witness parsers deciding which harvested sentences are clean enough to keep.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3991/ijes.v4i2.5345 · arXiv 1709.09404

@misc{bakari2017preliminary,
  title = {{A Preliminary Study for Building an Arabic Corpus of Pair Questions-Texts from the Web: AQA-Webcorp}},
  author = {Bakari, Wided and Bellot, Patrice and LSIS), Mahmoud Neji},
  year = {2017},
  doi = {10.3991/ijes.v4i2.5345},
  eprint = {1709.09404},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.3991/ijes.v4i2.5345}
}

References

39 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sinclair, J. (2005). Corpus and text - basic principles. In M. Wynne (Ed.), Developing linguistic corpora: A guide to good practice (pp. 1–16). Oxford, UK: Oxbow Books. find
  2. Rastier F. (2005). « Enjeux épistémologiques de la linguistique de corpus », in : Williams C. G. (dir), La linguistique de corpus, Rennes : P.U.R find
  3. Meftouh K., Smaïli K. et Laskri M.T. (2007). Constitution d’un corpus de la langue arabe à partir du Web. CITALA ’07. Colloque international du traitement automatique de la langue arabe. Iera, Rabat, Morocco, 17-18 juin 2007. find
  4. P. Resnik, Parallel strands: A preliminary investigation into mining the web for bilingual text, in conference of the association for machine translation in the Americas, 1998. find
  5. GATTO, Maristella. The ‘body’and the ‘web’: The web as corpus ten years on.ICAME JOURNAL, 2011, vol. 35, p. 35-58. find
  6. Kilgarriff, A., & Grefenstette, G. (2001, March). Web as corpus. In Proceedings of Corpus Linguistics 2001 (pp. 342-344). Corpus Linguistics. Readings in a Widening Discipline. find
  7. F. Issac, T. Hamon, L. Bouchard, L. Emirkanian, C. Fouqueré, extraction informatique de données sur le web : une expérience, in Multimédia, Internet et francophonie : à la recherche d’un dialogue, Vancouver, Canada, mars 2001. find
  8. Atwell, E., Al-Sulaiti, L., Al-Osaimi, S. & Abu Shawar, B. (2004). A review of Arabic corpus analysis tools. In B. Bel & I. Marlien (Eds.), Proceedings of TALN04: XI Conference sur le Traitement Automatique des Langues Naturelles (volume 2, pp. 229–234). ATALA. find
  9. Mohamed Abdelmageed Mansour, The Absence of Arabic Corpus Linguistics: A Call for Creating an Arabic National Corpus, International Journal of Humanities and Social Science, Vol. 3 No. 12 [Special Issue – June 2013] find
  10. R. Ghani, R. Jones, D. Mladenic, Mining the web to create minority language corpora, CIKM 2001, 279-286. find
  11. M. Baroni, S. Bernardini, Bootcat: Bootstrapping corpora and terms from the web, proceeding of LREC 2004, 1313-1316. find our card
  12. M. Ueyama, M. Baroni, Automated construction and evaluation of Japanese web-based reference corpora, proceedings of corpus linguistics, 2005. find
  13. S. Sharoff, Creating general–purpose corpora using automated search engine, in Wacky! Working papers on the web as corpus, Bologna: GEDIT 2006, 63-98. find
  14. Elghamry, K. (2008). Using the web in building a corpus-based hypernymy-hyponymy Lexicon with hierarchical structure for Arabic. Faculty of Computers and Information, 157-165. find
  15. Mohamed Hédi Maâloul,Iskander Keskes, Lamia Hadrich Belguith and Philippe Blache, “Automatic summarization of arabic texts based on RST technique”,12th International Conference on Enterprise Information Systems (ICEIS’2010), 8 au 12 juin 2010, Portugal. find
  16. Ghoul, D. Web Arabic corpus: Construction d’un large corpus arabe annoté morpho-syntaxiquement à partir du Web. In ACTES DU COLLOQUE , 2014 (p. 12). find
  17. Arts, T., Belinkov, Y., Habash, N., Kilgarriff, A., & Suchomel, V. (2014). arTenTen: Arabic Corpus and Word Sketches. Journal of King Saud University-Computer and Information Sciences, 26(4), 357-371. http://dx.doi.org/10.1016/j.jksuci.2014.06.009 ↗
  18. Jakubí!ek, M., Kilgarriff, A., Ková", V., Rychl#, P., and Suchomel, V. 2013.The TenTen Corpus Family.International Conference on Corpus Linguistics, Lancaster. find
  19. Pomikalek, J. 2011. Removing Boilerplate and Duplicate Content from Web Corpora. PhD thesis, Masaryk University, Brno. find
  20. Habash, N. and Rambow, O. 2005. Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop. In Proc. of the Association for Computational Linguistics (ACL’05), Ann Arbor, Michigan. find our card
  21. Habash, N., Rambow, O. and Roth, R. 2009. MADA+TOKAN: A Toolkit for Arabic Tokenization, Diacritization, Morphological Disambiguation, POS Tagging, Stemming and Lemmatization. In Proc. of the International Conference on Arabic Language Resources and Tools, Cairo, Egypt. find
  22. Volk, M. (2002). Using the web as corpus for linguistic research. Catcher of the Meaning. Pajusalu, R., Hennoste, T.(Eds.). Dept. of General Linguistics, 3. find
  23. Keller, F. and M. Lapata. 2003. Using the Web to Obtain Frequencies for Unseen Bigrams. Computational Linguistics 29:3, 459-484. http://dx.doi.org/10.1162/089120103322711604 ↗
  24. Villaseñor-Pineda, L., Montes-y-Gómez, M., Pérez-Coutiño, M. A., & Vaufreydaz, D. (2003). A corpus balancing method for language model construction. In Computational Linguistics and Intelligent Text Processing (pp. 393-401). Springer Berlin Heidelberg. http://dx.doi.org/10.1007/3-540-36456-0_40 PAPER A P RELIMINARY S TUDY FOR B UILDING AN A RABIC C ORPUS OF P AIR Q UESTIONS -T EXTS FROM THE W EB : AQA-W EBCORP ↗
  25. A. Kilgarrif, G. Grefenstette, Introduction to the special issue on the Web as corpus, in Association for computational linguistics 29(3): 333-347, 2003. find our card
  26. Elghamry, K., Al-Sabbagh, R., El-Zeiny, N. (2007). "Arabic Anaphora Resolution Using Web as Corpus", Proceedings of the seventh conference on language engineering, Cairo, Egypt. find
  27. Rahma Sellami, Fatiha Sadat, and Lamia Hadrich Belguith. 2013. Traduction automatique statistique à partir de corpus comparables : application aux couples de langues arabe-français. In CORIA, pages 431–440. find
  28. Greenwood, Mark, Ian Roberts, and Robert Gaizauskas. 2002. University of she_eld trec 2002 q & a system. In E. M. Voorhees and Lori P. Buckland, editors, The Eleventh Text REtrieval Conference (TREC-11), Washington. U.S. Government Printing Office. NIST Special Publication 500{XXX. find
  29. Dumais, Susan, Michele Banko, Eric Brill, Jimmy Lin, and Andrew Ng. 2002. Web question answering: is more always better? In Proc. 25th ACM SIGIR, pages 291{298, Tampere, Finland. http://dx.doi.org/10.1145/564376.564428 ↗
  30. Bakari, W., Bellot, P., and Neji, M. (2015). Literature Review of Arabic Question-Answering: Modeling, Generation, Experimentation and Performance Analysis. In Flexible Query Answering Systems 2015 (pp. 321-334). Springer International Publishing. find
  31. Bakari, W., Trigui, O., and Neji, M. (2014, December). Logic-based approach for improving Arabic question answering. In Computational Intelligence and Computing Research (ICCIC), 2014 IEEE International Conference on (pp. 1-6). IEEE. http://dx.doi.org/10.1109/iccic.2014.7238319 ↗
  32. Zheng, Zhiping. 2002. AnswerBus question answering system. In E. M. Voorhees and Lori P. Buckland, editors, Proceeding of HLT Human Language Technology Conference (HLT 2002), San Diego, CA, March 24{27. http://dx.doi.org/10.3115/1289189.1289238 ↗
  33. O. Trigui, L.H Belguith and P. Rosso, “DefArabicQA: Arabic Definition Question Answering System”, In Workshop on Language Resources and Human Language Technologies for Semitic Languages, 7th LREC. Valletta, Malta. 2010. find
  34. Volk, Martin. 2001. Exploiting the www as a corpus to resolve pp attachment ambiguities. In Proc. Corpus Linguistics 2001, Lancaster, UK. find
  35. McEnery, A. M. (2003). Corpus linguistics.In R. Mitkov (Ed.), The Oxford handbook of computational linguistics. (pp. 448-463). Oxford: Oxford University Press. find
  36. Hammo B., Abu-Salem H., Lytinen S., Evens M., "QARAB: A Question Answering System to Support the Arabic Language", Workshop on Computational Approaches to Semitic Languages. ACL 2002, Philadelphia, PA, July. p 55-65, 2002 http://dx.doi.org/10.3115/1118637.1118644 ↗
  37. Ezzeldin A. M. and Shaheen M. (2012). “A survey of Arabic question answering: challenges, tasks, approaches, tools, and future trends”, the 13th International Arab Conference on Information Technology ACIT’2012. Dec.10-13. ISSN 1812-0857. find
  38. Abdelnasser, H., Mohamed, R., Ragab, M., Mohamed, A., Farouk, B., El-Makky, N., & Torki, M. (2014). Al-Bayan: An Arabic Question Answering System for the Holy Quran. ANLP 2014, 57. http://dx.doi.org/10.3115/v1/w14-3607 ↗
  39. Bekhti S., Rehman A., AL-Harbi M. and Saba T. (2011).“AQUASYS: an arabic question-answering system based on extensive question analysis and answer relevance scoring”, In International Journal of Academic Research; Jul2011, Vol. 3 Issue 4, p45. A UTHORS Wided BAKARI received her Baccalaureate in 2004 in experimental science from Secondary school SIDI AICH / Gafsa/Tunisia, and her Technician degree in computer science from sciences in 2007 from Higher Institute of Technological Studies of Gafsa (ISETG)/Tunisia. In 2009, she got her second Bachelor degree in computer sciences from Faculty of Sciences of Gafsa/Tunisia, where she also obtained the master degree in Information system and new technologies in 2012 from the Faculty of Economic Sciences and Management of Sfax (FSEGS)/Tunisia. She is currently a member of the Multimedia and InfoRmation Systems and Advanced Computing Laboratory (MIR@CL). She is now preparing her PhD; her research interest includes information retrieval and Arabic Question Answering systems. She has publications in international conferences (Faculty of Economics and Management, 3018, Sfax Tunisia, MIR@CL, Sfax, Tunisia, Wided.bakkari@fsegs.rnu.tn). Patrice BELLOT is a professor at Aix-Marseille University, Head of the project-team DIMAG, Aix-Marseille University - CNRS, UMR 7296 LSIS (F-13397 Marseille, France, Patrice.bellot@gmail.com) Mahmoud Neji is an associate professor at the Faculty of Economics and Management of Sfax, University of Sfax, in Tunisia. He is a member of the Multimedia and InfoRmation Systems and Advanced Computing Laboratory (Faculty of Economics and Management, 3018, Sfax Tunisia, MIR@CL, Mahmoud.neji@gmail.com). Submitted 08 December 2015. Published as resubmitted by the authors 23 Februyra 2016. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bakari-2017-preliminary-study-building-arabic-corpus/