Elamur Research⌕
☰
home / papers / 2015 / Answer Selection in Arabic Community Question Answering: A Feature-Rich Approach

Answer Selection in Arabic Community Question Answering: A Feature-Rich Approach

Paper · 2015 · Yonatan Belinkov, Alberto Barrón-Cedeño and Hamdy Mubarak

Summary

The paper tackles answer selection in Arabic community question answering. Given a forum question and its attached comments, each comment must be labelled DIRECT, RELATED or IRRELEVANT. The authors use the SemEval-2015 Task 3 Arabic setup, built on the Fatwa corpus of religious questions answered by scholars. Every question there carries five comments, and only one of them is the true answer. They treat each question–comment pair as one training instance for a linear SVM. The classifier combines four feature families:

  • averaged lemma embeddings for question and comment;
  • a battery of lexical n-gram similarity scores;
  • binned scores from an SVM ranker;
  • the output of their own rule-based ranking system.

The best mix, embeddings plus rule-based ranking, reaches 79.25 macro-F1 on the test set. This beats the competition winner's 78.55. An error analysis shows that most mistakes involve RELATED comments that share little or only indirect vocabulary with the question. The work matters because it shows that cheap, heterogeneous features and existing Arabic preprocessing tools can set a new state of the art.

Key points

  • Data: the Fatwa corpus (from fatwa.islamweb.net) has 1,300 training, 200 dev and 200 test questions, each with five comments. RELATED comments come from linked related questions, and IRRELEVANT ones are sampled at random.
  • Metrics: the official measures are macro-averaged F1 and accuracy. Macro averaging stops the IRRELEVANT class, which is twice as frequent as the others, from dominating the score.
  • Embeddings: 100-dimensional Word2Vec vectors were trained on Arabic Gigaword, lemmatized with MADAMIRA. These worked better than vectors from KSUCCA, a classical-Arabic corpus closer to the domain, probably because Gigaword offers wider coverage.
  • Similarity features: computed on word 1–4-grams using greedy string tiling, longest common subsequence, Jaccard, containment and cosine. The rule-based ranker relies on QATARA stemming.
  • Results: vectors combined with rule-based ranking give 79.25 F1 and 83.42 accuracy on test. A heuristic that turned a full-feature ranker's order into labels won on dev but did worse on test.

Interesting ideas

  • I1. Bin ranking scores. A continuous ranker score is rescaled to [0,1] and split into 10 bins, each with its own binary feature. This lets a linear model learn a non-linear response to a confidence signal.
  • I2. Use a rule system's output as features. The hand-written rule ranker is not a competitor to the learned model; its binarized decision becomes input to the classifier, and that combination wins.
  • I3. Concatenate instead of compressing to a cosine. The question and answer representations are placed side by side rather than reduced to one similarity number, so the classifier can see which dimensions matter.
  • I4. Watch for easy artificial negatives. The authors point out that randomly sampled IRRELEVANT comments make that class almost trivial, so scores can look better than real difficulty warrants.
  • I5. Label errors by overlap pattern. Errors are sorted into under-specification, over-specification and indirect overlap through synonyms or lemmas. Each category points to a specific remedy.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): parser or witness confidence scores could be binned the same way before they feed a linear repair or arbitration model, while each bin stays readable as a rule condition.
  • I2 (answer): the output of a symbolic answer reader can be fed as an explicit, traceable feature into a learned answer selector rather than replacing it.
  • I3 (frames): when matching a question frame to candidate text frames, keep role-by-role comparison slots instead of one overall match score, so the explanation shows which role failed.
  • I4 (learn): negative controls in evaluation should include hard, topically close distractors, not only random ones, or the gains will be inflated.
  • I5 (world): QA failures can be tagged the same way, so that misses caused by synonymy are routed to the WordNet-based ontology layer rather than to the parser.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States243 works · 76,506 citations🔬MIT Computer Science and Artificial Intelligence LaboratoryCambridge · United States76 works · 32,207 citations🎓Hamad bin Khalifa UniversityDoha · Qatar144 works · 14,175 citations🤝Qatar FoundationDoha · Qatar17 works · 2,111 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksQuestion answering · 487 works

Cite

Cite as Yonatan Belinkov, Alberto Barrón-Cedeño and Hamdy Mubarak. 2015. Answer Selection in Arabic Community Question Answering: A Feature-Rich Approach. Proceedings of the Second Workshop on Arabic Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w15-3223

@inproceedings{belinkov2015answer,
  title = {{Answer Selection in Arabic Community Question Answering: A Feature-Rich Approach}},
  author = {Yonatan Belinkov, Alberto Barrón-Cedeño and Hamdy Mubarak},
  booktitle = {Proceedings of the Second Workshop on Arabic Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/w15-3223},
  url = {https://doi.org/10.18653/v1/w15-3223}
}

References

24 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lloyd Allison and Trevor Dix. 1986. A bit-string longest-common-subsequence algorithm. Inf. Process. Lett., 23(6):305–310, December. ↗
  2. Maha Alrabiah, AbdulMalik Al-Salman, and Eric Atwell. 2013. The design and construction of the 50 million words KSUCCA King Saud University Corpus of Classical Arabic. In Proceedings of WACL-2. find
  3. Yonatan Belinkov, Mitra Mohtarami, Scott Cyphers, and James Glass. 2015. VectorSLU: A continuous word vector approach to answer selection in community question answering systems. In Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval ’15, Denver, Colorado, USA. find
  4. Jiang Bian, Yandong Liu, Eugene Agichtein, and Hongyuan Zha. 2008. Finding the Right Facts in the Crowd: Factoid Question Answering over Social Media. In Proceedings of the 17th International Conference on World Wide Web, WWW ’08, pages 467–476, New York, NY, USA. ACM. find
  5. Kareem Darwish and Walid Magdy. 2013. Arabic information retrieval. Foundations and Trends R in Information Retrieval, 7(4):239–342. ↗
  6. Kareem Darwish, Ahmed Abdelali, and Hamdy Mubarak. 2014. Using Stem-Templates to Improve Arabic POS and Gender/Number Tagging. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014). European Language Resources Association (ELRA).
  7. Ahmed Ezzeldin and Mohamed Shaheen. 2012. A survey of Arabic question answering: Challenges, tasks, approaches, tools, and future trends. In Proceedings of the International Arabic Conference on Information Technology (ACIT). ↗
  8. Yongshuai Hou, Cong Tan, Xiaolong Wang, Yaoyun Zhang, Jun Xu, and Qingcai Chen. 2015. HITSZ-ICRC: Exploiting classification approach for answer selection in community question answering. In Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval ’15, Denver, Colorado, USA. find
  9. Paul Jaccard. 1901. Étude comparative de la distribution florale dans une portion des Alpes et des Jura. Bulletin del la Société Vaudoise des Sciences Naturelles, 37:547–579. find
  10. Thorsten Joachims. 1999. Making Large-scale Support Vector Machine Learning Practical. In Bernhard Schölkopf, Christopher J. C. Burges, and Alexander J. Smola, editors, Advances in Kernel Methods, pages 169–184. MIT Press, Cambridge, MA, USA. ↗
  11. Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’02, pages 133–142, New York, NY, USA. ACM. find
  12. Thorsten Joachims. 2006. Training linear svms in linear time. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’06, pages 217–226, New York, NY, USA. ACM. find
  13. Linguistic Data Consortium. 2011. Arabic Gigaword Fifth Edition. https://catalog.ldc.upenn.edu/LDC2011T11. ↗
  14. Caroline Lyon, James Malcolm, and Bob Dickerson. 2001. Detecting short passages of similar text in large document collections. In Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing, EMNLP ’01, pages 118–125, Pittsburgh, PA, USA. find
  15. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013a. Distributed Representations of Words and Phrases and their Compositionality. In C.J.C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 3111–3119. Curran Associates, Inc. ↗
  16. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT ’13, pages 746–751, Atlanta, GA, USA.
  17. Reham Mohamed, Maha Ragab, Heba Abdelnasser, Nagwa M. El-Makky, and Marwan Torki. 2015. Al-Bayan: A knowledge-based system for Arabic answer selection. In Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval ’15, Denver, Colorado, USA. ↗
  18. Preslav Nakov, Lluı́s Màrquez, Walid Magdy, Alessandro Moschitti, Jim Glass, and Bilal Randeree. 2015. Semeval-2015 task 3: Answer selection in community question answering. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 269–281, Denver, Colorado, June. Association for Computational Linguistics. find
  19. Kevin K. Nam, Ackerman Mark S. , and Lada A. Adamic. 2009. Questions in, Knowledge iN?: A study of Navers question answering community. In Proceedings of the SIG CHI Conference on Human Factors in Computing Systems, Boston, MA. find
  20. Massimo Nicosia, Simone Filice, Alberto Barrón-Cedeño, Iman Saleh, Hamdy Mubarak, Wei Gao, Preslav Nakov, Giovanni Da San Martino, Alessandro Moschitti, Kareem Darwish, Lluı́s Màrquez, Shafiq Joty, and Walid Magdy. 2015. QCRI: Answer selection for community question answering - experiments for Arabic and English. In Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval ’15, Denver, Colorado, USA.
  21. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1094–1101, Reykjavik, Iceland.
  22. Anselmo Peñas, Eduard Hovy, Pamela Forner, Álvaro Rodrigo, Richard Sutcliffe, Caroline Sporleder, Corina Forascu, Yassine Benajiba, and Petya Osenova. 2012. Overview of QA4MRE at CLEF 2012: Question Answering for Machine Reading Evaluation. Working Notes, CLEF. find
  23. Richard Sutcliffe, Anselmo Peñas, Eduard Hovy, Pamela Forner, Álvaro Rodrigo, Corina Forascu, Yassine Benajiba, and Petya Osenova. 2013. Overview of QA4MRE Main Task at CLEF 2013. Working Notes, CLEF. find
  24. Michael Wise. 1996. Yap3: Improved detection of similarities in computer program and other texts. In Proceedings of the Twenty-seventh SIGCSE Technical Symposium on Computer Science Education, SIGCSE ’96, pages 130–134, New York, NY, USA. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/belinkov-2015-answer-selection-arabic-community-question/