Elamur Research⌕
☰
home / papers / 2015 / QCRI: Answer Selection for Community Question Answering - Experiments for Arabic and English

QCRI: Answer Selection for Community Question Answering - Experiments for Arabic and English

Paper · 2015 · Massimo Nicosia, Simone Filice, Alberto Barrón-Cedeño et al.

Summary

The paper reports QCRI's entry in SemEval-2015 Task 3, which asks whether a comment in a Web forum thread actually answers the question that opened the thread. The Arabic data come from Fatwa, where scholars write the answers and each one is labeled DIRECT, RELATED or IRRELEVANT. The English data come from Qatar Living, a noisy user-written forum, where comments are labeled GOOD, POTENTIAL or BAD. A further English subtask asks for an overall YES, NO or UNSURE verdict on yes/no questions.

The team built supervised classifiers: logistic regression for Arabic and a linear SVM for English subtask A. These draw on many feature families:

  • lexical, syntactic (tree-kernel) and embedding-based question–comment similarity;
  • thread context and user behaviour;
  • n-grams;
  • simple heuristics;
  • sentiment polarity.

The system won the Arabic subtask by a wide margin and placed third in both English subtasks. Ablations after the competition show which feature families matter for each language. They also expose how fragile small-sample user statistics can be. This makes the paper a useful map of signals for judging answer quality.

Key points

  • Arabic subtask A: the primary run ranked first with macro-F1 78.55; the runner-up scored 70.99. After the competition, n-grams plus the stacked output of the contrastive run cont 1 reached 78.69.
  • English official results: 53.74 F1 on subtask A and 53.60 on subtask B, both third place. Dropping the n-grams from subtask A gave 55.17.
  • Similarity features combined several kinds of measure:
    • greedy string tiling, LCS, Jaccard, containment and cosine over 1–4-grams, plus idf-weighted overlap;
    • a partial tree kernel over shallow syntactic trees;
    • summed word vectors from LSA trained on Qatar Living, GloVe (Common Crawl, 300d) and COMPOSES (400d).
  • Polarity for subtask B was a summed word score from the NRC Hashtag Sentiment Lexicon v0.1, together with yes/no keyword flags. A rule-based run that read each comment's opening word and took a majority vote scored 47.06 F1 on UNSURE, against 36.36 for the primary run.
  • Subtask B results were unstable. Fixing a polarity bug gave 51.98. Removing per-category user counts raised this to 55.95. Training only on train data gave 69.35, but that gap came from just four misclassified questions and is not significant.

Interesting ideas

  • I1. Model the thread around a comment: whether the asker later thanks someone or asks again, with the signal decaying linearly by the number of comments in between (max(0, 1.1 − 0.1k)); this captures discourse reactions that the comment's own words cannot.
  • I2. A trivial rule (classify each answer by its first word as affirmative or negative, then take a majority vote with ties going to UNSURE) beat the learned model on the hardest class, showing that cheap transparent baselines can expose model blind spots.
  • I3. Run every feature family both alone ("only") and as a removal from the full set ("without"); this separates families that are strong by themselves from those that add uncorrelated information.
  • I4. Statistics computed over sparse slices, such as per-user counts within one category, shifted a lot when the training set grew and hurt results, so features should be checked for stability across data splits.
  • I5. Measure question–answer similarity with a partial tree kernel over lemma, POS and chunk trees instead of plain word overlap, so that structure contributes to relevance scoring.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): acknowledgments and follow-up questions could be tagged as speech acts linked to earlier turns, giving explainable evidence about whether a turn resolved a request.
  • I2 (answer): yes/no readers could keep such a first-word vote as a negative-control baseline and flag cases where the full frame-based reasoning disagrees with it.
  • I3 (learn): the same paired ablation could measure what each pipeline signal (lemmas, roles, world facts) contributes to answering accuracy.
  • I4 (learn): when silver data or repair rules depend on rare-context counts, compare them across resampled corpora and drop those that swing.
  • I5 (tree): matching subtrees between UD parses of a question and a candidate sentence could give an explainable alignment score in place of bag-of-words similarity.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TrentoTrento · Italy85 works · 15,524 citations🤝Qatar FoundationDoha · Qatar17 works · 2,111 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksSemitic languages · 2,908 worksQuestion answering · 441 works

Cite

Cite as Massimo Nicosia, Simone Filice, Alberto Barrón-Cedeño, Iman Saleh, Hamdy Mubarak, Wei Gao, Preslav Nakov, Giovanni Da San Martino, Alessandro Moschitti, Kareem Darwish, Lluís Màrquez, Shafiq Joty and Walid Magdy. 2015. QCRI: Answer Selection for Community Question Answering - Experiments for Arabic and English. Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s15-2036

@inproceedings{nicosia2015qcri,
  title = {{QCRI: Answer Selection for Community Question Answering - Experiments for Arabic and English}},
  author = {Massimo Nicosia, Simone Filice, Alberto Barrón-Cedeño, Iman Saleh, Hamdy Mubarak, Wei Gao, Preslav Nakov, Giovanni {Da San Martino}, Alessandro Moschitti, Kareem Darwish, Lluís Màrquez, Shafiq Joty and Walid Magdy},
  booktitle = {Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)},
  year = {2015},
  doi = {10.18653/v1/s15-2036},
  url = {https://doi.org/10.18653/v1/s15-2036}
}

References

14 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lloyd Allison and Trevor Dix. 1986. A bit-string longest-common-subsequence algorithm. Inf. Process. Lett., 23(6):305–310, December. ↗
  2. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL ’14, pages 238–247, Baltimore, MD, USA.
  3. Danilo Croce and Daniele Previtali. 2010. Manifold Learning for the Semi-Supervised Induction of FrameNet Predicates: An Empirical Investigation. In Proceedings of the 2010 Workshop on GEometrical Models of Natural Language Semantics, GEMS ’10, pages 7–16, Uppsala, Sweden. find
  4. Paul Jaccard. 1901. Étude comparative de la distribution florale dans une portion des Alpes et des Jura. Bulletin del la Société Vaudoise des Sciences Naturelles, 37:547–579. find
  5. Thorsten Joachims. 1999. Making Large-scale Support Vector Machine Learning Practical. In Bernhard Schölkopf, Christopher J. C. Burges, and Alexander J. Smola, editors, Advances in Kernel Methods, pages 169–184. MIT Press, Cambridge, MA, USA. ↗
  6. Caroline Lyon, James Malcolm, and Bob Dickerson. 2001. Detecting short passages of similar text in large document collections. In Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing, EMNLP ’01, pages 118–125, Pittsburgh, PA, USA. find
  7. Lluı́s Màrquez, James Glass, Walid Magdy, Alessandro Moschitti, Preslav Nakov, and Bilal Randeree. 2015. SemEval-2015 Task 3: Answer Selection in Community Question Answering. In Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval ’15, Denver, CO, USA. find
  8. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT ’13, pages 746– 751, Atlanta, GA, USA.
  9. Saif Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. NRC-Canada: Building the state-of-the-art in sentiment analysis of tweets. In Proceedings of the Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation, SemEval ’13, pages 321–327, Atlanta, GA, USA. ↗
  10. Alessandro Moschitti. 2006. Efficient Convolution Kernels for Dependency and Constituent Syntactic Trees. In Johannes Fürnkranz, Tobias Scheffer, and Myra Spiliopoulou, editors, Machine Learning: ECML 2006, volume 4212 of Lecture Notes in Computer Science, pages 318–329. Springer Berlin Heidelberg. find
  11. Ramesh Nallapati. 2004. Discriminative models for information retrieval. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’04, pages 64–71, Sheffield, United Kingdom. find
  12. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP ’14, pages 1532–1543, Doha, Qatar.
  13. Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28:11–21. ↗
  14. Michael Wise. 1996. Yap3: Improved detection of similarities in computer program and other texts. In Proceedings of the Twenty-seventh SIGCSE Technical Symposium on Computer Science Education, SIGCSE ’96, pages 130–134, New York, NY, USA. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nicosia-2015-qcri-answer-selection-community-question/