Elamur Research⌕
☰
home / papers / 2016 / ConvKN at SemEval-2016 Task 3: Answer and Question Selection for Question Answering on Arabic and English Fora

ConvKN at SemEval-2016 Task 3: Answer and Question Selection for Question Answering on Arabic and English Fora

Paper · 2016 · Alberto Barrón-Cedeño, Daniele Bonadiman, Giovanni Da San Martino et al.

Summary

The paper presents ConvKN, a joint QCRI and University of Trento entry to SemEval-2016 Task 3 on Community Question Answering. The task asks systems to rerank forum content. Tasks A–C use English threads from Qatar Living: comments against their own question, related questions against a new question, and comments against a new question. Task D uses Arabic question–answer pairs from three medical fora. In every task, a binary SVM's score serves as the relevance measure. The SVM combines several components:

  • tree kernels over shallow syntactic trees of the two texts, with links drawn where words match;
  • hand-crafted similarity and thread-context features carried over from SemEval-2015;
  • for one task, question and answer embeddings learned by a CNN.

The authors describe this as the first application of tree kernels to Arabic syntactic trees in a reranking setting. The primary runs placed second in Tasks A, B and D. Task C, submitted without tree kernels, placed only eighth. This pattern suggests that automatically engineered structural features carry much of the system's strength.

Key points

  • The primary metric is MAP. Primary-run scores:
    • Task A: 77.66 (best system 79.19).
    • Task B: 76.02 (best 76.70).
    • Task C: 47.15 (best 55.41).
    • Task D: 45.50, close to the best 45.83 and far above the random baseline of 28.88.
  • English preprocessing ran through DKPro Core, using OpenNLP's tokenizer, POS tagger and chunker plus Stanford's lemmatizer. Arabic text was segmented with MADAMIRA, split into sentences with the Stanford splitter and parsed with the Berkeley parser.
  • The tree kernels differ by language. PTK was used for English. SST was used for the large, noisy Arabic trees, with parameter L set to 0.1 to suppress noise.
  • In Arabic Task D, adding tree kernels to the word2vec embedding features (trained on Arabic Gigaword) improved MAP by more than 7 points. Adding MT-evaluation features instead (BLEU, TER, Meteor, NIST) raised MAP only from 38.33 to 39.98.
  • In Task A, a contrastive run added a fully-connected CRF that labels all comments of a thread jointly. It lifted MAP to 78.71, more than a point above the primary run.

Interesting ideas

  • I1. The question and candidate trees are tied together: POS nodes sharing a non-stop lexical match get a special tag, and so do the chunk nodes above them. The kernel then compares question parts with question parts and answer parts with answer parts, so the overlap becomes part of the structure itself.
  • I2. The kernel choice was adapted to data quality: the cheaper SST with a strong decay replaced PTK on large, noisy Arabic parses, which keeps the comparison from being swamped by spurious fragments.
  • I3. Comments in a thread were labeled jointly by a CRF whose pairwise potential only asks whether two labels are the same or different. This simple coupling captures that neighbouring posts, such as dialogue chatter, tend to share a label.
  • I4. For the new-question task, a rule-based run combined outputs from two other tasks: a comment counts as good only if it answers its own question well and that question matches the new one.
  • I5. Each contrastive run removes one component or changes the training data, which isolates what each piece contributes. The authors also report a cautionary result: they distrusted MT features and diverse training data because of development-set results, yet both helped on the test set.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): matched lemma nodes and their parent phrases could be tagged when aligning a question's UD tree with sentences of a tale, giving an alignment that is both structural and explainable.
  • I2 (tree): when gparse or the perceptron parser yields low-confidence trees, any tree-similarity step could fall back to coarser, heavily damped comparisons.
  • I3 (answer): candidate answers drawn from the same passage could be scored together, with a "same verdict / different verdict" coupling that encourages consistent answers.
  • I4 (answer): a question can be decomposed into chained, separately checkable decisions, so each failure points to one explainable link.
  • I5 (learn): ablation runs should be standard in evaluation, and conclusions drawn from a single development split should be treated as weak, especially decisions to discard training data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar144 works · 14,175 citations🎓University of TrentoTrento · Italy87 works · 15,761 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,379 worksSemitic languages · 3,066 worksQuestion answering · 487 works

Cite

Cite as Alberto Barrón-Cedeño, Daniele Bonadiman, Giovanni Da San Martino, Shafiq Joty, Alessandro Moschitti, Fahad Al Obaidli, Salvatore Romeo, Kateryna Tymoshenko and Antonio Uva. 2016. ConvKN at SemEval-2016 Task 3: Answer and Question Selection for Question Answering on Arabic and English Fora. Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s16-1138

@inproceedings{barroncedeno2016convkn,
  title = {{ConvKN at SemEval-2016 Task 3: Answer and Question Selection for Question Answering on Arabic and English Fora}},
  author = {Alberto Barrón-Cedeño, Daniele Bonadiman, Giovanni {Da San Martino}, Shafiq Joty, Alessandro Moschitti, Fahad {Al Obaidli}, Salvatore Romeo, Kateryna Tymoshenko and Antonio Uva},
  booktitle = {Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)},
  year = {2016},
  doi = {10.18653/v1/s16-1138},
  url = {https://doi.org/10.18653/v1/s16-1138}
}

References

30 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alberto Barrón-Cedeño, Simone Filice, Giovanni Da San Martino, Shafiq Joty, Lluı́s Màrquez, Preslav Nakov, and Alessandro Moschitti. 2015. Thread-level information for comment classification in community question answering. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 687–693, Beijing, China, July. Association for Computational Linguistics. find
  2. Yonatan Belinkov, Mitra Mohtarami, Scott Cyphers, and James Glass. 2015. Vectorslu: A continuous word vector approach to answer selection in community question answering systems. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 282–287, Denver, Colorado, June. Association for Computational Linguistics. find
  3. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the Second International Conference on Human Language Technology Research, HLT ’02, pages 138–145, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  4. Richard Eckart de Castilho and Iryna Gurevych. 2014. A broad-coverage collection of portable nlp components for building shareable analysis pipelines. In Proceedings of the Workshop on Open Infrastructures and Analysis Frameworks for HLT, pages 1–11, Dublin, Ireland, August. Association for Computational Linguistics and Dublin City University.
  5. Simone Filice, Giuseppe Castellucci, Danilo Croce, and Roberto Basili. 2015. Kelp: a kernel-based learning platform for natural language processing. In Proceedings of ACL-IJCNLP 2015 System Demonstrations, pages 19–24, Beijing, China, July. Association for Computational Linguistics and The Asian Federation of Natural Language Processing. find
  6. Thorsten Joachims. 1999. Making Large-scale Support Vector Machine Learning Practical. In Bernhard Schölkopf, Christopher J. C. Burges, and Alexander J. Smola, editors, Advances in Kernel Methods, pages 169–184. MIT Press, Cambridge, MA, USA. ↗
  7. Shafiq Joty, Alberto Barrón-Cedeño, Giovanni Da San Martino, Simone Filice, Lluı́s Màrquez, Alessandro Moschitti, and Preslav Nakov. 2015. Global thread-level inference for comment classification in community question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 573–578, Lisbon, Portugal, September. Association for Computational Linguistics. find
  8. Shafiq Joty, Llus Mrquez, and Preslav Nakov. 2016. Joint learning with global inference for comment classification in community question answering. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages xx– xx, San Diego, California, June. Association for Computational Linguistics. find
  9. Alon Lavie and Michael Denkowski. 2009. The ME-TEOR metric for automatic evaluation of machine translation. Machine Translation, 23(2–3):105–115. find
  10. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60. ↗
  11. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT ’13, pages 746– 751, Atlanta, GA, USA.
  12. Alessandro Moschitti, Silvia Quarteroni, Roberto Basili, and Suresh Manandhar. 2007. Exploiting syntactic and shallow semantic kernels for question answer classification. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 776–783, Prague, Czech Republic, June. Association for Computational Linguistics. find
  13. Alessandro Moschitti. 2006. Efficient Convolution Kernels for Dependency and Constituent Syntactic Trees. In Johannes Fürnkranz, Tobias Scheffer, and Myra Spiliopoulou, editors, Machine Learning: ECML 2006, volume 4212 of Lecture Notes in Computer Science, pages 318–329. Springer Berlin Heidelberg. find
  14. Alessandro Moschitti. 2008. Kernel Methods, Syntax and Semantics for Relational Text Categorization. In Proceeding of ACM 17th Conf. on Information and Knowledge Management (CIKM’08), Napa Valley, CA, USA. find
  15. Preslav Nakov, Lluı́s Màrquez, Walid Magdy, Alessandro Moschitti, Jim Glass, and Bilal Randeree. 2015. Semeval-2015 task 3: Answer selection in community question answering. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 269–281, Denver, Colorado, June. Association for Computational Linguistics. find
  16. Preslav Nakov, Lluı́s Màrquez, Alessandro Moschitti, Walid Magdy, Hamdy Mubarak, Abed Alhakim Freihat, Jim Glass, and Bilal Randeree. 2016. SemEval-2016 task 3: Community question answering. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, San Diego, California, June. Association for Computational Linguistics.
  17. Massimo Nicosia, Simone Filice, Alberto Barrón-Cedeño, Iman Saleh, Hamdy Mubarak, Wei Gao, Preslav Nakov, Giovanni Da San Martino, Alessandro Moschitti, Kareem Darwish, Lluı́s Màrquez, Shafiq Joty, and Walid Magdy. 2015. QCRI: Answer selection for community question answering - experiments for Arabic and English. In Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval ’15, Denver, Colorado, USA.
  18. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meting of the Association for Computational Linguistics, ACL ’02, pages 311–318, Philadelphia, Pennsylvania, USA.
  19. Robert Parker, David Graff, Ke Chen, Junbo Kong, and Kazuaki Maeda. 2011. Arabic Gigaword Fifth Edition. Linguistic Data Consortium (LDC), Philadelphia. ↗
  20. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland, may. European Language Resources Association (ELRA).
  21. Slav Petrov and Dan Klein. 2007. Improved inference for unlexicalized parsing. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 404–411, Rochester, New York, April. Association for Computational Linguistics.
  22. Aliaksei Severyn and Alessandro Moschitti. 2012. Structural relationships for large-scale learning of answer re-ranking. In Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’12, pages 741–750, New York, NY, USA. ACM. find
  23. Aliaksei Severyn and Alessandro Moschitti. 2013. Automatic feature engineering for answer selection and extraction. In EMNLP, pages 458–467. ACL. find
  24. Aliaksei Severyn and Alessandro Moschitti. 2015. Learning to rank short text pairs with convolutional deep neural networks. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 373– 382. ACM. find
  25. Aliaksei Severyn, Massimo Nicosia, and Alessandro Moschitti. 2013a. Building structures from classifiers for passage reranking. In Proceedings of the 22nd ACM international conference on Conference on information & knowledge management, CIKM ’13, pages 969–978, New York, NY, USA. ACM. find
  26. Aliaksei Severyn, Massimo Nicosia, and Alessandro Moschitti. 2013b. Learning adaptable patterns for passage reranking. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, CoNLL ’13, pages 75–83, Sofia, Bulgaria. find
  27. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Biennial Conference of the Association for Machine Translation in the Americas, AMTA ’06, Cambridge, Massachusetts, USA.
  28. Kateryna Tymoshenko and Alessandro Moschitti. 2015. Assessing the impact of syntactic and semantic structures for answer passages reranking. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, CIKM ’15, pages 1451–1460, New York, NY, USA. ACM. find
  29. Kateryna Tymoshenko, Alessandro Moschitti, and Aliaksei Severyn. 2014. Encoding semantic resources in syntactic structures for passage reranking. In 14th Conference of the European Chapter of the Association for Computational Linguistics 2014, EACL 2014, pages 664–672, Gothenburg, Sweden, April. Association for Computational Linguistics (ACL). find
  30. Kateryna Tymoshenko, Daniele Bonadiman, and Alessandro Moschitti. 2016. Convolutional neural networks vs. convolution kernels: Feature engineering for answer sentence reranking. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL ’16, San Diego, CA, USA, June. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cedeno-2016-convkn-semeval-2016-task-3/