Elamur Research⌕
☰
home / papers / 2016 / SemEval-2016 Task 3: Community Question Answering

SemEval-2016 Task 3: Community Question Answering

Summary

Community forums such as Qatar Living collect many loosely moderated threads in which useful replies are mixed with noise, typos and off-topic chatter. This makes it hard for users to find good answers. The organizers extended the 2015 answer-selection challenge into a fuller community question answering (CQA) setup. The core goal is to rank existing forum comments by how well they answer a new question.

English data from Qatar Living supported three subtasks: ranking comments within a question's own thread (A), ranking retrieved related questions (B), and ranking comments drawn from those related threads (C). A fourth subtask (D) used Arabic medical-forum data and asked systems to rerank question–answer pairs for a new question. Crowdsourced annotators labeled all the data, and MAP served as the official metric.

Eighteen teams submitted 95 runs. The best systems clearly beat the baselines. The released corpus contains more than 7,000 questions and more than 57,000 labeled comments, giving the community a reusable benchmark for CQA research.

Key points

  • Subtasks and labels:
    • A labels comments as Good, PotentiallyUseful or Bad, and only the first ten comments per thread are used.
    • B labels questions as PerfectMatch, Relevant or Irrelevant, with the first two merged into one positive class.
    • C judges external comments against the original question.
    • D labels Arabic pairs as Direct, Relevant or Irrelevant.
  • Participation: 18 teams submitted 95 runs (38 primary, 57 contrastive), up from 13 teams and 61 runs in 2015.
  • Best results: the top MAP scores were 79.19 (A), 76.70 (B), 55.41 (C) and 45.83 (D). In subtask A, the winner's Accuracy was 3 points absolute higher than the 2015 winner's.
  • Learning methods: SVMs were the most common approach, used for classification, regression and ranking.
    • Systems based on tree kernels (ConvKN, KeLP) took first and second place in A, plus high places in the other subtasks.
    • Neural networks won no subtask outright, but they contributed to every top system, for example through word2vec.
  • Arabic subtask: SLS won D using text- and vector-based similarity features. QU-IR introduced Covariance Word Embedding, and RDI used Arabic-Wikipedia to give medical terms more weight.

Interesting ideas

  • I1. Do the retrieval step for participants. The organizers ran the search engine themselves and handed out fixed candidate lists. People without IR expertise could then compete on reranking alone, which is a useful way to make any shared task more accessible.
  • I2. Split the main task into optional subtasks. Comment relevance and question similarity are smaller problems that can be combined into the full system. This lets teams contribute to only part of a complex pipeline.
  • I3. Collapse graded labels for a ranking metric. Annotators used three-way labels, but evaluation used a binary split, with PotentiallyUseful counted as Bad. This keeps the richer annotation available while giving a clear MAP-based target.
  • I4. Reuse machine translation metrics as similarity features. Scores such as BLEU, TER, NIST and Meteor served as features for comparing texts. This is a cheap, language-agnostic way to measure how close two pieces of text are.
  • I5. Build separate feature sets per text pair, then concatenate them. SLS computed one feature set for the original question vs. the related question, and another for the original question vs. the related answer, then joined them into one vector. This is a simple way to combine several pairwise signals for one candidate.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

75 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 74 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s16-1083

@inproceedings{nakov2016semeval,
  title = {{SemEval-2016 Task 3: Community Question Answering}},
  author = {Nakov, Preslav and Màrquez, Lluís and Moschitti, Alessandro and Magdy, Walid and Mubarak, Hamdy and Freihat, Abed Alhakim and Glass, Jim and Randeree, Bilal},
  booktitle = {Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)},
  year = {2016},
  doi = {10.18653/v1/s16-1083},
  url = {https://doi.org/10.18653/v1/s16-1083}
}

References

38 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Balchev, Yasen Kiprov, Ivan Koychev, and Preslav Nakov. 2016. PMI-cool at SemEval-2016 Task 3: Experiments with PMI and goodness polarity lexicons for community question answering. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, San Diego, CA. Alberto Ba… find
  2. Gordon V Cormack, Charles LA Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd international ACM SI-GIR conference on Research and development in information retrieval, SIGIR ’09, pages 758–759, Boston, M… find our card
  3. Chang e Jia. 2016. ITNLP-AiKF at SemEval-2016 Task 3: Community question answering. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, San Diego, CA. Yassine El Adlouni, Imane Lahbari, Horacio Rodrı́guez, Mohammed Meknassi, and Said Ouatik El Alaoui. 2016. UPC-USM… find
  4. Simone Filice, Giuseppe Castellucci, Danilo Croce, Giovanni Da San Martino, Alessandro Moschitti, and Roberto Basili. 2015. KeLP: a Kernel-based Learning Platform in java. In Proceedings of the workshop on Machine Learning Open Source Software: Open Ecosystems, Lille, France. find
  5. Simone Filice, Danilo Croce, Alessandro Moschitti, and Roberto Basili. 2016. KeLP at SemEval-2016 Task 3: Learning semantic relations between questions and answers. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, San Diego, CA. find
  6. Marc Franco-Salvador, Sudipta Kar, Thamar Solorio, and Paolo Rosso. 2016. UH-PRHLT at SemEval-2016 Task 3: Combining lexical and semantic-based features for community question answering. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, San Diego, CA. Francisco G… find
  7. Michael Heilman and Noah A. Smith. 2010. Tree edit models for recognizing textual entailments, paraphrases, and answers to questions. In Proceedings of the Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, NAACL-HL… find our card
  8. Doris Hoogeveen, Yitong Li, Huizhi Liang, Bahar Salehi, Timothy Baldwin, and Long Duong. 2016. UniMelb at SemEval-2016 Task 3: Identifying similar questions by combining a CNN with string similarity measures. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, San… find
  9. Yongshuai Hou, Cong Tan, Xiaolong Wang, Yaoyun Zhang, Jun Xu, and Qingcai Chen. 2015. HITSZ-ICRC: Exploiting classification approach for answer selection in community question answering. In Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval ’15, pages 196–202, Denver, CO. find
  10. Jiwoon Jeon, W. Bruce Croft, and Joon Ho Lee. 2005. Finding similar questions in large question and answer archives. In Proceedings of the 14th ACM International Conference on Information and Knowledge Management, CIKM ’05, pages 84–90, Bremen, Germany. Shafiq Joty, Alberto Barrón-Cedeño, Giovanni… find
  11. John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, pages 282–289, San Francisco, CA. find
  12. Ahmed Magooda, Amr Gomaa, Ashraf Mahgoub, Hany Ahmed, Mohsen Rashwan, Hazem Raafat, Eslam Kamal, and Ahmad Al Sallab. 2016. RDI at SemEval-2016 Task 3: RDI unsupervised framework for text ranking. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, San Diego, CA. find
  13. Rana Malhas, Marwan Torki, and Tamer Elsayed. 2016. QU-IR at SemEval-2016 Task 3: Learning to rank on Arabic community question answering forums with word embedding. In Proceedings of the 10th International Workshop on Semantic Evaluation, Sem-Eval ’16, San Diego, CA. find
  14. Todor Mihaylov and Preslav Nakov. 2016a. Hunting for troll comments in news community forums. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL ’16, Berlin, Germany. find
  15. Todor Mihaylov and Preslav Nakov. 2016b. SemanticZ at SemEval-2016 Task 3: Ranking relevant answers in community question answering using semantic similarity based on fine-tuned word embeddings. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, San Diego, CA. find
  16. Todor Mihaylov, Georgi Georgiev, and Preslav Nakov. 2015a. Finding opinion manipulation trolls in news community forums. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning, CoNLL ’15, pages 310–314, Beijing, China. find
  17. Todor Mihaylov, Ivan Koychev, Georgi Georgiev, and Preslav Nakov. 2015b. Exposing paid opinion manipulation trolls. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP ’15, pages 443–450, Hissar, Bulgaria. find
  18. Tsvetomila Mihaylova, Pepa Gencheva, Martin Boyanov, Ivana Yovcheva, Todor Mihaylov, Momchil Hardalov, Yasen Kiprov, Daniel Balchev, Ivan Koychev, Preslav Nakov, Ivelina Nikolova, and Galia Angelova. 2016. SUper Team at SemEval-2016 Task 3: Building a feature-rich system for community question answe… find
  19. Mitra Mohtarami, Yonatan Belinkov, Wei-Ning Hsu, Yu Zhang, Tao Lei, Kfir Bar, Scott Cyphers, and Jim Glass. 2016. SLS at SemEval-2016 Task 3: Neural-based approaches for ranking in community question answering. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, Sa… find
  20. Alessandro Moschitti, Silvia Quarteroni, Roberto Basili, and Suresh Manandhar. 2007. Exploiting syntactic and shallow semantic kernels for question answer classification. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, ACL ’07, pages 776–783, Prague, Czech… find
  21. Alessandro Moschitti. 2008. Kernel methods, syntax and semantics for relational text categorization. In Proceedings of the 17th ACM Conference on Information and Knowledge Management, CIKM ’08, pages 253– 262, Napa Valley, California, USA. Preslav Nakov, Lluı́s Màrquez, Walid Magdy, Alessandro Mosc… find
  22. Filip Radlinski and Thorsten Joachims. 2005. Query chains: Learning to rank from implicit feedback. In Proceedings of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery in Data Mining, KDD ’05, pages 239–248, Chicago, IL. find
  23. Aliaksei Severyn and Alessandro Moschitti. 2012. Structural relationships for large-scale learning of answer re-ranking. In Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’12, pages 741–750, Portland, OR. find
  24. Aliaksei Severyn and Alessandro Moschitti. 2013. Automatic feature engineering for answer selection and extraction. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP ’13, pages 458–467, Seattle, WA. find
  25. Aliaksei Severyn and Alessandro Moschitti. 2015. Learning to rank short text pairs with convolutional deep neural networks. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’15, pages 373–382, Santiago, Chile. find
  26. Dan Shen and Mirella Lapata. 2007. Using semantic roles to improve question answering. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL ’07, pages 12–21, Prague, Czech Republic. find our card
  27. Mihai Surdeanu, Massimiliano Ciaramita, and Hugo Zaragoza. 2008. Learning to rank answers on large online QA collections. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics and the Human Language Technology Conference, ACL-HLT ’08, pages 719–727, Columbus, Ohi… find
  28. Kateryna Tymoshenko and Alessandro Moschitti. 2015. Assessing the impact of syntactic and semantic structures for answer passages reranking. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, CIKM ’15, pages 1451–1460, Melbourne, Australia. find
  29. Kateryna Tymoshenko, Daniele Bonadiman, and Alessandro Moschitti. 2016. Convolutional neural networks vs. convolution kernels: Feature engineering for answer sentence reranking. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Huma… find
  30. Mengqiu Wang and Christopher D. Manning. 2010. Probabilistic tree-edit models with structured latent variables for textual entailment and question answering. In Proceedings of the 23rd International Conference on Computational Linguistics, COLING ’10, pages 1164–1172, Beijing, China. find
  31. Hujie Wang and Pascal Poupart. 2016. Overfitting at SemEval-2016 Task 3: Detecting semantically similar questions in community question answering forums with word embeddings. In Proceedings of the 10th International Workshop on Semantic Evaluation, Sem-Eval ’16, San Diego, CA. find
  32. Mengqiu Wang, Noah A. Smith, and Teruko Mitamura. 2007. What is the Jeopardy model? A quasi-synchronous grammar for QA. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL ’07, pages 22– 32, Prague,… find our card
  33. Guoshun Wu and Man Lan. 2016. ECNU at SemEval-2016 Task 3: Exploring traditional method and deep learning method for question retrieval and answer ranking in community question answering. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, San Diego, CA. find
  34. Yunfang Wu and Minghua Zhang. 2016. ICL00 at SemEval-2016 Task 3: Translation-based method for cqa system. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, San Diego, CA. find
  35. Xuchen Yao, Benjamin Van Durme, Chris Callison-Burch, and Peter Clark. 2013. Answer extraction as sequence tagging with tree edit distance. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT ’13… find
  36. Xiaoqiang Zhou, Baotian Hu, Qingcai Chen, Buzhou Tang, and Xiaolong Wang. 2015a. Answer sequence learning with neural networks for answer selection in community question answering. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pages 713–718, Beijing, Chi… find
  37. Xiaoqiang Zhou, Baotian Hu, Jiaxin Lin, Yang Xiang, and Xiaolong Wang. 2015b. ICRC-HIT: A deep learning based comment sequence labeling system for answer selection challenge. In Proceedings of the 9th International Workshop on Semantic Evaluation, Sem-Eval ’15, pages 210–214, Denver, CO. 2 3 6 7 9 1… find
  38. Submission Kelp-primary ConvKN-contrastive1 SUper team-contrastive1 ConvKN-primary SemanticZ-primary ConvKN-contrastive2 ECNU-primary SemanticZ-contrastive1 SUper team-primary MTE-NN-contrastive2 SUper team-contrastive2 MTE-NN-contrastive1 SLS-contrastive2 SLS-contrastive1 MTE-NN-primary SLS-primary… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nakov-2016-semeval-2016-task-3-community/