Elamur Research⌕
☰
home / papers / 2016 / Addressing Community Question Answering in English and Arabic

Addressing Community Question Answering in English and Arabic

Paper · 2016 · Giovanni Da San Martino, Alberto Barron-Cedeno, Salvatore Romeo et al.

Summary

The paper tackles question re-ranking in community Question Answering (cQA). When a user posts a new question, the system must find earlier forum questions with the same information need, even though forum text is short and noisy. The authors work on two SemEval-2016 Task 3 subtasks: B (English, Qatar Living forum) and D (Arabic, three medical forums). They train learning-to-rank models that combine several feature types: lexical similarity measures, syntactic tree kernels (TKs) applied to pairs of question parses, features encoding the position assigned by Google's original ranking, embeddings, and machine translation evaluation scores. On English, similarity features alone fall below the Google ranking. Combining TKs with an RBF-kernel rank feature beats it by a statistically significant margin. On Arabic, adding tree kernels raises MAP by over seven points. This is the first reported use of TKs on Arabic syntactic trees. Both systems took second place in their subtasks. The work shows that shallow syntactic structure is robust to noisy text and works across languages.

Key points

  • Data come from SemEval-2016 Task 3. The English Question–Question set has 2,669 train, 500 dev and 700 test pairs, and each original question comes with 10 candidates retrieved by Google. Arabic data come from webteb, altibbi and consult islamweb.
  • On the English test set, the Google ranking (GR) baseline reaches 74.75 MAP. Similarity features alone score 70.70, and TKs alone 73.98. Combining TKs and similarities with rank features under an RBF kernel reaches up to 76.47, which is 1.72 points above GR.
  • The top English system, UH-PRHLT, scored 76.70 MAP, which is not statistically different from 76.47. Unlike this paper's system, it relied on BabelNet and FrameNet.
  • Arabic (Task D) results:
    • Embeddings alone: 38.33 MAP (11th place).
    • Adding MTE features: 39.98 (7th).
    • Embeddings plus the syntactic tree kernel (STK) on constituency trees: 45.50 (2nd). The best system scored 45.83 and the baseline 28.88.
  • The paper's motivating example is a pair of "visa for Qatar / visa for my wife" questions. They have nearly identical bag-of-words profiles but ask for different things, which shows why structure matters.

Interesting ideas

  • I1. Learning instances are pairs of trees, original plus candidate, rather than single trees, so the model learns which structural changes keep the meaning and which change it.
  • I2. A strong external ranking is not treated as a rival; its position (or its inverse) is fed in as a feature through an RBF kernel, so candidates with nearby ranks look alike.
  • I3. Same-words, different-meaning question pairs serve as the core argument that bag-of-words similarity is not enough.
  • I4. Machine translation evaluation scores are reused as similarity signals between two monolingual texts, and they gave a cheap gain on Arabic.
  • I5. The authors note that merging all sentences of a question into one flat macro-tree loses the distinction between subject, subquestions and elaborations, and they propose structuring these parts.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Paired-tree comparison could flag when two sentences with the same words get parses that imply different readings, helping to test parser stability on paraphrases.
  • I2 (learn): A trusted witness parser's confidence or rank could be used as a soft feature rather than a hard vote, when silver data are agreed across parsers.
  • I3 (learn): Such minimal pairs with swapped arguments make good negative controls, checking that a reader's answer changes when roles change even though the vocabulary stays fixed.
  • I4 (back): MT metrics could serve as one automatic check in the text → graph → text compressor, measuring how far the regenerated text drifts from the input.
  • I5 (answer): QA over fables and tests could keep the question's sections separate, so the actual request is matched against the text world apart from background details.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksSemitic languages · 2,908 worksQuestion answering · 441 works

Cite

Cite as Giovanni Da San Martino, Alberto Barron-Cedeno, Salvatore Romeo, Alessandro Moschitti, Shafiq Joty, Fahad A. Al Obaidli, Kateryna Tymoshenko and Antonio Uva. 2016. Addressing Community Question Answering in English and Arabic. arXiv:1610.05522.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1610.05522 · arXiv 1610.05522

@misc{martino2016addressing,
  title = {{Addressing Community Question Answering in English and Arabic}},
  author = {Giovanni {Da San Martino}, Alberto Barron-Cedeno, Salvatore Romeo, Alessandro Moschitti, Shafiq Joty, Fahad A. {Al Obaidli}, Kateryna Tymoshenko and Antonio Uva},
  year = {2016},
  doi = {10.48550/arxiv.1610.05522},
  eprint = {1610.05522},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1610.05522}
}

References

31 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. L. Allison and T. Dix. A bit-string longest-common-subsequence algorithm. Inf. Process. Lett., 23(6):305–310, Dec. 1986. ↗
  2. C. F. Baker, C. J. Fillmore, and J. B. Lowe. The berkeley framenet project. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics - Volume 1, ACL ’98, pages 86–90, Stroudsburg, PA, USA, 1998. Association for Computational Linguistics.
  3. A. Barrón-Cedeño, S. Filice, G. Da San Martino, S. Joty, L. Màrquez, P. Nakov, and A. Moschitti. Thread-level information for comment classification in community question answering. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 687–693, Beijing, China, July 2015. Association for Computational Linguistics. find
  4. Y. Belinkov, M. Mohtarami, S. Cyphers, and J. Glass. VectorSLU: A continuous word vector approach to answer selection in community question answering systems. In Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval ’15, Denver, Colorado, USA, 2015. find
  5. X. Cao, G. Cong, B. Cui, C. S. Jensen, and C. Zhang. The use of categorization information in language models for question retrieval. In CIKM, pages 265–274, 2009. find
  6. G. Doddington. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the Second International Conference on Human Language Technology Research, HLT ’02, pages 138–145, 2002. find
  7. H. Duan, Y. Cao, C.-Y. Lin, and Y. Yu. Searching questions by identifying question topic and question focus. In ACL, pages 156–164, 2008. find
  8. R. Eckart de Castilho and I. Gurevych. A broad-coverage collection of portable nlp components for building shareable analysis pipelines. In Proceedings of the Workshop on Open Infrastructures and Analysis Frameworks for HLT, Dublin, Ireland, August 2014.
  9. P. Jaccard. Étude comparative de la distribution florale dans une portion des Alpes et des Jura. Bulletin del la Société Vaudoise des Sciences Naturelles, pages 547–579, 1901. find
  10. Z. Ji, F. Xu, B. Wang, and B. He. Question-answer topic model for question retrieval in community question answering. In CIKM, pages 2471–2474, 2012. find
  11. T. Joachims. Making Large-scale Support Vector Machine Learning Practical. In Advances in Kernel Methods. MIT Press, Cambridge, MA, USA, 1999. ↗
  12. T. Joachims. Optimizing search engines using clickthrough data. KDD, pages 133–142, 2002. find
  13. S. Joty, A. Barrón-Cedeño, G. Da San Martino, S. Filice, L. Màrquez, A. Moschitti, and P. Nakov. Global thread-level inference for comment classification in community question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 573–578, Lisbon, Portugal, September 2015. Association for Computational Linguistics. find
  14. A. Lavie and M. Denkowski. The METEOR metric for automatic evaluation of machine translation. Machine Translation, 23(2–3):105–115, 2009. ↗
  15. C. Lyon, J. Malcolm, and B. Dickerson. Detecting short passages of similar text in large document collections. EMNLP, pages 118–125, 2001. find
  16. C. D. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. J. Bethard, and D. McClosky. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60, 2014.
  17. T. Mikolov, W.-t. Yih, and G. Zweig. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT ’13, pages 746–751, Atlanta, GA, USA, 2013.
  18. A. Moschitti. Efficient Convolution Kernels for Dependency and Constituent Syntactic Trees. In ECML, pages 318–329. 2006. find
  19. P. Nakov, L. Màrquez, A. Moschitti, W. Magdy, H. Mubarak, A. A. Freihat, J. Glass, and B. Randeree. SemEval-2016 task 3: Community question answering. In Proceedings of SemEval ’16. ACL, 2016.
  20. M. Nicosia, S. Filice, A. Barrón-Cedeño, I. Saleh, H. Mubarak, W. Gao, P. Nakov, G. Da San Martino, A. Moschitti, K. Darwish, L. Màrquez, S. Joty, and W. Magdy. QCRI: Answer selection for community question answering - experiments for Arabic and English. In Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval ’15, Denver, Colorado, USA, 2015.
  21. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. BLEU: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meting of the Association for Computational Linguistics, ACL ’02, pages 311–318, Philadelphia, Pennsylvania, USA, 2002.
  22. R. Parker, D. Graff, K. Chen, J. Kong, and K. Maeda. Arabic Gigaword Fifth Edition. Linguistic Data Consortium (LDC), Philadelphia, 2011. find
  23. A. Pasha, M. Al-Badrashiny, M. Diab, A. E. Kholy, R. Eskander, N. Habash, M. Pooleery, O. Rambow, and R. Roth. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland, may 2014.
  24. S. Petrov and D. Klein. Improved inference for unlexicalized parsing. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 404–411, Rochester, New York, April 2007. Association for Computational Linguistics.
  25. A. Severyn and A. Moschitti. Structural relationships for large-scale learning of answer re-ranking. SIGIR, pages 741–750, 2012. find
  26. M. Snover, B. Dorr, R. Schwartz, L. Micciulla, and J. Makhoul. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Biennial Conference of the Association for Machine Translation in the Americas, AMTA ’06, Cambridge, Massachusetts, USA, 2006.
  27. K. Tymoshenko and A. Moschitti. Assessing the impact of syntactic and semantic structures for answer passages reranking. In Proceedings of CIKM ’15, pages 1451–1460, New York, NY, USA, 2015. ACM. find
  28. K. Wang, Z. Ming, and T.-S. Chua. A syntactic tree matching approach to finding similar questions in community-based qa services. In SIGIR, pages 187–194, 2009. find
  29. M. Wise. Yap3: Improved detection of similarities in computer program and other texts. In SIGCSE, pages 130–134, 1996. find
  30. K. Zhang, W. Wu, H. Wu, Z. Li, and M. Zhou. Question retrieval with high quality answers in community question answering. In CIKM, pages 371–380, 2014. find
  31. G. Zhou, L. Cai, J. Zhao, and K. Liu. Phrase-based translation model for question retrieval in community question answer archives. In ACL, pages 653–662, 2011. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/martino-2016-addressing-community-question-answering-english/