Elamur Research⌕
☰
home / papers / 2011 / Learning to Rank Answers to Non-Factoid Questions from Web Collections

Learning to Rank Answers to Non-Factoid Questions from Web Collections

Paper · 2011 · Mihai Surdeanu, Massimiliano Ciaramita and Hugo Zaragoza

Summary

Most question answering research targets factoid questions with short entity-like answers. This paper instead tackles ranking answers to non-factoid "How to" (manner) questions, where annotated training data is scarce. The authors argue that community QA sites can serve as free supervision. Yahoo! Answers threads come with user-chosen best answers, which they treat as positives and all other answers as negatives. Their pipeline first retrieves candidates with a standard IR model. A re-ranker called FMIX then scores them as a linear combination of four feature groups:

  • similarity,
  • question-to-answer translation,
  • content frequency/density,
  • correlation with external collections.

These features are computed over several text views, from bags of words and n-grams to syntactic dependencies and semantic role dependencies, with WordNet-based semantic tags used as back-off. Combining the features raises MRR and Precision@1 by 14–21% relative to the baseline. This is strong large-scale evidence that rich linguistic analysis can improve retrieval on noisy web data, without any manual annotation.

Key points

  • The evaluation covers over 25,000 "How to" questions from Yahoo! Answers, each with up to 100 candidate answers. All text was processed with unadapted, off-the-shelf NL tools.
  • Translation-style features linking question structure to answer structure contribute the most. Still, the best results require combining all feature families.
  • Semantic role dependencies and WordNet-sense labels give a statistically significant gain over bag-of-words and n-gram representations.
  • Treating syntactic and semantic structures as simple "bags" of dependencies performs at least as well as heavier models such as tree kernels.
  • Suggested future work includes graded relevance from non-best but upvoted answers and adapting the ranker to other domains (news, blogs).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,462 citations🎓Stanford UniversityStanford · United States506 works · 330,052 citations🏢YahooSunnyvale · United States8 works · 11,973 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 552 works

Cite

Cite as Mihai Surdeanu, Massimiliano Ciaramita and Hugo Zaragoza. 2011. Learning to Rank Answers to Non-Factoid Questions from Web Collections. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00051

@article{surdeanu2011learning,
  title = {{Learning to Rank Answers to Non-Factoid Questions from Web Collections}},
  author = {Mihai Surdeanu, Massimiliano Ciaramita and Hugo Zaragoza},
  journal = {Computational Linguistics},
  year = {2011},
  doi = {10.1162/coli\_a\_00051},
  url = {https://doi.org/10.1162/coli_a_00051}
}

References

46 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agichtein, Eugene, Carlos Castillo, Debora Donato, Aristides Gionis, and Gilad Mishne. 2008. Finding high-quality content in social media, with an application to community-based question answering. In Proceedings of the Web Search and Data Mining Conference (WSDM), pages 183–194, Stanford, CA. find
  2. Agichtein, Eugene, Steve Lawrence, and Luis Gravano. 2001. Learning search engine specific query transformations for question answering. In Proceedings of the World Wide Web Conference, pages 169–178, Hong Kong. find
  3. Attardi, Giuseppe, Felice Dell’Orletta, Maria Simi, Atanas Chanev, and Massimiliano Ciaramita. 2007. Multilingual dependency parsing and domain adaptation using DeSR. In Proceedings of the Shared Task of the Conference on Computational Natural Language Learning (CoNLL), pages 1112–1118, Prague. find
  4. Berger, Adam, Rich Caruana, David Cohn, Dayne Freytag, and Vibhu Mittal. 2000. Bridging the lexical chasm: Statistical approaches to answer finding. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research & Development on Information Retrieval, pages 192–199, Athens, Greece. find
  5. Bilotti, Matthew W., Paul Ogilvie, Jamie Callan, and Eric Nyberg. 2007. Structured retrieval for question answering. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research & Development on Information Retrieval, pages 351–358, Amsterdam. find
  6. Brill, Eric, Jimmy Lin, Michele Banko, Susan Dumais, and Andrew Ng. 2001. Data-intensive question answering. In Proceedings of the Text REtrieval Conference (TREC), pages 393–400, Gaithersburg, MD, USA. find
  7. Brown, Peter F., Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311. find
  8. Charniak, Eugene. 2000. A maximum-entropy-inspired parser. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 132–139, Seattle, WA.
  9. Ciaramita, Massimiliano and Yasemin Altun. 2006. Broad coverage sense disambiguation and information extraction with a supersense sequence tagger. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 594–602, Sidney.
  10. Ciaramita, Massimiliano, Giuseppe Attardi, Felice Dell’Orletta, and Mihai Surdeanu. 2008. Desrl: A linear-time semantic role labeling system. In Proceedings of the Shared Task of the 12th Conference on Computational Natural Language Learning (CoNLL-2008), pages 258–262, Manchester. find
  11. Ciaramita, Massimiliano and Mark Johnson. 2003. Supersense tagging of unknown nouns in WordNet. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, pages 168–175, Sapporo.
  12. Ciaramita, Massimiliano, Vanessa Murdock, and Vassilis Plachouras. 2008. Semantic associations for contextual advertising. Journal of Electronic Commerce Research—Special Issue on Online Advertising and Sponsored Search, 9(1):1–15. find
  13. Collins, Michael and Nigel Duffy. 2001. Convolution kernels for natural language. In Proceedings of the Neural Information Processing Systems Conference (NIPS), pages 625–632, Vancouver, Canada. find
  14. Cui, Hang, Renxu Sun, Keya Li, Min-Yen Kan, and Tat-Seng Chua. 2005. Question answering passage retrieval using dependency relations. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 400–407, Salvador. find
  15. Echihabi, Abdessamad and Daniel Marcu. 2003. A noisy-channel approach to question answering. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL), pages 16–23, Sapporo. find
  16. Freund, Yoav and Robert E. Schapire. 1999. Large margin classification using the perceptron algorithm. Machine Learning, 37:277–296. find
  17. Girju, Roxana. 2003. Automatic detection of causal relations for question answering. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL), Workshop on Multilingual Summarization and Question Answering, pages 76–83, Sapporo.
  18. Harabagiu, Sanda, Dan Moldovan, Marius Pasca, Rada Mihalcea, Mihai Surdeanu, Razvan Bunescu, Roxana Girju, Vasile Rus, and Paul Morarescu. 2000. Falcon: Boosting knowledge for answer engines. In Proceedings of the Text REtrieval Conference (TREC), pages 479–487, Gaithersburg, MD. find
  19. Higashinaka, Ryuichiro and Hideki Isozaki. 2008. Corpus-based question answering for why-questions. In Proceedings of the Third International Joint Conference on Natural Language Processing (IJCNLP), pages 418–425, Hyderabad. find
  20. Jeon, Jiwoon, W. Bruce Croft, and Joon Hoo Lee. 2005. Finding similar questions in large question and answer archives. In Proceedings of the ACM Conference on Information and Knowledge Management (CIKM), pages 84–90, Bremen. find
  21. Jeon, Jiwoon, W. Bruce Croft, Joon Hoo Lee, and Soyeon Park. 2006. A framework to predict the quality of answers with non-textual features. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 228–235, Seattle, WA. find
  22. Joachims, Thorsten. 2006. Training linear svms in linear time. In KDD ’06: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 217–226, New York, NY. find
  23. Ko, Jeongwoo, Teruko Mitamura, and Eric Nyberg. 2007. Language-independent probabilistic answer ranking for question answering. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, pages 784–791, Prague. find
  24. Li, Xin and Dan Roth. 2006. Learning question classifiers: The role of semantic information. Natural Language Engineering, 12:229–249. find
  25. Li, Yaoyong, Hugo Zaragoza, Ralf Herbrich, John Shawe-Taylor, and Jaz S. Kandola. 2002. The perceptron algorithm with uneven margins. In Proceedings of the Nineteenth International Conference on Machine Learning, pages 379–386, Sidney. find
  26. Magnini, Bernardo, Matteo Negri, Roberto Prevete, and Hristo Tanev. 2002. Comparing statistical and content-based techniques for answer validation on the web. In Proceedings of the VIII Convegno AI*IA, Siena, Italy. find
  27. Marcus, Mitchell P., Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330.
  28. Moldovan, Dan, Sanda Harabagiu, Marius Pasca, Rada Mihalcea, Richard Goodrum, Roxana Girju, and Vasile Rus. 1999. Lasso—a tool for surfing the answer net. In Proceedings of the Text REtrieval Conference (TREC), pages 175–183, Gaithersburg, MD. find
  29. Moschitti, Alessandro, Silvia Quarteroni, Roberto Basili, and Suresh Manandhar. 2007. Exploiting syntactic and shallow semantic kernels for question/answer classification. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL), pages 776–783, Prague. find
  30. Murdock, Vanessa and W. Bruce Croft. 2005. A translation model for sentence retrieval. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 684–691, Vancouver. find
  31. Palmer, Martha, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–106
  32. Punyakanok, Vasin, Dan Roth, and Wen-tau Yih. 2004. Mapping dependencies trees: An application to question answering. Proceedings of AI&Math 2004, pages 1–10, Fort Lauderdale, FL. find
  33. Riezler, Stefan, Alexander Vasserman, Ioannis Tsochantaridis, Vibhu Mittal, and Yi Liu. 2007. Statistical machine translation for query expansion in answer retrieval. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL), pages 464–471, Prague.
  34. Robertson, Stephen and Stephen G. Walker. 1997. On relevance weights with little relevance information. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 16–24, New York, NY. find
  35. Shen, Dan and Dietrich Klakow. 2006. Exploring correlation of dependency relation paths for answer extraction. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics, pages 889–896, Sydney. find
  36. Shen, Libin and Aravind K. Joshi. 2005. Ranking and reranking with perceptron. Machine Learning. Special Issue on Learning in Speech and Language Technologies, 60(1):73–96. find
  37. Soricut, Radu and Eric Brill. 2006. Automatic question answering using the Web: Beyond the factoid. Journal of Information Retrieval—Special Issue on Web Information Retrieval, 9(2):191–206. find
  38. Surdeanu, Mihai and Massimiliano Ciaramita. 2007. Robust information extraction with perceptrons. In Proceedings of the NIST 2007 Automatic Content Extraction Workshop (ACE07), College Park, MD. Available at: http://www.surdeanu.name/mihai/papers/ace07a.pdf. ↗
  39. Surdeanu, Mihai, Richard Johansson, Adam Meyers, Lluis Marquez, and Joakim Nivre. 2008. The CoNLL-2008 shared task on joint parsing of syntactic and semantic dependencies. In Proceedings of the Conference on Computational Natural Language Learning (CoNLL), pages 159–177, Manchester.
  40. Surdeanu, Mihai, Lluis Marquez, Xavier Carreras, and Pere R. Comas. 2007. Combination strategies for semantic role labeling. Journal of Artificial Intelligence Research, 29:105–151. find
  41. Tao, Tao and ChengXiang Zhai. 2007. An exploration of proximity measures in information retrieval. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 259–302, Amsterdam. find
  42. Tsochantaridis, Ioannis, Thomas Hofmann, Thorsten Joachims, and Yasemin Altun. 2004. Support vector machine learning for interdependent and structured output spaces. In ICML ’04: Proceedings of the Twenty-First International Conference on Machine Learning, pages 104–111, New York, NY. find
  43. Verberne, Suzan, Lou Boves, Nelleke Oostdijk, and Peter-Arno Coppen. 2010. What is not in the bag of words for why-qa? Computational Linguistics, 36(2):229–245. find
  44. Voorhees, Ellen M. 2001. Overview of the TREC-9 question answering track. In Proceedings of the Text REtrieval Conference (TREC) TREC-9 Proceedings, pages 1–15, Gaithersburg, MD. find
  45. Wang, Mengqiu, Noah A. Smith, and Teruko Mitamura. 2007. What is the Jeopardy model? A quasi-synchronous grammar for QA. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 22–32, Prague.
  46. Xue, Xiaobing, Jiwoon Jeon, and W. Bruce Croft. 2008. Retrieval models for question and answer archives. In Proceedings of the Annual ACM SIGIR Conference on Research and Development in Information Retrieval, pages 475–482, Singapore. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/surdeanu-2011-learning-rank-answers-non-factoid/