Elamur Research⌕
☰
home / papers / 2016 / Question Answering on Freebase via Relation Extraction and Textual Evidence

Question Answering on Freebase via Relation Extraction and Textual Evidence

Summary

The paper addresses question answering over Freebase (KB-QA). It targets a known weakness of relation-extraction approaches: they cope well with limited training data but return too many answers when a question carries extra constraints. Examples are superlatives such as "highest" or words with hidden conditions, like "mother" meaning parent plus female.

The proposed system works in two stages:

  1. Freebase stage. It finds candidate entities with S-MART and predicts the KB relation with a Multi-Channel Convolutional Neural Network (MCCNN) that reads both the dependency path and the surrounding words. Entity linking and relation prediction are then resolved jointly, producing a candidate answer set.
  2. Wikipedia stage. A refinement model examines sentences from the topic entity's Wikipedia page and keeps only the candidates that the text supports.

Questions with several relations are first split into sub-questions using dependency patterns, and the answer sets of the sub-questions are intersected. On WebQuestions the full system reaches an F1 of 53.3%, which was state of the art. This shows that unstructured text can compensate for the limited expressiveness of shallow structured inference.

Key points

  • Evaluated on WebQuestions with a final F1 of 53.3%. Even without the Wikipedia step, the system beats the MCCNN-based approach of Dong et al. (2015) by about 7 points.
  • Entity linking:
    • Hand-written POS-tag patterns propose mention spans, and S-MART supplies the top 5 Freebase entities for each span.
    • About 80% of linking errors involve the wrong entity being chosen even though the mention span was correct.
  • The MCCNN has two input channels whose max-pooled outputs are concatenated and fed into a softmax over Freebase relations. Training uses cross-entropy with L2 regularization and SGD with AdaGrad.
    • Syntactic channel: the shortest dependency path from the Stanford CoreNLP parser, with the entity mention and question word removed.
    • Sentential channel: the remaining words of the question.
  • Question decomposition:
    • About 15% of dev questions (136) are compositional and yield 292 sub-questions.
    • The manual decomposition rules were judged about 95% accurate on QALD-5.
  • Code, data and outputs are released at github.com/syxu828/QuestionAnsweringOverFB. The method is limited by Freebase coverage and still fails on nested questions (e.g., a mother of a father).

Interesting ideas

  • I1. Generate from the KB, then verify against text. Using a recall-oriented structured step followed by a text-based filter avoids explicitly modeling hard operations like superlatives or gender constraints. The same pattern could work for any KB-backed QA system that has a parallel document collection, such as encyclopedia articles in a low-resource language.
  • I2. Predict relations instead of ranking entities. Framing QA as classification over a fixed set of relations keeps the search space much smaller than scoring every possible answer entity. This makes the model cheaper to run and easier to train when annotated data is scarce.
  • I3. Strip the endpoints before encoding the path. Removing the entity mention and question word from the dependency path pushes the model toward a general representation of the relation rather than memorizing specific names. This is a simple trick reusable in other relation-extraction tasks.
  • I4. Joint decisions to limit error propagation. Choosing the entity and relation together, rather than fixing the entity first, helps because a plausible relation can rescue an ambiguous entity link. The error analysis suggests that entity choice, not span detection, is where pipelines break.
  • I5. Rule-based decomposition with answer intersection. A few dependency patterns that split multi-constraint questions into simple ones, combined by intersecting their answers, offer a cheap and transparent way to handle compositionality. Checking these rules on another dataset (QALD-5) is a good habit for testing whether hand-made rules transfer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

78 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 76 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1220

@inproceedings{xu2016question,
  title = {{Question Answering on Freebase via Relation Extraction and Textual Evidence}},
  author = {Xu, Kun and Reddy, Siva and Feng, Yansong and Huang, Songfang and Zhao, Dongyan},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1220},
  url = {https://doi.org/10.18653/v1/p16-1220}
}

References

56 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sren Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary G. Ives. 2007. Dbpedia: A nucleus for a web of open data. In ISWC/ASWC. find
  2. Michele Banko, Michael J Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. 2007. Open information extraction for the web. In IJCAI. find
  3. Junwei Bao, Nan Duan, Ming Zhou, and Tiejun Zhao. 2014. Knowledge-based question answering as machine translation. In ACL. find
  4. Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In ACL. find
  5. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems. find our card
  6. Jonathan Berant and Percy Liang. 2015. Imitation learning of agenda-based semantic parsers. Transactions of the Association for Computational Linguistics, 3:545–558. find
  7. Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In ACL. find
  8. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In EMNLP. find
  9. Thorsten Joachims. 2006. Training linear svms in linear time. In SIGKDD. find
  10. Hannah Bast and Elmar Haussmann. 2015. More accurate question answering on freebase. In CIKM. find
  11. Kurt D. Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In SIGMOD. find
  12. Antoine Bordes, Sumit Chopra, and Jason Weston. 2014. Question answering with subgraph embeddings. In EMNLP. find
  13. Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. CoRR, abs/1506.02075. find
  14. Qingqing Cai and Alexander Yates. 2013. Large-scale semantic parsing via schema matching and lexicon extension. In ACL. find
  15. Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R Hruschka Jr, and Tom M Mitchell. 2010. Toward an architecture for never-ending language learning. In AAAI. find
  16. Chih-Chung Chang and Chih-Jen Lin. 2011. LIB-SVM: A library for support vector machines. ACM TIST, 2(3):27. find
  17. Mandar Joshi, Uma Sawant, and Soumen Chakrabarti. 2014. Knowledge graph and corpus driven segmentation and answer inference for telegraphic entity-seeking queries. In EMNLP. find
  18. Jayant Krishnamurthy and Tom M Mitchell. 2012. Weakly supervised training of semantic parsers. In EMNLP-CoNLL. find
  19. Tom Kwiatkowski, Eunsol Choi, Yoav Artzi, and Luke S. Zettlemoyer. 2013. Scaling semantic parsers with on-the-fly ontology matching. In EMNLP. find
  20. Yang Liu, Furu Wei, Sujian Li, Heng Ji, Ming Zhou, and Houfeng WANG. 2015. A dependency-based neural network for relation classification. In ACL. find
  21. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In ACL System Demonstrations. find our card
  22. Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In ACL. find
  23. Xiao Cheng and Dan Roth. 2013. Relational inference for wikification. In ACL. find
  24. Ankur P. Parikh, Hoifung Poon, and Kristina Toutanova. 2015. Grounded semantic parsing for complex knowledge extraction. In NAACL. find
  25. Li Dong, Furu Wei, Ming Zhou, and Ke Xu. 2015. Question answering over freebase with multi-column convolutional neural networks. In ACL-IJCNLP. find
  26. Siva Reddy, Mirella Lapata, and Mark Steedman. 2014. Large-scale semantic parsing without question-answer pairs. Transactions of the Association of Computational Linguistics, pages 377–392. find
  27. John C. Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121–2159. find
  28. Siva Reddy, Oscar Täckström, Michael Collins, Tom Kwiatkowski, Dipanjan Das, Mark Steedman, and Mirella Lapata. 2016. Transforming Dependency Structures to Logical Forms for Semantic Parsing. Transactions of the Association for Computational Linguistics, 4. find
  29. Anthony Fader, Luke S. Zettlemoyer, and Oren Etzioni. 2013. Paraphrase-driven learning for open question answering. In ACL. find our card
  30. Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2014. Open question answering over curated and extracted knowledge bases. In SIGKDD. find
  31. Michael Heilman and Noah A Smith. 2010. Tree edit models for recognizing textual entailments, paraphrases, and answers to questions. In NAACL. find our card
  32. Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013. Relation extraction with matrix factorization and universal schemas. In NAACL. find
  33. Denis Savenkov and Eugene Agichtein. 2016. When a knowledge base is not enough: Question answering over knowledge bases with external text data. In SIGIR. find
  34. Iulian Vlad Serban, Alberto Garcı́a-Durán, Çaglar Gülçehre, Sungjin Ahn, Sarath Chandar, Aaron C. Courville, and Yoshua Bengio. 2016. Generating factoid questions with recurrent neural networks: The 30m factoid question-answer corpus. In ACL. find
  35. Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: a core of semantic knowledge. In WWW. find
  36. Huan Sun, Hao Ma, Wen-tau Yih, Chen-Tse Tsai, Jingjing Liu, and Ming-Wei Chang. 2015. Open domain question answering via semantic enrichment. In WWW. find
  37. Joseph P. Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In ACL 2010, Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, July 11-16, 2010, Uppsala, Sweden, pages 384–394. find our card
  38. Christina Unger, Lorenz Bühmann, Jens Lehmann, Axel-Cyrille Ngonga Ngomo, Daniel Gerber, and Philipp Cimiano. 2012. Template-based question answering over rdf data. In WWW. find
  39. Ellen M Voorhees and Dawn M. Tice. 1999. The trec-8 question answering track report. In TREC. find our card
  40. Mengqiu Wang, Noah A Smith, and Teruko Mitamura. 2007. What is the jeopardy model? a quasi-synchronous grammar for qa. In EMNLP-CoNLL. find our card
  41. Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight. In ACL. find
  42. Xuchen Yao and Benjamin Van Durme. 2014. Information extraction over structured data: Question answering with freebase. In ACL. find
  43. Xuchen Yao, Benjamin Van Durme, and Peter Clark. 2013. Answer extraction as sequence tagging with tree edit distance. In NAACL. find
  44. Xuchen Yao. 2015. Lean question answering over freebase from scratch. In NAACL. find
  45. Wen-tau Yih, Ming-Wei Chang, Christopher Meek, and Andrzej Pastusiak. 2013. Question answering using enhanced lexical semantic models. In ACL. find
  46. Wen-tau Yih, Xiaodong He, and Christopher Meek. 2014. Semantic parsing for single-relation question answering. In ACL. find our card
  47. Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In ACL-IJCNLP. find
  48. Lei Yu, Karl Moritz Hermann, Phil Blunsom, and Stephen Pulman. 2014. Deep learning for answer sentence selection. arXiv preprint arXiv:1412.1632. — [arXiv:1412.1632](https://arxiv.org/abs/1412.1632) find
  49. John M Zelle and Raymond J Mooney. 1996. Learning to parse database queries using inductive logic programming. In AAAI. find
  50. Yuanzhe Zhang, Shizhu He, Kang Liu, and Jun Zhao. 2016. A joint model for question answering over multiple knowledge bases. In AAAI. find
  51. Kun Xu, Sheng Zhang, Yansong Feng, and Dongyan Zhao. 2014. Answering natural language questions via phrasal semantic parsing. In Natural Language Processing and Chinese Computing - Third CCF Conference, NLPCC 2014, Shenzhen, China, December 5-9, 2014. Proceedings, pages 333–344. find
  52. Kun Xu, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2015. Semantic relation classification via convolutional neural networks with simple negative sampling. In EMNLP. find
  53. Mohamed Yahya, Klaus Berberich, Shady Elbassuoni, Maya Ramanath, Volker Tresp, and Gerhard Weikum. 2012. Natural language questions for the web of data. In EMNLP. find
  54. Mohamed Yahya, Denilson Barbosa, Klaus Berberich, Qiuyue Wang, and Gerhard Weikum. 2016. Relationship queries on extended knowledge graphs. In WSDM. find
  55. Yi Yang and Ming-Wei Chang. 2015. S-mart: Novel tree-based structured learning algorithms applied to tweet entity linking. In ACL-IJNLP. find
  56. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain question answering. In EMNLP. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2016-question-answering-freebase-relation-extraction/