Elamur Research⌕
☰
home / papers / 2013 / Paraphrase-Driven Learning for Open Question Answering

Paraphrase-Driven Learning for Open Question Answering

Summary

The paper tackles open-domain question answering over a noisy knowledge base built automatically by information extraction from web text. The authors present PARALEX, which converts a natural-language question into a single-relation query of the form r(?, e) or r(e, ?) and runs it against a large tuple store. The key point is that no questions are labeled by hand. Instead, the system learns from WikiAnswers paraphrase clusters, where all questions in a cluster share an answer whose query is never stated.

Training starts from a small set of seed question templates combined with string matching. Word alignments then broadly expand this seed into a large lexicon of entity, relation and question patterns. A linear ranking model, trained with a scalable parallelized perceptron, keeps the useful entries and discards the rest.

On known-answerable questions, recall rises more than fourfold compared with the seed lexicon, while precision drops only slightly. This shows that cheap, noisy community paraphrase data can replace annotated question–query pairs at web scale.

Key points

  • Training data: about 18 million paraphrase pairs from WikiAnswers. The answer database holds roughly 15 million facts extracted from web text with ReVerb (Fader et al., 2011).
  • Supervision is limited to 16 seed question templates plus string matching. These automatically assign queries to a small portion of the corpus.
  • Results on held-out known-answerable questions: 42% recall at 77% precision. This is over four times the seed baseline's recall, at a cost of about 8% precision.
  • Performance on unrestricted WikiAnswers questions is much weaker, with recall peaking near 6% at precision 0.4.
  • An error analysis of 100 failed questions breaks down as follows:
    • 36% were too complex for a single-relation query (n-ary relations, how-to answers, joins).
    • 32% involved entity or relation recognition failures.
    • 28% produced a correct query that found nothing in the database.
    • 4% contained typos or could not be understood.

Interesting ideas

  • I1. Paraphrase clusters as indirect supervision. Questions grouped as paraphrases presumably share one meaning. A query that is reliably found for one member can therefore serve as a label for the others. This trick could bootstrap semantic parsers in languages or domains that lack annotated logical forms but have forums or FAQ-style duplicate-question data.
  • I2. Over-generate, then filter with a ranker. Lexicon expansion via word alignment is deliberately permissive. Precision is restored afterwards by a learned linear ranking function. This keeps recall high during induction and moves noise control to a single, trainable component.
  • I3. Factored lexical entries. Question templates are kept separate from entity strings and relation strings. One learned synonym (e.g., "big" → population) can then be reused across many unseen question phrasings. This modular design suits any pattern-based extraction or QA system with sparse data.
  • I4. Error analysis that separates fault sources. The authors check whether a failed question was representable at all, whether a correct query was derived, and whether the database simply lacked the fact. Such an audit clarifies whether to invest in better parsing, richer query languages, or knowledge-base coverage.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

120 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 113 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{fader2013paraphrase,
  title = {{Paraphrase-Driven Learning for Open Question Answering}},
  author = {Fader, Anthony and Zettlemoyer, Luke and Etzioni, Oren},
  booktitle = {Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2013},
  url = {https://aclanthology.org/P13-1158/}
}

References

31 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michele Banko, Michael J. Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. 2007. Open Information Extraction from the Web. In Proceedings of the 20th international joint conference on Artifical intelligence. find our card
  2. Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with Bilingual Parallel Corpora. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics. find our card
  3. Regina Barzilay and Lillian Lee. 2003. Learning to Paraphrase: An Unsupervised Approach Using Multiple-Sequence Alignment. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics. ↗ our card
  4. Regina Barzilay and Kathleen R. McKeown. 2001. Extracting Paraphrases from a Parallel Corpus. In Proceedings of the 39th Annual Meeting on Association for Computational Linguistics. find our card
  5. Eric Brill, Susan Dumais, and Michele Banko. 2002. An Analysis of the AskMSR Question-Answering System. In Proceedings of Empirical Methods in Natural Language Processing. find
  6. Qingqing Cai and Alexander Yates. 2013. Large-scale Semantic Parsing via Schema Matching and Lexicon Extension. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
  7. Chris Callison-Burch. 2008. Syntactic Constraints on Paraphrases Extracted from Parallel Corpora. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. find our card
  8. James Clarke, Dan Goldwasser, Ming-Wei Chang, and Dan Roth. 2010. Driving Semantic Parsing from the World’s Response. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning. find
  9. Michael Collins. 2002. Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find our card
  10. Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying Relations for Open Information Extraction. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. find
  11. Qin Gao and Stephan Vogel. 2008. Parallel Implementations of Word Alignment Tool. In Proc. of the ACL 2008 Software Engineering, Testing, and Quality Assurance Workshop. find
  12. Barbara J. Grosz, Douglas E. Appelt, Paul A. Martin, and Fernando C. N. Pereira. 1987. TEAM: An Experiment in the Design of Transportable Natural-Language Interfaces. Artificial Intelligence, 32(2):173–243. ↗
  13. Raphael Hoffmann, Congle Zhang, and Daniel S. Weld. 2010. Learning 5000 relational extractors. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. find
  14. Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S. Weld. 2011. Knowledge-Based Weak Supervision for Information Extraction of Overlapping Relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics. find
  15. Boris Katz. 1997. Annotating the World Wide Web using Natural Language. In RIAO, pages 136–159. find
  16. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical Phrase-Based Translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics. ↗
  17. Cody Kwok, Oren Etzioni, and Daniel S. Weld. 2001. Scaling Question Answering to the Web. ACM Trans. Inf. Syst., 19(3):242–262. ↗
  18. Percy Liang, Michael Jordan, and Dan Klein. 2011. Learning Dependency-Based Compositional Semantics. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics. ↗
  19. Yuval Marton, Chris Callison-Burch, and Philip Resnik. 2009. Improved Statistical Machine Translation Using Monolingually-Derived Paraphrases. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing. find our card
  20. Ryan McDonald, Keith Hall, and Gideon Mann. 2010. Distributed training strategies for the structured perceptron. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. find
  21. Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant Supervision for Relation Extraction Without Labeled Data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL. find
  22. Franz Josef Och and Hermann Ney. 2000. Improved Statistical Alignment Models. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics. find our card
  23. Ana-Maria Popescu, Alex Armanasu, Oren Etzioni, David Ko, and Alexander Yates. 2004. Modern Natural Language Interfaces to Databases: Composing Statistical Parsing with Semantic Tractability. In Proceedings of the Twentieth International Conference on Computational Linguistics. find
  24. Chris Quirk, Chris Brockett, and William Dolan. 2004. Monolingual Machine Translation for Paraphrase Generation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. find our card
  25. Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling Relations and Their Mentions without Labeled Text. In Proceedings of the 2010 European conference on Machine learning and Knowledge Discovery in Databases. find
  26. Lappoon R. Tang and Raymond J. Mooney. 2001. Using Multiple Clause Constructors in Inductive Logic Programming for Semantic Parsing. find
  27. Christina Unger, Lorenz Bühmann, Jens Lehmann, Axel-Cyrille Ngonga Ngomo, Daniel Gerber, and Philipp Cimiano. 2012. Template-Based Question Answering over RDF Data. In Proceedings of the 21st World Wide Web Conference 2012. find
  28. Mohamed Yahya, Klaus Berberich, Shady Elbassuoni, Maya Ramanath, Volker Tresp, and Gerhard Weikum. 2012. Natural Language Questions for the Web of Data. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. find
  29. Limin Yao, Sebastian Riedel, and Andrew McCallum. 2012. Unsupervised Relation Discovery with Sense Disambiguation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics. find
  30. John M. Zelle and Raymond J. Mooney. 1996. Learning to Parse Database Queries Using Inductive Logic Programming. In Proceedings of the Thirteenth National Conference on Artificial Intelligence. find
  31. Luke S. Zettlemoyer and Michael Collins. 2005. Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars. In Proceedings of the 21st Conference in Uncertainty in Artificial Intelligence. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fader-2013-paraphrase-driven-learning-open-question/