Elamur Research⌕
☰
home / papers / 2013 / Semantic Parsing on Freebase from Question-Answer Pairs

Semantic Parsing on Freebase from Question-Answer Pairs

Summary

The authors build a semantic parser for the Freebase knowledge base. It maps natural-language questions to logical forms, and executing these forms against the base yields the answer. The main novelty is that the system is trained only on question–answer pairs. Costly manual annotation of logical forms is not needed: the logical forms remain latent variables. The hardest part is selecting, from more than 19 thousand Freebase properties, the few that are relevant to a given question. Two mechanisms are combined for this. The first is a coarse lexicon: phrases from a large web corpus are aligned with predicates of the base through shared entity pairs. The second is the bridging operation, which selects a predicate that is type-compatible with its neighbors. It helps when the question contains no explicit word for the relation, for example with light verbs or prepositions. Logical forms are written in λ-DCS, and candidates are ranked by a log-linear model with beam search. On Free917 the system outperformed the best parser of the time, even though it never saw a single annotated logical form. The authors also collected a new, more realistic dataset, WebQuestions. The work showed that open-domain semantic parsing is feasible with weak supervision.

Key points

  • Freebase in this work contains 41M entities, 19K properties, and 596M assertions.
  • The lexicon is built on ~15M ReVerb triples from ClueWeb09. This is a subset of Lin et al. (2012) in which entities are already linked to the base; dates are normalized with SUTime. To distinguish the senses of ambiguous phrases, each has a type signature, for example [Person, Location].
  • On Free917 the accuracy is 62%, versus 59% for the system of Cai and Yates (2013), which was trained on annotated logical forms.
  • The questions for WebQuestions were collected by a breadth-first crawl via the Google Suggest API, and the answers were provided by Amazon Mechanical Turk workers. The accuracy here is 31.4%, which is 4.5 percentage points better than a natural baseline system.
  • The code of the SEMPRE parser and both datasets have been publicly released. Error analysis revealed three main causes: entities are hard to disambiguate among 41M candidates; bridging fails when an entity is compatible with many properties; the system sometimes takes the verb from a subordinate clause.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics: Knowledge graphs & ontologies · Semantic parsing & meaning representation

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{berant2013semantic,
  title = {{Semantic Parsing on Freebase from Question-Answer Pairs}},
  author = {Berant, Jonathan and Chou, Andrew and Frostig, Roy and Liang, Percy},
  booktitle = {Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
  url = {https://aclanthology.org/D13-1160/}
}

References

43 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Artzi and L. Zettlemoyer. 2011. Bootstrapping semantic parsers from conversations. In Empirical Methods in Natural Language Processing (EMNLP), pages 421–432. find
  2. Y. Artzi and L. Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics (TACL), 1:49–62. find our card
  3. M. Banko, M. J. Cafarella, S. Soderland, M. Broadhead, and O. Etzioni. 2007. Open information extraction from the web. In International Joint Conference on Artificial Intelligence (IJCAI), pages 2670–2676. find our card
  4. S. Branavan, L. Zettlemoyer, and R. Barzilay. 2010. Reading between the lines: Learning to map high-level instructions to commands. In Association for Computational Linguistics (ACL), pages 1268–1277. find
  5. S. Branavan, D. Silver, and R. Barzilay. 2011. Learning to win by reading manuals in a Monte-Carlo framework. In Association for Computational Linguistics (ACL), pages 268–277. find
  6. S. Branavan, N. Kushman, T. Lei, and R. Barzilay. 2012. Learning high-level planning from text. In Association for Computational Linguistics (ACL), pages 126–135. find
  7. Q. Cai and A. Yates. 2013. Large-scale semantic parsing via schema matching and lexicon extension. In Association for Computational Linguistics (ACL). find
  8. A. Carlson, J. Betteridge, B. Kisiel, B. Settles, E. R. H. Jr, and T. M. Mitchell. 2010. Toward an architecture for never-ending language learning. In Association for the Advancement of Artificial Intelligence (AAAI). find
  9. A. X. Chang and C. Manning. 2012. SUTime: A library for recognizing and normalizing time expressions. In Language Resources and Evaluation (LREC), pages 3735–3740. find
  10. D. Chen. 2012. Fast online lexicon learning for grounded language acquisition. In Association for Computational Linguistics (ACL). find
  11. H. H. Clark. 1975. Bridging. In Workshop on theoretical issues in natural language processing, pages 169–174. find
  12. J. Clarke, D. Goldwasser, M. Chang, and D. Roth. 2010. Driving semantic parsing from the world’s response. In Computational Natural Language Learning (CoNLL), pages 18–27. find
  13. J. Duchi, E. Hazan, and Y. Singer. 2010. Adaptive subgradient methods for online learning and stochastic optimization. In Conference on Learning Theory (COLT). find
  14. A. Fader, S. Soderland, and O. Etzioni. 2011. Identifying relations for open information extraction. In Empirical Methods in Natural Language Processing (EMNLP). find
  15. A. Fader, L. Zettlemoyer, and O. Etzioni. 2013. Paraphrase-driven learning for open question answering. In Association for Computational Linguistics (ACL). find our card
  16. D. Goldwasser, R. Reichart, J. Clarke, and D. Roth. 2011. Confidence driven unsupervised semantic parsing. In Association for Computational Linguistics (ACL), pages 1486–1495. find
  17. Google. 2013. Freebase data dumps (2013-06-09). https://developers.google.com/freebase/data. ↗
  18. M. A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In Interational Conference on Computational linguistics, pages 539–545. find
  19. R. Hoffmann, C. Zhang, X. Ling, L. S. Zettlemoyer, and D. S. Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In Association for Computational Linguistics (ACL), pages 541–550. find
  20. J. Krishnamurthy and T. Kollar. 2013. Jointly learning to parse and perceive: Connecting natural language to the physical world. Transactions of the Association for Computational Linguistics (TACL), 1:193–206. find
  21. J. Krishnamurthy and T. Mitchell. 2012. Weakly supervised training of semantic parsers. In Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP/CoNLL), pages 754–765. find
  22. N. Kushman and R. Barzilay. 2013. Using semantic unification to generate regular expressions from natural language. In Human Language Technology and North American Association for Computational Linguistics (HLT/NAACL), pages 826–836. find
  23. T. Kwiatkowski, L. Zettlemoyer, S. Goldwater, and M. Steedman. 2010. Inducing probabilistic CCG grammars from logical form with higher-order unification. In Empirical Methods in Natural Language Processing (EMNLP), pages 1223–1233. find
  24. T. Kwiatkowski, L. Zettlemoyer, S. Goldwater, and M. Steedman. 2011. Lexical generalization in CCG grammar induction for semantic parsing. In Empirical Methods in Natural Language Processing (EMNLP), pages 1512–1523. find our card
  25. N. Lao, T. Mitchell, and W. W. Cohen. 2011. Random walk inference and learning in a large scale knowledge base. In Empirical Methods in Natural Language Processing (EMNLP). find
  26. T. Lei, F. Long, R. Barzilay, and M. Rinard. 2013. From natural language specifications to program input parsers. In Association for Computational Linguistics (ACL). find
  27. P. Liang, M. I. Jordan, and D. Klein. 2011. Learning dependency-based compositional semantics. In Association for Computational Linguistics (ACL), pages 590–599. find
  28. P. Liang. 2013. Lambda dependency-based compositional semantics. Technical report, ArXiv. find
  29. T. Lin, Mausam, and O. Etzioni. 2012. Entity linking at web scale. In Knowledge Extraction Workshop (AKBC-WEKEX). find
  30. Masaum, M. Schmitz, R. Bart, S. Soderland, and O. Etzioni. 2012. Open language learning for information extraction. In Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP/CoNLL), pages 523–534. find
  31. C. Matuszek, N. FitzGerald, L. Zettlemoyer, L. Bo, and D. Fox. 2012. A joint model of language and perception for grounded attribute learning. In International Conference on Machine Learning (ICML). find
  32. R. McDonald, K. Crammer, and F. Pereira. 2005. Online large-margin training of dependency parsers. In Association for Computational Linguistics (ACL), pages 91–98. find our card
  33. H. Poon. 2013. Grounded unsupervised semantic parsing. In Association for Computational Linguistics (ACL). find
  34. A. Popescu, O. Etzioni, and H. Kautz. 2003. Towards a theory of natural language interfaces to databases. In International Conference on Intelligent User Interfaces (IUI), pages 149–157. find
  35. S. Riedel, L. Yao, and A. McCallum. 2010. Modeling relations and their mentions without labeled text. In Machine Learning and Knowledge Discovery in Databases (ECML PKDD), pages 148–163. find
  36. M. Surdeanu, J. Tibshirani, R. Nallapati, and C. D. Manning. 2012. Multi-instance multi-label learning for relation extraction. In Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP/CoNLL), pages 455– 465. find
  37. S. Tellex, T. Kollar, S. Dickerson, M. R. Walter, A. G. Banerjee, S. J. Teller, and N. Roy. 2011. Understanding natural language commands for robotic navigation and mobile manipulation. In Association for the Advancement of Artificial Intelligence (AAAI). find
  38. C. Unger, L. Bhmann, J. Lehmann, A. Ngonga, D. Gerber, and P. Cimiano. 2012. Template-based question answering over RDF data. In World Wide Web (WWW), pages 639–648. find
  39. Y. W. Wong and R. J. Mooney. 2007. Learning synchronous grammars for semantic parsing with lambda calculus. In Association for Computational Linguistics (ACL), pages 960–967. find our card
  40. M. Yahya, K. Berberich, S. Elbassuoni, M. Ramanath, V. Tresp, and G. Weikum. 2012. Natural language questions for the web of data. In Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP/CoNLL), pages 379–390. find
  41. M. Zelle and R. J. Mooney. 1996. Learning to parse database queries using inductive logic proramming. In Association for the Advancement of Artificial Intelligence (AAAI), pages 1050–1055. find
  42. L. S. Zettlemoyer and M. Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In Uncertainty in Artificial Intelligence (UAI), pages 658–666. find our card
  43. C. Zhang, R. Hoffmann, and D. S. Weld. 2012. Ontological smoothing for relation extraction with minimal supervision. In Association for the Advancement of Artificial Intelligence (AAAI). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/