Elamur Research
☰
home / papers / 2013 / Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions

Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/Q13-1005.pdf

Summary

The authors train a system to map natural-language navigation instructions to sequences of agent actions on a map. No annotated logical forms are available for training. The method is built on a weighted CCG grammar with a factored lexicon. Sentence meanings are written in λ-calculus, and this notation is adapted to the spatial language of instructions: references to objects, implicit steps, and constraints such as "twice". The key idea is that the model parses a sentence and executes it in the environment at the same time. As a result, context, such as the objects visible to the agent, directly influences which interpretation is chosen. Instead of annotations, learning relies on a validation function. It checks either the entire path followed by a human or only the final position. To this end, the authors adapted a loss-sensitive Perceptron, and the lexicon is expanded via GENLEX with coarse-to-fine inference. On the SAIL corpus, the system fully executes 60% more instruction sets than the best previous approach. The work shows that a weak signal from the environment can replace costly manual semantic annotation.

Key points

  • The agent has four actions: LEFT, RIGHT, MOVE, NULL. A state is defined by grid coordinates and orientation, and the maps in the experiments contain 141 states on average.
  • The initial lexicon was built manually from 12 instruction sequences, yielding 141 lexical entries.
  • On SAIL (cross-validation): individual sentences are executed correctly in 65.28% of cases, and complete sequences in 31.93%. Kim & Mooney (2012) reported 57.22% and 20.17%, respectively. When only the final state is validated, the results barely drop: 64.25% and 30.9%.
  • On the filtered oracle corpus (test set, trajectory validation), accuracy is 78.63% for sentences and 58.05% for sequences. Logical form (LF) accuracy is 51.05%; it was measured on 105 sentences manually annotated with logical forms.
  • Without implicit actions and without joint execution during parsing (ablations), performance drops, especially on complete sequences.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

186 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 175 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00209

@misc{artzi2013weakly,
  title = {{Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions}},
  author = {Artzi, Yoav and Zettlemoyer, Luke},
  year = {2013},
  doi = {10.1162/tacl\_a\_00209},
  url = {https://doi.org/10.1162/tacl_a_00209}
}

References

51 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Artzi, Y., & Zettlemoyer, L. (2011). Bootstrapping Semantic Parsers from Conversations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  2. Barwise, J., & Cooper, R. (1981). Generalized Quantifiers and Natural Language. Linguistics and Philosophy, 4(2), 159–219. find
  3. Branavan, S., Chen, H., Zettlemoyer, L., & Barzilay, R. (2009). Reinforcement learning for mapping instructions to actions. In Proceedings of the Joint Conference of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing. find
  4. Branavan, S., Zettlemoyer, L., & Barzilay, R. (2010). Reading between the lines: learning to map high-level instructions to commands. In Proceedings of the Conference of the Association for Computational Linguistics. find
  5. Bugmann, G., Klein, E., Lauria, S., & Kyriacou, T. (2004). Corpus-based robotics: A route instruction example. In Proceedings of Intelligent Autonomous Systems. find
  6. Carpenter, B. (1997). Type-Logical Semantics. The MIT Press. find
  7. Chen, D. L. (2012). Fast Online Lexicon Learning for Grounded Language Acquisition. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
  8. Chen, D., Kim, J., & Mooney, R. (2010). Training a multilingual sportscaster: using perceptual context to learn language. Journal of Artificial Intelligence Research, 37(1), 397–436. find
  9. Chen, D., & Mooney, R. (2011). Learning to Interpret Natural Language Navigation Instructions from Observations. In Proceedings of the National Conference on Artificial Intelligence. find
  10. Clark, S., & Curran, J. (2007). Wide-coverage efficient statistical parsing with CCG and log-linear models. Computational Linguistics, 33(4), 493–552. find our card
  11. Clarke, J., Goldwasser, D., Chang, M., & Roth, D. (2010). Driving Semantic Parsing from the World’s Response. In Proceedings of the Conference on Computational Natural Language Learning. find
  12. Collins, M. (2004). Parameter estimation for statistical parsing models: Theory and practice of distribution-free methods. In New Developments in Parsing Technology. find
  13. Davidson, D. (1967). The logical form of action sentences. Essays on actions and events, 105–148. find
  14. Di Eugenio, B., & White, M. (1992). On the Interpretation of Natural Language Instructions. In Proceedings of the Conference of the Association of Computational Linguistics. find
  15. Dzifcak, J., Scheutz, M., Baral, C., & Schermerhorn, P. (2009). What to Do and How to Do It: Translating Natural Language Directives Into Temporal and Dynamic Logic Representation for Goal Management and Action Execution. In Proceedings of the IEEE International Conference on Robotics and Automation… find
  16. Goldwasser, D., Reichart, R., Clarke, J., & Roth, D. (2011). Confidence Driven Unsupervised Semantic Parsing. In Proceedings of the Association of Computational Linguistics. find
  17. Goldwasser, D., & Roth, D. (2011). Learning from Natural Instructions. In Proceedings of the International Joint Conference on Artificial Intelligence. find
  18. Heim, I., & Kratzer, A. (1998). Semantics in Generative Grammar. Blackwell Oxford. find
  19. Kate, R., & Mooney, R. (2006). Using String-Kernels for Learning Semantic Parsers. In Proceedings of the Conference of the Association for Computational Linguistics. find
  20. Kim, J., & Mooney, R. J. (2012). Unsupervised PCFG Induction for Grounded Language Learning with Highly Ambiguous Supervision. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  21. Kollar, T., Tellex, S., Roy, D., & Roy, N. (2010). Toward Understanding Natural Language Directions. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction. find
  22. Krishnamurthy, J., & Mitchell, T. (2012). Weakly Supervised Training of Semantic Parsers. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. find
  23. Kwiatkowski, T., Goldwater, S., Zettlemoyer, L., & Steedman, M. (2012). A probabilistic model of syntactic and semantic acquisition from child-directed utterances and their meanings. Proceedings of the Conference of the European Chapter of the Association of Computational Linguistics. find
  24. Kwiatkowski, T., Zettlemoyer, L., Goldwater, S., & Steedman, M. (2010). Inducing probabilistic CCG grammars from logical form with higher-order unification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  25. Muresan, S. (2011). Learning for Deep Language Understanding. In Proceedings of the International Joint Conference on Artificial Intelligence. find
  26. Kwiatkowski, T., Zettlemoyer, L., Goldwater, S., & Steedman, M. (2011). Lexical Generalization in CCG Grammar Induction for Semantic Parsing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  27. Parsons, T. (1990). Events in the Semantics of English. The MIT Press. find
  28. Lafferty, J., McCallum, A., & Pereira, F. (2001). Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In Proceedings of the International Conference on Machine Learning. find
  29. Lewis, D. (1970). General Semantics. Synthese, 22(1), 18–67. find
  30. Liang, P., Jordan, M., & Klein, D. (2009). Learning semantic correspondences with less supervision. In Proceedings of the Joint Conference of the Association for Computational Linguistics the International Joint Conference on Natural Language Processing. find
  31. Liang, P., Jordan, M., & Klein, D. (2011). Learning Dependency-Based Compositional Semantics. In Proceedings of the Conference of the Association for Computational Linguistics. find
  32. MacMahon, M. (2007). Following Natural Language Route Instructions. Ph.D. thesis, University of Texas at Austin. find
  33. MacMahon, M., Stankiewics, B., & Kuipers, B. (2006). Walk the Talk: Connecting Language, Knowledge, Action in Route Instructions. In Proceedings of the National Conference on Artificial Intelligence. find
  34. Matuszek, C., FitzGerald, N., Zettlemoyer, L., Bo, L., & Fox, D. (2012). A Joint Model of Language and Perception for Grounded Attribute Learning. Proceedings of the International Conference on Machine Learning. find
  35. Matuszek, C., Fox, D., & Koscher, K. (2010). Following directions using statistical machine translation. In Proceedings of the international conference on Human-robot interaction. find
  36. Matuszek, C., Herbst, E., Zettlemoyer, L. S., & Fox, D. (2012). Learning to Parse Natural Language Commands to a Robot Control System. In Proceedings of the International Symposium on Experimental Robotics. find
  37. Montague, R. (1973). The Proper Treatment of Quantification in Ordinary English. Approaches to natural language, 49, 221–242. find
  38. Singh-Miller, N., & Collins, M. (2007). Trigger-based language modeling using a loss-sensitive perceptron algorithm. In IEEE International Conference on Acoustics, Speech and Signal Processing. find
  39. Steedman, M. (1996). Surface Structure and Interpretation. The MIT Press. find
  40. Steedman, M. (2000). The Syntactic Process. The MIT Press. find
  41. Steedman, M. (2011). Taking Scope. The MIT Press. find
  42. Taskar, B., Guestrin, C., & Koller, D. (2003). Max-Margin Markov Networks. In Proceedings of the Conference on Neural Information Processing Systems. find
  43. Taskar, B., Klein, D., Collins, M., Koller, D., & Manning, C. (2004). Max-Margin Parsing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  44. Tellex, S., Kollar, T., Dickerson, S., Walter, M., Banerjee, A., Teller, S., & Roy, N. (2011). Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation. In Proceedings of the National Conference on Artificial Intelligence. find
  45. Vogel, A., & Jurafsky, D. (2010). Learning to follow navigational directions. In Proceedings of the Conference of the Association for Computational Linguistics. find
  46. Webber, B., Badler, N., Di Eugenio, B., Geib, C., Lev-ison, L., & Moore, M. (1995). Instructions, Intentions and Expectations. Artificial Intelligence, 73(1), 253–269. find
  47. Wei, Y., Brunskill, E., Kollar, T., & Roy, N. (2009). Where To Go: Interpreting Natural Directions Using Global Inference. In Proceedings of the IEEE International Conference on Robotics and Automation. find
  48. Winograd, T. (1972). Understanding Natural Language. Cognitive Psychology, 3(1), 1–191. find
  49. Wong, Y., & Mooney, R. (2007). Learning Synchronous Grammars for Semantic Parsing with Lambda Calculus. In Proceedings of the Conference of the Association for Computational Linguistics. find
  50. Zettlemoyer, L., & Collins, M. (2005). Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In Proceedings of the Conference on Uncertainty in Artificial Intelligence. find
  51. Zettlemoyer, L., & Collins, M. (2007). Online learning of relaxed CCG grammars for parsing to logical form. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. find