Elamur Research
☰
home / papers / 2018 / Unified Pragmatic Models for Generating and Following Instructions

Unified Pragmatic Models for Generating and Following Instructions

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N18-1177.pdf

Summary

Explicit pragmatic reasoning means agents model each other's intentions. This paper asks whether it helps in grounded instruction tasks with long, sequential actions. Earlier computational pragmatics mostly tested such reasoning on simple reference games. Fried, Andreas and Klein train two neural sequence-to-sequence base models: a listener L0 that turns directions into action sequences, and a speaker S0 that turns actions into directions. On top of these they build rational agents in the rational speech acts tradition. One base model proposes candidates and the other rescores them. The pragmatic speaker therefore picks descriptions that a simulated listener would execute correctly. The pragmatic listener favours interpretations that a simulated speaker would plausibly have phrased that way. Experiments use the SAIL navigation corpus and three SCONE domains (Alchemy, Scene, Tangrams). The pragmatic layer improves instruction following in most domains. It also makes generated instructions much easier for humans to follow, matching or surpassing earlier state-of-the-art systems. The work shows that the same human annotations can yield agents that communicate better than a direct learned mapping.

Key points

  • Data: the SAIL corpus (drawing on Chen and Mooney, 2011, as well as MacMahon et al., 2006) and the SCONE domains Alchemy, Scene and Tangrams (Long et al., 2016); code is released at github.com/dpfried/pragmatic-instructions.
  • Human evaluation of generation: 50 test action sequences per domain, each followed by three Mechanical Turk workers. The rational speaker S0·S1 beats S0 by 12.4 to 46.0 absolute accuracy points. In Alchemy, only 29.3% of S0's directions are followed correctly, versus 83.3% of human directions.
  • In Tangrams, humans follow the rational speaker's directions more accurately than the original human-written ones. On SAIL, the rational speaker is comparable to the system of Daniele et al. (2017), which needs extra structured annotation.
  • Instruction following improves by up to 10% accuracy. The gains are not just an ensembling effect: with 20 models in total (10 listeners + 10 speakers versus 20 listeners), accuracy rises 68.5→71.6% on SAIL single-sentence Rel and 70.1→72.0% on Alchemy.
  • BLEU turns out to be a weak predictor of whether generated directions can actually be followed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

48 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 47 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-1177 · arXiv 1711.04987

@inproceedings{fried2018unified,
  title = {{Unified Pragmatic Models for Generating and Following Instructions}},
  author = {Fried, Daniel and Andreas, Jacob and Klein, Dan},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1177},
  eprint = {1711.04987},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n18-1177}
}

References

48 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anne H. Anderson, Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jacqueline Kowtko, Jan McAllister, Jim Miller, et al. 1991. The HCRC map task corpus. Language and speech 34(4):351–366. find
  2. Jacob Andreas and Dan Klein. 2015. Alignment-based compositional semantics for instruction following. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  3. Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find our card
  4. Yoav Artzi, Dipanjan Das, and Slav Petrov. 2014. Learning compact lexicons for CCG semantic parsing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
  5. Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics 1(1):49–62. find our card
  6. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. International Conference on Learning Representations . find our card
  7. Leon Bergen, Roger Levy, and Noah Goodman. 2016. Pragmatic reasoning through semantic inference. Semantics and Pragmatics 9. find
  8. S.R.K. Branavan, Harr Chen, Luke S. Zettlemoyer, and Regina Barzilay. 2009. Reinforcement learning for mapping instructions to actions. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pages 82–90. find
  9. Donna Byron, Alexander Koller, Kristina Striegnitz, Justine Cassell, Robert Dale, Johanna Moore, and Jon Oberlander. 2009. Report on the first NLG challenge on generating instructions in virtual environments (GIVE). In Proceedings of the 12th european workshop on natural language generation. Association for Computational Linguistics, pages 165–173. find
  10. David L Chen. 2012. Fast online lexicon learning for grounded language acquisition. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. pages 430–439. find
  11. David L. Chen and Raymond J. Mooney. 2011. Learning to interpret natural language navigation instructions from observations. In Proceedings of the Meeting of the Association for the Advancement of Artificial Intelligence. volume 2, pages 1–2. find
  12. Robert Dale, Sabine Geldof, and Jean-Philippe Prost. 2005. Using natural language generation in automatic route. Journal of Research and practice in Information Technology 37(1):89. find
  13. Andrea F. Daniele, Mohit Bansal, and Matthew R. Walter. 2017. Navigational instruction generation as inverse reinforcement learning with neural machine translation. Proceedings of Human-Robot Interaction . find
  14. Michael C Frank and Noah D Goodman. 2012. Predicting pragmatic reasoning in language games. Science 336(6084):998–998. ↗ our card
  15. Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In Advances in Neural Information Processing Systems 29 (NIPS). find our card
  16. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In AISTATS. volume 9, pages 249–256. find
  17. Robert Goeddel and Edwin Olson. 2012. Dart: A particle-based method for generating easy-to-follow directions. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on. IEEE, pages 1213–1219. find
  18. Dave Golland, Percy Liang, and Dan Klein. 2010. A game-theoretic approach to generating spatial descriptions. In Proceedings of the 2010 conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 410–419. find
  19. Noah D Goodman and Andreas Stuhlmüller. 2013. Knowledge and implicature: Modeling language understanding as social cognition. Topics in cognitive science 5(1):173–184. find
  20. Klaus Greff, Rupesh K Srivastava, Jan Koutnı́k, Bas R Steunebrink, and Jürgen Schmidhuber. 2016. Lstm: A search space odyssey. IEEE transactions on neural networks and learning systems . find
  21. H. P. Grice. 1975. Logic and conversation. In P. Cole and J. L. Morgan, editors, Syntax and Semantics: Vol. 3: Speech Acts, Academic Press, San Diego, CA, pages 41–58. find
  22. Kelvin Guu, Panupong Pasupat, Evan Zheran Liu, and Percy Liang. 2017. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. In Association for Computational Linguistics (ACL). find
  23. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735–1780. ↗
  24. Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L Berg. 2014. ReferItGame: Referring to objects in photographs of natural scenes. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. pages 787–798. find
  25. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. International Conference on Learning Representations . find
  26. Will Monroe and Christopher Potts. 2015. Learning in the Rational Speech Acts model. In Proceedings of 20th Amsterdam Colloquium. ILLC, Amsterdam. find
  27. Alexander Koller, Kristina Striegnitz, Andrew Gargett, Donna Byron, Justine Cassell, Robert Dale, Johanna Moore, and Jon Oberlander. 2010. Report on the second NLG challenge on generating instructions in virtual environments (GIVE-2). In Proceedings of the 6th international natural language generation conference. Association for Computational Linguistics, pages 243–250. find
  28. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, pages 311–318. find our card
  29. Emiel Krahmer and Mariët Theune, editors. 2010. Empirical Methods in Natural Language Generation: Data-oriented Methods and Empirical Evaluation. Springer-Verlag, Berlin, Heidelberg. find
  30. Mike Lewis, Denis Yarats, Yann N Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning for negotiation dialogues. In Empirical Methods in Natural Language Processing (EMNLP). find
  31. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics. find our card
  32. Reginald Long, Panupong Pasupat, and Percy Liang. 2016. Simpler context-dependent logical forms via model projections. In Association for Computational Linguistics (ACL). find
  33. Gary Wai Keung Look. 2008. Cognitively-inspired direction giving. Ph.D. thesis, Massachusetts Institute of Technology. find
  34. Ruotian Luo and Gregory Shakhnarovich. 2017. Comprehension-guided referring expressions. In Computer Vision and Pattern Recognition. find
  35. Matt MacMahon, Brian Stankiewicz, and Benjamin Kuipers. 2006. Walk the talk: Connecting language, knowledge, and action in route instructions. Proceedings of the Meeting of the Association for the Advancement of Artificial Intelligence 2(6):4. find
  36. Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions. In Computer Vision and Pattern Recognition. find our card
  37. Hongyuan Mei, Mohit Bansal, and Matthew Walter. 2016. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In Proceedings of the Meeting of the Association for the Advancement of Artificial Intelligence. find
  38. Will Monroe, Robert X.D. Hawkins, Noah D. Goodman, and Christopher Potts. 2017. Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics . find our card
  39. Christopher Potts. 2011. Pragmatics, Oxford University Press. find
  40. Nathaniel J Smith, Noah Goodman, and Michael Frank. 2013. Learning and using language via recursive pragmatic reasoning about other agents. In Advances in Neural Information Processing Systems. pages 3039–3047. find
  41. Kristina Striegnitz, Alexandre Denis, Andrew Gargett, Konstantina Garoufi, Alexander Koller, and Mariët Theune. 2011. Report on the second second challenge on generating instructions in virtual environments GIVE-2.5. In Proceedings of the 13th European workshop on natural language generation. Association for Computational Linguistics, pages 270– 279. find
  42. Jong-Chyi Su, Chenyun Wu, Huaizu Jiang, and Subhransu Maji. 2017. Reasoning about fine-grained attribute phrases using reference games. In International Conference on Computer Vision. find
  43. Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R. Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy. 2011. Understanding natural language commands for robotic navigation and mobile manipulation. In In Proceedings of the National Conference on Artificial Intelligence. find
  44. Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, and Gal Chechik. 2017. Context-aware captions from context-agnostic supervision. In Computer Vision and Pattern Recognition (CVPR). volume 3. find
  45. Sida I. Wang, Percy Liang, and Christopher D. Manning. 2016. Learning language games through interaction. In Association for Computational Linguistics (ACL). find
  46. Tom Williams, Gordon Briggs, Bradley Oosterveld, and Matthias Scheutz. 2015. Going beyond literal command-based instructions: Extending robotic natural language interaction capabilities. In AAAI. pages 1387–1393. find
  47. Lei Yu, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Tomas Kocisky. 2017a. The neural noisy channel. International Conference on Learning Representations . find
  48. Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L. Berg. 2017b. A joint speaker-listener-reinforcer model for referring expressions. In Computer Vision and Pattern Recognition. type arguments contextual embedding Alchemy M IX P OUR D RAIN source i source i, target j amount a, source i contents of i contents of i and j a, contents of i Scene E NTER color c, source i E XIT source i M OVE source i, target j S WITCH source i, target j T AKE H AT source i, target j Tangrams R EMOVE position i S WAP positions i and j I NSERT position i, shape s people at i − 1 and i + 1 people at i, i − 1, i + 1 people at i, j − 1, j + 1 people at i and j people at i and j — — index of step when s was removed *Table 5: Action types, arguments, and elements of the world state or action history that are extracted to produce contextual action embeddings.* find