Elamur Research⌕
☰
home / papers / 2017 / Simulating Action Dynamics with Neural Process Networks

Simulating Action Dynamics with Neural Process Networks

Paper · 2017 · Antoine Bosselut, Omer Levy, Ari Holtzman et al.

Summary

Procedural texts such as recipes often leave the consequences of actions unstated. For example, baking implies that an ingredient becomes hot and ends up in the oven. Most NLU models cannot infer such implicit effects. The authors propose Neural Process Networks (NPN), which read instructions sentence by sentence and simulate their effects on entities.

A GRU sentence encoder feeds two selectors. One softly picks actions from a fixed action set, and the other uses recurrent attention to pick the affected entities, which handles elided arguments and pronouns. An applicator, built on a third-order tensor, combines the chosen action embeddings with the entity states. The result overwrites the selected entity memories, and classifiers then forecast each entity's end state across six attributes: its cleanliness, shape, composition, temperature, degree of cooking, and where it is located. Because prediction errors flow back into the action embeddings, actions are learned as state-transforming operators.

Training uses cooking recipes and a crowdsourced lexicon of action effects. The paper reports that NPN gives better context for both tracking state changes and generating recipe text than Recurrent Entity Networks and GRU/seq2seq baselines, with more interpretable internal representations. The authors also release a new dataset with fine-grained state-change annotations.

Key points

  • The action lexicon covers 384 cooking verbs, each labeled by seven Amazon Mechanical Turk workers using majority vote. Of these, 342 verbs received at least one state-change type, and 74 cause several types.
  • End-state label sets vary in size: some dimensions are binary (cooked/raw), while LOCATION has more than 260 possible end states (pan, pot, oven, …).
  • Baselines are a joint GRU and Recurrent Entity Networks (EntNet) for tracking, and seq2seq, attentive seq2seq and an EntNet-based generator for generation. Vocabularies contain 7358 words and 2996 entities.
  • In a development-set ablation with full training labels, NPN reaches 56.84 entity-selection F1, 50.56 state-change F1 and 57.87 accuracy. Keeping only 25% of the labels drops state-change F1 to 2.52.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States402 works · 212,762 citations

Also on the byline: University of Washington Bothell.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Antoine Bosselut, Omer Levy, Ari Holtzman, Corin Ennis, Dieter Fox and Yejin Choi. 2017. Simulating Action Dynamics with Neural Process Networks. arXiv:1711.05313.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1711.05313 · arXiv 1711.05313

@misc{bosselut2017simulating,
  title = {{Simulating Action Dynamics with Neural Process Networks}},
  author = {Antoine Bosselut, Omer Levy, Ari Holtzman, Corin Ennis, Dieter Fox and Yejin Choi},
  year = {2017},
  doi = {10.48550/arxiv.1711.05313},
  eprint = {1711.05313},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1711.05313}
}

References

32 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations, 2015.
  2. Silvia Chiappa, Sébastien Racanière, Daan Wierstra, and Shakir Mohamed. Recurrent environment simulators. In Proceedings of the 5th International Conference on Learning Representations, 2017. find
  3. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, 2014.
  4. Qiaozi Gao, Malcolm Doering, Shaohua Yang, and Joyce Y Chai. Physical causality of action verbs in grounded language understanding. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 2016. find
  5. Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. Tracking the world state with recurrent entity networks. In Proceedings of the 5th International Conference on Learning Representations, 2017.
  6. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. The goldilocks principle: Reading children’s books with explicit memory representations. In Proceedings of the 4th International Conference on Learning Representations, 2016. — The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
  7. Yangfeng Ji, Chenhao Tan, Sebastian Martschat, Yejin Choi, and Noah A Smith. Dynamic entity representations in neural language models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017.
  8. Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017.
  9. Chloé Kiddon, Ganesa Thandavam Ponnuraj, Luke Zettlemoyer, and Yejin Choi. Mise en place: Unsupervised interpretation of instructional recipes. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015. find
  10. Chloé Kiddon, Luke Zettlemoyer, and Yejin Choi. Globally coherent text generation with neural checklist models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016.
  11. Jin-Hwa Kim, Kyoung Woon On, Jeonghee Kim, JungWoo Ha, and Byoung-Tak Zhang. Hadamard product for low-rank bilinear pooling. In Proceedings of the 5th International Conference on Learning Representations, 2017. find
  12. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, 2015.
  13. Omer Levy, Steffen Remus, Christian Biemann, and Ido Dagan. Do supervised distributional methods really learn lexical inference relations? In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics – Human Language Technologies, 2015.
  14. Changsong Liu, Shaohua Yang, Sari Saba-Sadiya, Nishant Shukla, Yunzhong He, Song-Chun Zhu, and Joyce Y Chai. Jointly learning grounded task structures from language instruction and visual demonstration. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016. find
  15. Li Lucy and Jon Gauthier. Are distributional representations ready for the real world? evaluating word vectors for grounded perceptual meaning. In Proceedings of the RoboNLP Workshop at the 55th Annual Meeting of the Association for Computational Linguistics, 2017. find
  16. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013a.
  17. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26, 2013b.
  18. Alexander H. Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. Key-value memory networks for directly reading documents. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016. find
  19. Shinsuke Mori, Tetsuro Sasada, Yoko Yamakata, and Koichiro Yoshino. A machine learning approach to recipe text processing. In Proceedings of the 1st Cooking with Computer Workshop, 2012. find
  20. Shinsuke Mori, Hirokuni Maeta, Yoko Yamakata, and Tetsuro Sasada. Flow graph corpus from recipe texts. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, 2014. find
  21. Arvind Neelakantan, Quoc V. Le, and Ilya Sutskever. Neural programmer: Inducing latent programs with gradient descent. In Proceedings of the 4th International Conference on Learning Representations, 2016. find
  22. Arvind Neelakantan, Quoc V. Le, Martı́n Abadi, Andrew McCallum, and Dario Amodei. Learning a natural language interface with neural programmer. In Proceedings of the 5th International Conference on Learning Representations, 2017. find
  23. Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L. Lewis, and Satinder P. Singh. Action-conditional video prediction using deep networks in atari games. In Advances in neural information processing systems, 2015.
  24. Minjoon Seo, Sewon Min, Ali Farhadi, and Hannaneh Hajishirzi. Query-reduction networks for question answering. In Proceedings of the 5th International Conference on Learning Representations, 2017. find
  25. Zhangzhang Si, Mingtao Pei, Benjamin Yao, and Song-Chun Zhu. Unsupervised learning of event and-or grammar and semantics from video. In Proceedings of the 2011 IEEE International Conference on Computer Vision, 2011. find
  26. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 2014.
  27. Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. In Advances in neural information processing systems, 2015. find
  28. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, 2014.
  29. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. Modeling coverage for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 2016.
  30. Niklas Wahlstrom, Thomas B. Schön, and Marc Peter Deisenroth. From pixels to torques: Policy learning with deep dynamical models. CoRR, abs/1502.02251, 2015. — [arXiv:1502.02251](https://arxiv.org/abs/1502.02251) find
  31. Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. In Proceedings of the 3rd International Conference on Learning Representations, 2015. find
  32. Zichao Yang, Phil Blunsom, Chris Dyer, and Wang Ling. Reference-aware language models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bosselut-2017-simulating-action-dynamics-neural-process/