Elamur Research⌕
☰
home / papers / 2015 / Language Understanding for Text-based Games Using Deep Reinforcement Learning

Language Understanding for Text-based Games Using Deep Reinforcement Learning

Paper · 2015 · Karthik Narasimhan, Tejas Kulkarni and Regina Barzilay

Summary

The paper tackles automatic play of text-based adventure games (Multi-User Dungeons). In these games the true world state is hidden, and the agent sees only natural-language descriptions, so it has to interpret text before it can act well. Narasimhan, Kulkarni and Barzilay treat gameplay as a Markov Decision Process. They train a deep Q-network, LSTM-DQN, using only in-game rewards, with no annotated state labels. An LSTM with mean pooling turns each description into a state vector, and a multi-layer action scorer then rates action–object commands. Both parts are learned jointly. Experiments use two worlds built on the Evennia framework: a controlled Home world and a Fantasy world whose descriptions are written by humans and vary more. LSTM-DQN beats a random agent and bag-of-words or bag-of-bigrams DQNs in both quest completion and accumulated reward. Its learned representations also transfer to a re-mapped world. The work shows that rich text representations can be learned from reward alone, which links grounded language understanding with reinforcement learning.

Key points

  • On the Fantasy world, LSTM-DQN completes about 96% of quests with an average reward of −11.33. BOW-DQN reaches about 82% (reward −12.68), and the random agent solves about 5%. BI-DQN completes 97% of quests but earns a worse reward (−26.68) because its solutions are sub-optimal.
  • Transfer test: the LSTM parameters (θ_R) were copied from the original Home world into a Home world with a new map. This agent reached the optimal policy roughly 20 epochs sooner than one trained from scratch.
  • Prioritized minibatch sampling reached the optimal policy about 50 epochs earlier than uniform sampling.
  • In t-SNE plots, words learned from random initialization form coherent groups for each room, its objects and its quest words (e.g., "hungry" sits near kitchen and pizza). Nearest neighbors by cosine similarity tend to be descriptions of the same or nearby states.
  • Code is released at people.csail.mit.edu/karthikn/mud-play (arXiv:1506.08941, EMNLP 2015).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States243 works · 76,506 citations🔬MIT Computer Science and Artificial Intelligence LaboratoryCambridge · United States76 works · 32,207 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Karthik Narasimhan, Tejas Kulkarni and Regina Barzilay. 2015. Language Understanding for Text-based Games Using Deep Reinforcement Learning. arXiv:1506.08941.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1506.08941 · arXiv 1506.08941

@misc{narasimhan2015language,
  title = {{Language Understanding for Text-based Games Using Deep Reinforcement Learning}},
  author = {Karthik Narasimhan, Tejas Kulkarni and Regina Barzilay},
  year = {2015},
  doi = {10.48550/arxiv.1506.08941},
  eprint = {1506.08941},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1506.08941}
}

References

29 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christopher Amato and Guy Shani. 2010. High-level reinforcement learning in strategy games. In Proceedings of the 9th International Conference on Autonomous Agents and Multi-agent Systems: Volume 1, pages 75–82. International Foundation for Autonomous Agents and Multiagent Systems. find
  2. Eyal Amir and Patrick Doyle. 2002. Adventure games: A challenge for cognitive robotics. In Proc. Int. Cognitive Robotics Workshop, pages 148–155. find
  3. Jacob Andreas and Dan Klein. 2015. Alignment-based compositional semantics for instruction following. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  4. Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, 1(1):49–62.
  5. SRK Branavan, Luke S Zettlemoyer, and Regina Barzilay. 2010. Reading between the lines: Learning to map high-level instructions to commands. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1268–1277. Association for Computational Linguistics. find
  6. SRK Branavan, David Silver, and Regina Barzilay. 2011a. Learning to win by reading manuals in a monte-carlo framework. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 268–277. Association for Computational Lingui… find
  7. SRK Branavan, David Silver, and Regina Barzilay. 2011b. Non-linear monte-carlo search in Civilization II. AAAI Press/International Joint Conferences on Artificial Intelligence. find
  8. Pavel Curtis. 1992. Mudding: Social phenomena in text-based virtual realities. High noon on the electronic frontier: Conceptual issues in cyberspace, pages 347–374. find
  9. Mark A DePristo and Robert Zubek. 2001. being-in-the-world. In Proceedings of the 2001 AAAI Spring Symposium on Artificial Intelligence and Interactive Entertainment, pages 31–34. find
  10. Jacob Eisenstein, James Clarke, Dan Goldwasser, and Dan Roth. 2009. Reading to learn: Constructing features from semantic abstracts. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 958–967, Singapore, August. Association for Computational Linguistics. find
  11. Peter Gorniak and Deb Roy. 2005. Speaking with your sidekick: Understanding situated speech in computer role playing games. In R. Michael Young and John E. Laird, editors, Proceedings of the First Artificial Intelligence and Interactive Digital Entertainment Conference, June 1-5, 2005, Marina del Re… find
  12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  13. Thomas Kollar, Stefanie Tellex, Deb Roy, and Nicholas Roy. 2010. Toward understanding natural language directions. In Human-Robot Interaction (HRI), 2010 5th ACM/IEEE International Conference on, pages 259–266. IEEE. find
  14. Jan Koutnı́k, Giuseppe Cuccu, Jürgen Schmidhuber, and Faustino Gomez. 2013. Evolving large-scale neural networks for vision-based reinforcement learning. In Proceedings of the 15th annual conference on Genetic and evolutionary computation, pages 1061–1068. ACM. find
  15. Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014. Learning to automatically solve algebra word problems. ACL (1), pages 271–281. find
  16. Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer, and Dieter Fox. 2013. Learning to parse natural language commands to a robot control system. In Experimental Robotics, pages 403– 415. Springer. find
  17. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
  18. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis H… find
  19. Andrew W Moore and Christopher G Atkeson. 1993. Prioritized sweeping: Reinforcement learning with less data and less time. Machine Learning, 13(1):103–130. find
  20. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014), 12.
  21. David Silver, Richard S Sutton, and Martin Müller. 2007. Reinforcement learning of local shape in the game of go. In IJCAI, volume 7, pages 1053–1058. find
  22. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104– 3112.
  23. Richard S Sutton and Andrew G Barto. 1998. Introduction to reinforcement learning. MIT Press. find
  24. István Szita. 2012. Reinforcement learning in games. In Reinforcement Learning, pages 539–577. Springer. find
  25. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural La… find
  26. Tijmen Tieleman and Geoffrey Hinton. 2012. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4. find
  27. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of Machine Learning Research, 9(2579-2605):85. find
  28. Adam Vogel and Dan Jurafsky. 2010. Learning to follow navigational directions. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 806–814. Association for Computational Linguistics. find
  29. Christopher JCH Watkins and Peter Dayan. 1992. Q-learning. Machine learning, 8(3-4):279–292. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/narasimhan-2015-language-understanding-text-based-games/