Elamur Research⌕
☰
home / papers / 2020 / Keep CALM and Explore: Language Models for Action Generation in Text-based Games

Keep CALM and Explore: Language Models for Action Generation in Text-based Games

Paper · 2020 · Shunyu Yao, Rohan Rao, Matthew Hausknecht et al.

Summary

Text-based games are hard for reinforcement learning agents because the space of possible text commands is combinatorially huge, and only a few commands in each state are valid or useful. Earlier agents often depended on Jericho's handicap that lists admissible actions. The authors propose CALM (Contextual Action Language Model), which is trained on human play transcripts to suggest a small set of plausible actions for the current situation. The model sees the previous observation, the previous action and the current observation. A DRRN agent then scores these candidates with Q-values learned from game rewards. They compare n-gram and GPT-2 versions. None of the evaluation games appear in CALM's training data. On 28 Jericho games, CALM (GPT-2) with DRRN raises the average normalized score by 69% relative to the previous best handicap-free method. On several games it matches or beats agents that use the admissible-action handicap. The work shows that general linguistic and gameplay knowledge from humans can transfer to unseen games and make exploration tractable.

Key points

  • New dataset: 426 human gameplay transcripts covering 590 games, collected from ClubFloyd. The dataset and code are released at github.com/princeton-nlp/calm-textgame.
  • The action space is very large. ZORK1 alone has a 697-word vocabulary, which gives roughly 200 billion possible 4-word commands.
  • On 8 games CALM outperforms handicap-assisted agents. For example, on INHUMANE it scores 25.7, while KG-A2C scores 3.
  • Ablations show that both GPT-2 pre-training and ClubFloyd fine-tuning matter. A candidate set size of k = 10 performs clearly worse than k = 20–40. Replacing DRRN with a random agent gives only 1.8%.
  • During exploration the agent reached higher scores than it finally kept. If RL fully exploited the best trajectories it saw, the average normalized score would reach 14.7%, which exceeds all compared models, including those with handicaps.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States240 works · 123,537 citations🎓Princeton UniversityPrinceton · United States61 works · 36,471 citations

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 1,343 works

Cite

Cite as Shunyu Yao, Rohan Rao, Matthew Hausknecht and Karthik Narasimhan. 2020. Keep CALM and Explore: Language Models for Action Generation in Text-based Games. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.704

@inproceedings{yao2020keep,
  title = {{Keep CALM and Explore: Language Models for Action Generation in Text-based Games}},
  author = {Shunyu Yao, Rohan Rao, Matthew Hausknecht and Karthik Narasimhan},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.704},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.704}
}

References

28 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Prithviraj Ammanabrolu, William Broniec, Alex Mueller, Jeremy Paul, and Mark O Riedl. 2019. Toward automated quest generation in text-adventure games. arXiv preprint arXiv:1909.06283. find
  2. Prithviraj Ammanabrolu and Matthew Hausknecht. 2020. Graph constrained reinforcement learning for natural language action spaces. arXiv preprint arXiv:2001.08837. — [arXiv:2001.08837](https://arxiv.org/abs/2001.08837) find
  3. Prithviraj Ammanabrolu and Mark Riedl. 2019. Playing text-adventure games with graph-based deep reinforcement learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers),… find
  4. Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017. A survey on dialogue systems: Recent advances and new frontiers. Acm Sigkdd Explorations Newsletter, 19(2):25–35. find
  5. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder–decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103–111, Doha, Qatar. Associati…
  6. Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. 2018. Textworld: A learning environment for text-based games. In Workshop on Computer Games, pages 41–75. Springer.
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  8. Nancy Fulda, Daniel Ricks, Ben Murdoch, and David Wingate. 2017. What can you do with a rock? affordance extraction via word embeddings. CoRR, abs/1703.03429. find
  9. Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. 2019a. Interactive fiction games: A colossal adventure. CoRR, abs/1909.05398. — [arXiv:1909.05398](https://arxiv.org/abs/1909.05398) find
  10. Matthew Hausknecht, Ricky Loynd, Greg Yang, Adith Swaminathan, and Jason D Williams. 2019b. Nail: A general interactive fiction agent. arXiv preprint arXiv:1902.04259. — [arXiv:1902.04259](https://arxiv.org/abs/1902.04259) find
  11. Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. 2015. Deep reinforcement learning with a natural language action space. arXiv preprint arXiv:1511.04636. find
  12. Vishal Jain, William Fedus, Hugo Larochelle, Doina Precup, and Marc G. Bellemare. 2019. Algorithmic improvements for deep reinforcement learning applied to interactive fiction. CoRR, abs/1903.03094. find
  13. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431. Association for Computation… find
  14. Daniel Jurafsky and James H. Martin. 2009. Speech and Language Processing (2nd Edition). Prentice-Hall, Inc., USA. find
  15. Bartosz Kostka, Jaroslaw Kwiecien, Jakub Kowalski, and Pawel Rychlikowski. 2017. Text-based adventures of the golovin AI agent. CoRR, abs/1705.05637. — [arXiv:1705.05637](https://arxiv.org/abs/1705.05637) find
  16. Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. 2020. Multi-agent communication meets natural language: Synergies between functional and structural language learning. arXiv preprint arXiv:2005.07064. find
  17. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pa…
  18. Karthik Narasimhan, Tejas D. Kulkarni, and Regina Barzilay. 2015. Language understanding for text-based games using deep reinforcement learning. In EMNLP, pages 1–11. find
  19. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. find
  20. Yiping Song, Rui Yan, Xiang Li, Dongyan Zhao, and Ming Zhang. 2016. Two are better than one: An ensemble of retrieval-and generation-based dialog systems. arXiv preprint arXiv:1610.07149. — [arXiv:1610.07149](https://arxiv.org/abs/1610.07149) find
  21. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find
  22. Ruo Yu Tao, Marc-Alexandre Côté, Xingdi Yuan, and Layla El Asri. 2018. Towards solving text-based games by producing adaptive action spaces. arXiv preprint arXiv:1812.00855. — [arXiv:1812.00855](https://arxiv.org/abs/1812.00855) find
  23. Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Jason Weston. 2019. Learning to speak and act in a fantasy text adventure game. CoRR, abs/1903.03094. find
  24. Nick Walton. 2019. Ai dungeon 2: Creating infinitely generated text adventures with deep learning language models. find
  25. Jason D. Williams, Kavosh Asadi, and Geoffrey Zweig. 2017. Hybrid code networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 665… find
  26. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
  27. Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J Mankowitz, and Shie Mannor. 2018. Learn what not to learn: Action elimination with deep reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing S… find
  28. Tiancheng Zhao and Maxine Eskenazi. 2016. Towards end-to-end learning for dialog state tracking and management using deep reinforcement learning. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 1–10, Los Angeles. Association for Computational… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yao-2020-keep-calm-explore-language-models/