Elamur Research
☰
home / papers / 2018 / TextWorld: A Learning Environment for Text-based Games

TextWorld: A Learning Environment for Text-based Games

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1806.11532

Summary

Text adventure games such as Zork are a hard testbed for reinforcement learning. Both the observations and the actions are free-form, compositional language, the world is only partly visible, and rewards are rare. Hand-written games of this kind remain out of reach for current agents. The authors present TextWorld, a Python sandbox that runs text games and handles state tracking and reward assignment. Beyond replaying a curated, analyzed set of existing games, it can build new ones. A logic engine assembles maps, objects, obstacles and quests, and a context-free grammar with synonym-rich vocabulary renders the state as prose. Because the full game state is known, the framework supports exact tracking, optional intermediate rewards and tunable difficulty. These controls enable studies of curriculum learning, generalization and transfer. The paper frames text games as POMDPs and releases a generated "treasure hunter" benchmark. Two existing agents, BYU and Golovin, nearly fail on unseen generated games, which shows that fresh, controllable games are needed to measure real progress.

Key points

  • Commands are restricted to at most L words from a fixed vocabulary V, with the pattern verb [noun phrase [adverb phrase]] and no determiners. Generation is word by word and ends with a <return> token.
  • The treasure hunter benchmark has 30 levels in three tiers:
    • easy: 5 rooms, quest length 1–5;
    • medium: 10 rooms, quest length 2–10, closed doors and containers;
    • hard: 20 rooms, quest length 3–20, locked doors and containers.
  • The one-life evaluation used 100 new games per level, each capped at 1000 steps. Picking up the correct object scored +1, the wrong one −1, and finding nothing 0.
  • BYU and Golovin scored about 0.75–0.78 on level 1 but stayed near 0 on harder levels. A random agent choosing from admissible commands scored roughly 0.2–0.4, but the authors caution that its results cannot be fairly set side by side with those of BYU and Golovin.
  • Stated limitations:
    • quests are linear chains;
    • there are no time-triggered events, NPCs or multi-user play;
    • CFG-generated text has weak continuity between sentences.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

122 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 112 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1806.11532 · arXiv 1806.11532

@misc{cote2018textworld,
  title = {{TextWorld: A Learning Environment for Text-based Games}},
  author = {Cote, Marc-Alexandre and Kadar, Akos and Yuan, Xingdi and Kybartas, Ben and Barnes, Tavian and Fine, Emery and Moore, James and Tao, Ruo Yu and Hausknecht, Matthew and Asri, Layla El and Adada, Mahmoud and Tay, Wendy and Trischler, Adam},
  year = {2018},
  doi = {10.48550/arxiv.1806.11532},
  eprint = {1806.11532},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1806.11532}
}

References

40 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Douglas Adams and Steve Meretzky. The hitchhiker’s guide to the galaxy, 1984. URL http: //ifdb.tads.org/viewgame?id=ouv80gvsl32xlion. find
  2. Timothy Atkinson, Hendrik Baier, Tara Copplestone, Sam Devlin, and Jerry Swan. The text-based adventure ai competition. arXiv preprint arXiv:1808.01262, 2018. — [arXiv:1808.01262](https://arxiv.org/abs/1808.01262) find
  3. Marco Baroni, Armand Joulin, Allan Jabri, Germàn Kruszewski, Angeliki Lazaridou, Klemen Simonic, and Tomas Mikolov. Commai: Evaluating the first steps towards a useful general ai. arXiv preprint arXiv:1701.08954, 2017. find
  4. Bob Bates. Sherlock: The riddle of the crown jewels, 1987. URL http://ifdb.tads.org/viewgame?id=j8lmspy4iz73mx26. ↗
  5. Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03801, 2016. find
  6. Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. J. Artif. Intell. Res.(JAIR), 47:253–279, 2013. find
  7. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48. ACM, 2009. find our card
  8. Michael Berlyn. Infidel, anu79a4n1jedg5mm. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016. find
  9. Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. arXiv preprint arXiv:1706.07230, 2017. — [arXiv:1706.07230](https://arxiv.org/abs/1706.07230) find
  10. N. Chomsky. Three models for the description of language. IEEE Transactions on Information Theory, 2(3):113–124, sep 1956. doi: 10.1109/tit.1956.1056813. find
  11. William Crowther and Donald Woods. viewgame?id=fft6pu91j85y4acv. Brian Desilets. The enterprise incidents, 2002. URL http://ifdb.tads.org/viewgame?id=ld1f3t5epeagilfz. ↗
  12. Richard E Fikes and Nils J Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial intelligence, 2(3-4):189–208, 1971. find
  13. Nancy Fulda, Daniel Ricks, Ben Murdoch, and David Wingate. What can you do with a rock? affordance extraction via word embeddings. arXiv preprint arXiv:1703.03429, 2017. find
  14. Stu Galley and Jim Lawrence. Seastalker, 1984. URL http://ifdb.tads.org/viewgame?id=56wb8hflec2isvzm. ↗
  15. Michael Genesereth, Nathaniel Love, and Barney Pell. General game playing: Overview of the aaai competition. AI magazine, 26(2):62, 2005. find
  16. Russell Glasser. Reverberations, 1996. dop7nbjl90r5zmf9. find
  17. J.J. Guest. Goldilocks is a fox!, 2002. 59ztsy9p01avd6wp. 1983. URL http://ifdb.tads.org/viewgame?id= ↗
  18. URL http://ifdb.tads.org/1998. URL http://ifdb.tads.org/viewgame?id=URL http://ifdb.tads.org/viewgame?id=URL http://ifdb.tads.org/viewgame?id=Matan Haroush, Tom Zahavy, Daniel J Mankowitz, and Shie Mannor. Learning how not to act in text-based games. International Conference on Learning Representati… ↗
  19. Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. Deep reinforcement learning with a natural language action space. arXiv preprint arXiv:1511.04636, 2015. find
  20. Matthew Henderson, Blaise Thomson, and Jason D Williams. The second dialog state tracking challenge. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 263–272, 2014. find
  21. Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojtek Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded language learning in a simulated 3d world. arXiv preprint arXiv:1706.06551, 2017. find
  22. Infocom. Zork I, 1980. URL http://ifdb.tads.org/viewgame?id=0dbnusxunq7fw5ro. ↗
  23. Infocom. Zork III, 1982. URL http://ifdb.tads.org/viewgame?id=vrsot1zgy1wfcdru. ↗
  24. Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99–134, 1998. find
  25. Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski. Vizdoom: A doom-based ai research platform for visual reinforcement learning. In Computational Intelligence and Games (CIG), 2016 IEEE Conference on, pages 1–8. find
  26. Bartosz Kostka, Jaroslaw Kwiecieli, Jakub Kowalski, and Pawel Rychlikowski. Text-based adventures of the golovin ai agent. In Computational Intelligence and Games (CIG), 2017 IEEE Conference on, pages 181–188. find
  27. Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. arXiv preprint arXiv:1709.06009, 2017. find
  28. Chris Martens. Ceptre: A language for modeling generative interactive systems. In Eleventh Artificial Intelligence and Interactive Digital Entertainment Conference, 2015. find
  29. Roger McFarlane. A survey of exploration strategies in reinforcement learning. 2003. URL http: //www.cs.mcgill.ca/\\begingroup\\let\\relax\\relax\\endgroup[Pleaseinsert\\ PrerenderUnicode{âĹij}intopreamble]cs526/roger.pdf. find
  30. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015. find
  31. Brian Moriarty. Wishbringer, 1985. z02joykzh66wfhcl. find
  32. Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. Language understanding for text-based games using deep reinforcement learning. arXiv preprint arXiv:1506.08941, 2015. find
  33. Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning. arXiv preprint arXiv:1702.08360, 2017. — [arXiv:1702.08360](https://arxiv.org/abs/1702.08360) find
  34. Karl Pearson. The problem of the random walk. Nature, 72(1867):342, 1905. find
  35. Andrew Plotkin. Inhumane, wvs2vmbigm9unlpd. Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., New York, NY, USA, 1st edition, 1994. ISBN 0471619779. find
  36. Stuart J Russell and Peter Norvig. Artificial intelligence: a modern approach. Malaysia; Pearson Education Limited„ 2016. find
  37. URL http://ifdb.tads.org/viewgame?id=1985. URL http://ifdb.tads.org/viewgame?id=James Ryan, Ethan Seither, Michael Mateas, and Noah Wardrip-Fruin. Expressionist: An Authoring Tool for In-Game Text Generation, pages 221–233. Springer International Publishing, Cham, 2016. ISBN 978-3-319-48279-8. ↗
  38. Sainbayar Sukhbaatar, Arthur Szlam, Gabriel Synnaeve, Soumith Chintala, and Rob Fergus. Mazebase: A sandbox for learning from games. arXiv preprint arXiv:1511.07401, 2015. — [arXiv:1511.07401](https://arxiv.org/abs/1511.07401) find
  39. Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press Cambridge, 2018. find
  40. Will Wright. SimCity. Erbe, 1996. find