Elamur Research⌕
☰
home / papers / 2018 / BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning

BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning

Paper · 2018 · Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou et al.

Summary

Training agents to follow language instructions with a human in the loop could make AI assistants easier to customize. It would also connect machine learning research with work on how children acquire language. The obstacle is cost: deep imitation and reinforcement learning methods need far more interaction than a person could reasonably supply.

The authors release BabyAI, a platform for measuring this cost. It is built on MiniGrid, a fast, partially observable 2D gridworld with keys, doors, balls and boxes. Tasks are written in the Baby Language, a synthetic subset of English defined by a grammar. A curriculum of 19 levels raises both environment and language difficulty, and a scripted bot plays the role of a human teacher.

Using imitation and reinforcement learning baselines, the authors estimate how many demonstrations or episodes are needed to reach near-perfect performance on several levels. They also test pretraining and interactive imitation learning. Their central finding is that current deep learning approaches are not sample-efficient enough to learn a compositional language. BabyAI provides a controlled benchmark for tracking progress toward practical human-in-the-loop training.

Key points

  • BabyAI contains 19 levels of increasing difficulty, ranging from simple tasks such as GoToObj up to the BossLevel. The code and pretrained models are released at github.com/mila-iqia/babyai.
  • The Baby Language is defined by a BNF context-free grammar and allows about 2.48 × 10^19 distinct instructions. These cover going to objects, picking them up, opening doors and placing objects next to others, and can be chained with "then" or "after you".
  • In MiniGrid, the agent sees an egocentric 7x7 window encoded as a 7x7x3 tensor and chooses among seven discrete actions. Rewards are sparse, with a bonus for finishing quickly. The simulator exceeds 3000 frames per second on a laptop and integrates with OpenAI Gym.
  • The bot breaks each instruction into a stack of subgoals (GoNextTo, Open, Explore, Pickup, Drop). It can produce demonstrations on demand and advise the learner, which supports interactive methods like DAGGER.
  • Unlike earlier 3D or template-based environments, BabyAI combines world-state manipulation, partial observability, systematic language semantics, and a simulated teacher.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université de MontréalMontreal · Canada87 works · 91,597 citations🎓University of OregonEugene · United States26 works · 3,555 citations🎓Indian Institute of Technology BombayMumbai · India32 works · 2,535 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen and Yoshua Bengio. 2018. BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning. arXiv:1810.08272.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1810.08272 · arXiv 1810.08272

@misc{chevalierboisvert2018babyai,
  title = {{BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning}},
  author = {Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen and Yoshua Bengio},
  year = {2018},
  doi = {10.48550/arxiv.1810.08272},
  eprint = {1810.08272},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1810.08272}
}

References

47 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., Reid, I., Gould, S., and Hengel, A. v. d. (2018). Vision-and-Language Navigation: Interpreting visually-grounded naviga-IL from RL Expert 1.53 - 2.11 36.6 - 44.5 74.2 - 81.8 find
  2. Interactive IL from Bot 1.71 - 1.88 31.8 - 36 93 - 107 tion instructions in real environments. In Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  3. Andreas, J., Rohrbach, M., Darrell, T., and Klein, D. (2016). Neural Module Networks. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  4. Artzi, Y. and Zettlemoyer, L. (2013). Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, 1:49–62.
  5. Bahdanau, D., Cho, K., and Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of the 2015 International Conference on Learning Representations.
  6. Bahdanau, D., Hill, F., Leike, J., Hughes, E., Hosseini, A., Kohli, P., and Grefenstette, E. (2018). Learning to Understand Goal Specifications by Modelling Reward. In ICLR. find
  7. Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. (2013). The Arcade Learning Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelligence Research, 47:253–279. arXiv: 1207.4708.
  8. Bengio, Y., Louradour, J., Collobert, R., and Weston, J. (2009). Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48.
  9. Chaplot, D. S., Sathyendra, K. M., Pasumarthi, R. K., Rajagopal, D., and Salakhutdinov, R. (2018). Gated-Attention Architectures for Task-Oriented Language Grounding. In Proceedings of 32nd AAAI Conference on Artificial Intelligence. find
  10. Chen, D. L. and Mooney, R. J. (2011). Learning to Interpret Natural Language Navigation Instructions from Observations. In Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, pages 859–865. find
  11. Cho, K., van Merrienboer, B., Gulcehre, C., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  12. Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems 30. arXiv: 1706.03741. find
  13. Daumé Iii, H., Langford, J., and Marcu, D. (2009). Search-based structured prediction. Machine learning, 75(3):297–325. find
  14. DeepMind (2017). PycoLab. find
  15. Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., Legg, S., and Kavukcuoglu, K. (2018). IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures. In Proceedings of the 22nd international conference o… find
  16. Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., Colmenarejo, S. G., Grefenstette, E., Ramalho, T., Agapiou, J., Badia, A. P., Hermann, K. M., Zwols, Y., Ostrovski, G., Cain, A., King, H., Summerfield, C., Blunsom, P., Kavukcuoglu, K., and Hassabis, D. (2016).… find
  17. Hermann, K. M., Hill, F., Green, S., Wang, F., Faulkner, R., Soyer, H., Szepesvari, D., Czarnecki, W. M., Jaderberg, M., Teplyashin, D., Wainwright, M., Apps, C., Hassabis, D., and Blunsom, P. (2017). Grounded Language Learning in a Simulated 3d World. arXiv:1706.06551 [cs, stat].
  18. Hochreiter, S. and Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8):1735–1780. find
  19. Ioffe, S. and Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pages 448–456. find
  20. Kempka, M., Wydmuch, M., Runc, G., Toczek, J., and Jaskowski, W. (2016). ViZDoom: A Doom-based AI research platform for visual reinforcement learning. In CIG, pages 1–8. IEEE. find
  21. Kingma, D. P. and Ba, J. (2015). Adam: A Method for Stochastic Optimization. In Proceedings of the 2015 International Conference on Learning Representations. arXiv: 1412.6980.
  22. Koenig, N. and Howard, A. (2004). Design and Use Paradigms for Gazebo, An Open-Source Multi-Robot Simulator. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2149–2154, Sendai, Japan. find
  23. Kolve, E., Mottaghi, R., Gordon, D., Zhu, Y., Gupta, A., and Farhadi, A. (2017). AI2-THOR: An Interactive 3d Environment for Visual AI. CoRR, abs/1712.05474.
  24. Kumar, M. P., Packer, B., and Koller, D. (2010). Self-Paced Learning for Latent Variable Models. In Advances in Neural Information Processing Systems 23, pages 1189–1197. Curran Associates, Inc. find
  25. Macmahon, M., Stankiewicz, B., and Kuipers, B. (2006). Walk the Talk: Connecting Language, Knowledge, Action in Route Instructions. In In Proc. of the Nat. Conf. on Artificial Intelligence (AAAI, pages 1475–1482. find
  26. Mei, H., Bansal, M., and Walter, M. R. (2016). Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences. In Proceedings of the 2016 AAAI Conference on Artificial Intelligence. find
  27. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. (2011). Scikit-learn: machine learning in Python. Journal of Machine Learning… find
  28. Perez, E., Strub, F., de Vries, H., Dumoulin, V., and Courville, A. (2017). FiLM: Visual Reasoning with a General Conditioning Layer. In In Proceedings of the 2017 AAAI Conference on Artificial Intelligence.
  29. Rasmussen, C. E. and Williams, C. K. I. (2005). Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). find
  30. Reed, S. and de Freitas, N. (2015). find
  31. Neural Programmer-Interpreters. In 2016 International Conference on Learning Representations. arXiv: 1511.06279. find
  32. Ross, S., Gordon, G., and Bagnell, D. (2011). A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. In PMLR, pages 627–635. find
  33. Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. (2015). High-Dimensional Continuous Control Using Generalized Advantage Estimation. In Advances in Neural Information Processing Systems 30. find
  34. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs]. arXiv: 1707.06347. find
  35. Sukhbaatar, S., Szlam, A., Synnaeve, G., Chintala, S., and Fergus, R. (2015). MazeBase: A Sandbox for Learning from Games. arXiv:1511.07401 [cs]. arXiv: 1511.07401. find
  36. Sutskever, I., Vinyals, O., and Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems 27, pages 3104–3112.
  37. Tellex, S., Kollar, T., Dickerson, S., Walter, M. R., Banerjee, A. G., Teller, S., and Roy, N. (2011). Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation. In Twenty-Fifth AAAI Conference on Artificial Intelligence. find
  38. Wang, S. I., Liang, P., and Manning, C. D. (2016). Learning Language Games through Interaction. In Proceedings Of the 54th Annual Meeting of the Association for Computational Linguistics. arXiv: 1606.02447. find
  39. Warnell, G., Waytowich, N., Lawhern, V., and Stone, P. (2017). Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces. In Proceedings of 32nd AAAI Conference on Artificial Intelligence. arXiv: 1709.10163. — [arXiv:1709.10163](https://arxiv.org/abs/1709.10163) find
  40. Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., and Mikolov, T. (2016). Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks.
  41. Williams, E. C., Gopalan, N., Rhee, M., and Tellex, S. (2018). Learning to Parse Natural Language to Grounded Reward Functions with Weak Supervision. In 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018, pages 1–7. find
  42. Wilson, A., Fern, A., and Tadepalli, P. (2012). A Bayesian Approach for Policy Learning from Trajectory Preference Queries. In Pereira, F., Burges, C. J. C., Bottou, L., and Weinberger, K. Q., editors, Advances in Neural Information Processing Systems 25, pages 1133–1141. Curran Associates, Inc. find
  43. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., and others (2016). Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv preprint arXiv:1609.08144. find
  44. Wu, Y., Wu, Y., Gkioxari, G., and Tian, Y. (2018). Building Generalizable Agents with a Realistic and Rich 3d Environment. arXiv:1801.02209 [cs]. arXiv: 1801.02209. — [arXiv:1801.02209](https://arxiv.org/abs/1801.02209) find
  45. Yu, H., Zhang, H., and Xu, W. (2018). Interactive Grounded Language Acquisition and Generalization in 2d Environment. In ICLR. find
  46. Zaremba, W. and Sutskever, I. (2015). Learning to Execute. In 2015 International Conference on Learning Representations. arXiv: 1410.4615. find
  47. Ziebart, B. D., Maas, A., Bagnell, J. A., and Dey, A. K. (2008). Maximum Entropy Inverse Reinforcement Learning. In Proc. AAAI, pages 1433–1438. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chevalier-boisvert-2018-babyai-platform-study-sample-efficiency/