Elamur Research⌕
☰
home / papers / 2015 / Reinforcement Learning Neural Turing Machines - Revised

Reinforcement Learning Neural Turing Machines - Revised

Paper · 2015 · Wojciech Zaremba and Ilya Sutskever

Summary

Neural Turing Machines extend neural networks with external memory, but that memory is continuous and differentiable. Many useful external tools, such as databases or search engines, accept only discrete actions, and the cost of continuous memory access grows with memory size. This paper asks whether a network can learn to operate discrete interfaces. The authors present the RL–NTM, a controller connected to three discrete tapes: input, memory, and write-only output. Head movements and the decision to emit a symbol are trained with the Reinforce algorithm. Memory contents and output predictions are trained with ordinary backpropagation. With an unbounded memory and a self-timed output, the system is Turing complete. On simple algorithmic tasks (reversal, repeated copying, ForwardReverse), the model learns correct solutions. Success depends heavily on the controller architecture. The work also contributes practical tools for Reinforce, including a numerical gradient check and a learned baseline. These matter for future models that must interact with non-differentiable external systems.

Key points

  • The training split is hybrid: Reinforce learns where to move the heads on the input and memory tapes (moves of −1, 0 or +1) and whether to predict. Backpropagation learns what to write to memory and output.
  • The total number of steps is capped. Once the remaining steps equal the remaining outputs, the model is forced to emit predictions.
  • Controller variants include a plain LSTM, a "direct access" LSTM and a feed-forward network. Results are highly sensitive to this choice. For example, RepeatCopy fails when the input head can only move forward, because the memory pointer then wanders randomly.
  • To reduce variance in Reinforce, a separate baseline LSTM first reads the entire input tape and then predicts a per-step baseline. This gives a biased estimator but works well in practice.
  • The authors propose a general procedure for numerically checking Reinforce gradients. Code is available at github.com/ilyasu123/rlntm.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,463 citations🏢MetaMenlo Park · United States214 works · 241,492 citations🎓New York UniversityNew York · United States219 works · 103,487 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputational Theory and Mathematics · 321 works

Cite

Cite as Wojciech Zaremba and Ilya Sutskever. 2015. Reinforcement Learning Neural Turing Machines - Revised. arXiv:1505.00521.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1505.00521 · arXiv 1505.00521

@misc{zaremba2015reinforcement,
  title = {{Reinforcement Learning Neural Turing Machines - Revised}},
  author = {Wojciech Zaremba and Ilya Sutskever},
  year = {2015},
  doi = {10.48550/arxiv.1505.00521},
  eprint = {1505.00521},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1505.00521}
}

References

17 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aberdeen, Douglas and Baxter, Jonathan. Scaling internal-state policy-gradient methods for pomdps. In MACHINE LEARNING-INTERNATIONAL WORKSHOP THEN CONFERENCE-, pp. 3–10, 2002. find
  2. Ba, Jimmy, Mnih, Volodymyr, and Kavukcuoglu, Koray. Multiple object recognition with visual attention. arXiv preprint arXiv:1412.7755, 2014. find
  3. Bengio, Yoshua, Louradour, Jérôme, Collobert, Ronan, and Weston, Jason. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41–48. ACM, 2009.
  4. Graves, Alex, Wayne, Greg, and Danihelka, Ivo. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014a. find
  5. Graves, Alex, Wayne, Greg, and Danihelka, Ivo. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014b. find
  6. Grefenstette, Edward, Hermann, Karl Moritz, Suleyman, Mustafa, and Blunsom, Phil. Learning to transduce with unbounded memory. arXiv preprint arXiv:1506.02516, 2015. find
  7. Joulin, Armand and Mikolov, Tomas. Inferring algorithmic patterns with stack-augmented recurrent nets. arXiv preprint arXiv:1503.01007, 2015. find
  8. Kohl, Nate and Stone, Peter. Policy gradient reinforcement learning for fast quadrupedal locomotion. In Robotics and Automation, 2004. Proceedings. ICRA’04. 2004 IEEE International Conference on, volume 3, pp. 2619– 2624. find
  9. Levine, Sergey, Finn, Chelsea, Darrell, Trevor, and Abbeel, Pieter. End-to-end training of deep visuomotor policies. arXiv preprint arXiv:1504.00702, 2015. find
  10. Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Graves, Alex, Antonoglou, Ioannis, Wierstra, Daan, and Riedmiller, Martin. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. find
  11. Mnih, Volodymyr, Heess, Nicolas, Graves, Alex, et al. Recurrent models of visual attention. In Advances in Neural Information Processing Systems, pp. 2204–2212, 2014. find
  12. Peters, Jan and Schaal, Stefan. Policy gradient methods for robotics. In Intelligent Robots and Systems, 2006 IEEE/RSJ International Conference on, pp. 2219–2225. find
  13. Schmidhuber, Juergen. Self-delimiting neural networks. arXiv preprint arXiv:1210.0118, 2012. — [arXiv:1210.0118](https://arxiv.org/abs/1210.0118) find
  14. Schmidhuber, Jürgen. Optimal ordered problem solver. Machine Learning, 54(3):211–254, 2004. find
  15. Sukhbaatar, Sainbayar, Szlam, Arthur, Weston, Jason, and Fergus, Rob. Weakly supervised memory networks. arXiv preprint arXiv:1503.08895, 2015. find
  16. Williams, Ronald J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992. find
  17. Zaremba, Wojciech and Sutskever, Ilya. Learning to execute. arXiv preprint arXiv:1410.4615, 2014. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zaremba-2015-reinforcement-learning-neural-turing-machines/