Elamur Research⌕
☰
home / papers / 2015 / Neural Random-Access Machines

Neural Random-Access Machines

Paper · 2015 · Karol Kurach, Marcin Andrychowicz and Ilya Sutskever

Summary

Kurach, Andrychowicz and Sutskever (Google) ask whether a neural network can learn pointer-based algorithms, which store addresses, follow them and edit memory, using only input-output pairs and backpropagation. They propose the Neural Random-Access Machine (NRAM). Its trainable controller, a feedforward net or an LSTM, never touches data directly. At each timestep it wires a soft "circuit" from fixed integer modules such as addition, comparisons, min/max, READ and WRITE. Registers and memory cells hold probability distributions over {0,…,M−1}, so any value can serve as a fuzzy address while the whole system stays differentiable. The memory tape also carries the input and receives the output. With curriculum training on synthetic tasks such as Copy, Reverse, Swap, Merge and walking a binary search tree, NRAM learns working procedures on linked lists and trees. For simpler tasks, the learned solutions extend to sequences of any length. The work shows that built-in primitives strongly shape what deep memory-augmented models can learn in practice. It also shows how hard such deep, highly nonlinear models are to optimize.

Key points

  • Architecture: R registers are combined through Q prespecified modules. Module inputs and register updates are softmax-weighted mixtures chosen by the controller. Modules are extended to distributions by summing over all input pairs.
  • Controller input: The controller sees only one scalar per register, the probability that its value is 0. This keeps the parameter count independent of M, which lets the model handle different memory sizes. It also pushes the controller to rely on the modules.
  • Memory: The tape is an M×M probability matrix accessed solely through READ (computed as Mᵀp) and WRITE. The authors note that inference-time memory access can be constant-time under certain assumptions.
  • Tasks and circuits: Tasks include Access, Increment, Copy, Reverse, Swap, Merge and WalkBST. In the extracted circuits, the first timestep handles initialization and later steps repeat one fixed circuit. Swap is the exception, alternating between two circuits on even and odd steps.
  • Training: The curriculum follows Zaremba & Sutskever (2014). Difficulty is sampled uniformly at random 10% of the time, from [1, D+e] 25% of the time, and set to D+e 65% of the time, with e geometric (p = 1/2). The authors report that standard optimizers struggle with these very deep models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States533 works · 443,885 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksComputational Theory and Mathematics · 496 worksBiochemistry, Genetics and Molecular Biology · 273 worksMolecular Biology · 270 works

Cite

Cite as Karol Kurach, Marcin Andrychowicz and Ilya Sutskever. 2015. Neural Random-Access Machines. arXiv:1511.06392.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1511.06392 · arXiv 1511.06392

@misc{kurach2015neural,
  title = {{Neural Random-Access Machines}},
  author = {Karol Kurach, Marcin Andrychowicz and Ilya Sutskever},
  year = {2015},
  doi = {10.48550/arxiv.1511.06392},
  eprint = {1511.06392},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1511.06392}
}

References

21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
  2. Bengio, Yoshua, Simard, Patrice, and Frasconi, Paolo. Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on, 5(2):157–166, 1994. find
  3. Bengio, Yoshua, Louradour, Jérôme, Collobert, Ronan, and Weston, Jason. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41–48. ACM, 2009.
  4. Chan, William, Jaitly, Navdeep, Le, Quoc V, and Vinyals, Oriol. Listen, attend and spell. arXiv preprint arXiv:1508.01211, 2015. find
  5. Graves, Alex, Wayne, Greg, and Danihelka, Ivo. arXiv:1410.5401, 2014. find
  6. Grefenstette, Edward, Hermann, Karl Moritz, Suleyman, Mustafa, and Blunsom, Phil. Learning to transduce with unbounded memory. arXiv preprint arXiv:1506.02516, 2015. find
  7. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. find
  8. Joulin, Armand and Mikolov, Tomas. Inferring algorithmic patterns with stack-augmented recurrent nets. arXiv preprint arXiv:1503.01007, 2015. find
  9. Kalchbrenner, Nal, Danihelka, Ivo, and Graves, Alex. Grid long short-term memory. arXiv preprint arXiv:1507.01526, 2015. find
  10. Kingma, Diederik and Ba, Jimmy. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  11. Luong, Minh-Thang, Pham, Hieu, and Manning, Christopher D. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025, 2015.
  12. Nair, Vinod and Hinton, Geoffrey E. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 807– 814, 2010. find
  13. Neelakantan, Arvind, Vilnis, Luke, Le, Quoc V, Sutskever, Ilya, Kaiser, Lukasz, Kurach, Karol, and Martens, James. Adding gradient noise improves learning for very deep networks. arXiv preprint arXiv:1511.06807, 2015. find
  14. Solomonoff, Ray J. A formal theory of inductive inference. part i. Information and control, 7(1): 1–22, 1964. find
  15. Sukhbaatar, Sainbayar, Szlam, Arthur, Weston, Jason, and Fergus, Rob. End-to-end memory networks. arXiv preprint arXiv:1503.08895, 2015. find
  16. Vinyals, Oriol, Kaiser, Lukasz, Koo, Terry, Petrov, Slav, Sutskever, Ilya, and Hinton, Geoffrey. Grammar as a foreign language. arXiv preprint arXiv:1412.7449, 2014. find
  17. Vinyals, Oriol, Fortunato, Meire, and Jaitly, Navdeep. arXiv:1506.03134, 2015. find
  18. Weston, Jason, Chopra, Sumit, and Bordes, Antoine. arXiv:1410.3916, 2014. find
  19. Zaremba, Wojciech and Sutskever, Ilya. Learning to execute. arXiv preprint arXiv:1410.4615, 2014. find
  20. Zaremba, Wojciech and Sutskever, Ilya. Reinforcement learning neural turing machines. arXiv preprint arXiv:1505.00521, 2015.
  21. Neural turing machines. arXiv preprint Pointer networks. arXiv preprint Memory networks. arXiv preprint find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kurach-2015-neural-random-access-machines/