Elamur Research⌕
☰
home / papers / 2015 / Order Matters: Sequence to sequence for sets

Order Matters: Sequence to sequence for sets

Paper · 2015 · Oriol Vinyals, Samy Bengio and Manjunath Kudlur

Summary

Seq2seq models assume that inputs and outputs arrive as ordered sequences. Many problems, such as sorting a collection of numbers or predicting the joint distribution of random variables, involve data with no obvious order. The authors show that the chosen ordering can strongly affect how well such models learn, even when no natural ordering exists. They attribute this to non-convex optimization and to how well an ordering fits the model's prior.

For unordered inputs, they introduce the Read-Process-and-Write architecture. It stores set elements in a memory and reads them through content-based attention driven by an LSTM, so the encoding does not change when elements are permuted and grows with the size of the set. For unordered outputs, they propose a training loss that searches over candidate output orderings rather than fixing one in advance.

Experiments cover sorting, graphical-model estimation, language modeling and parsing. In a 5-gram task, the model finds a good ordering on its own. The work matters for any pipeline that must linearize sets or structures before feeding them to recurrent networks.

Key points

  • Earlier work showed order effects: reversing source sentences gave Sutskever et al. (2014) a 5.0 BLEU gain in translation, and gave roughly 0.5% absolute F1 in constituency parsing.
  • Sorting points by angle before convex hull computation with Pointer Networks made training faster and raised accuracy by up to 10% absolute on the hardest instances.
  • Read-Process-and-Write combines an attention-based memory with an LSTM "process" block, yielding permutation-invariant set representations. It can be viewed as a special case of Memory Networks or Neural Turing Machines.
  • In the 5-gram language modeling task, the natural order reached a validation perplexity of 225, while a fixed shuffled order (5,1,3,4,2) gave 280. When searching over 2 orders or over all 5! orders, the model still converged to 225.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,463 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Oriol Vinyals, Samy Bengio and Manjunath Kudlur. 2015. Order Matters: Sequence to sequence for sets. arXiv:1511.06391.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1511.06391 · arXiv 1511.06391

@misc{vinyals2015order,
  title = {{Order Matters: Sequence to sequence for sets}},
  author = {Oriol Vinyals, Samy Bengio and Manjunath Kudlur},
  year = {2015},
  doi = {10.48550/arxiv.1511.06391},
  eprint = {1511.06391},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1511.06391}
}

References

23 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In Proc. ICLR, 2015a.
  2. Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., and Bengio, Y. End-to-end attention-based large vocabulary speech recognition. arXiv preprint arXiv:1508.04395, 2015b. find
  3. Bakir, G., Hofmann, T., Scholkopf, B., Smola, A. J., Taskar, B., and Vishwanathan, S.V.N. (eds.). Predicting Structured Data. MIT Press, 2007. find
  4. Chan, W., Jaitly, N., Le, Q. V., and Vinyals, O. Listen, attend and spell. arXiv, abs/1508.01211, 2015. URL http://arxiv.org/abs/1508.01211. ↗
  5. Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proc. EMNLP, 2014. — Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
  6. Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., and Darrell, T. Long-term recurrent convolutional networks for visual recognition and description. In Proc. CVPR, 2015.
  7. Graves, A. Supervised Sequence Labelling with Recurrent Neural Networks. Springer, 2012. find
  8. Graves, A., Wayne, G., and Danihelka, I. Neural turing machines. In arXiv preprint arXiv:1410.5401, 2014. find
  9. Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., and Kingsbury, B. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29:82–97, 2012. find
  10. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Computation, 9(8), 1997. find
  11. Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML, 2015. find
  12. Kalchbrenner, N. and Blunsom, P. Recurrent continuous translation models. In Proc. EMNLP, 2013.
  13. Maas, A. L., Miller, S. D., O’Neil, T. M., and Ng, A. Y. Word-level acoustic modeling with convolutional vector regression. In ICML 2012 Workshop on Representation Learning, 2012. find
  14. Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., and Yuille, A. L. Deep captioning with multimodal recurrent neural networks (m-RNN). In International Conference on Learning Representations, 2015.
  15. Socher, R., Manning, C. D., and Ng, A. Y. Learning continuous phrase representations and syntactic parsing with recursive neural networks. In Advances in Neural Information Processing Systems, 2010. find
  16. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc V. Sequence to sequence learning with neural networks. In Proc. NIPS, 2014.
  17. Vinyals, O., Fortunato, M., and Jaitly, N. Pointer networks. In Advances in Neural Information Processing Systems, NIPS, 2015a. find
  18. Vinyals, O., Kaiser, L., Koo, T., Petrov, S., Sutskever, I., and Hinton, G. Grammar as a foreign language. In Advances in Neural Information Processing Systems, 2015b. find
  19. Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. Show and tell: A neural image caption generator. In Proc. CVPR, 2015c.
  20. Weston, J., Chopra, S., and Bordes, A. Memory networks. In International Conference on Learning Representations, ICLR, 2015. find
  21. Zaremba, W. and Sutskever, I. Learning to execute. arXiv, abs/1410.4615, 2014. find
  22. Zaremba, W. and Sutskever, I. Reinforcement learning neural turing machines. arXiv, abs/1505.00521, 2015.
  23. Zaremba, W., Sutskever, I., and Vinyals, O. Recurrent neural network regularization. arXiv, abs/1409.2329, 2014.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vinyals-2015-order-matters-sequence-sequence-sets/