Elamur Research⌕
☰
home / papers / 2022 / Transformer Language Models without Positional Encodings Still Learn Positional Information

Transformer Language Models without Positional Encodings Still Learn Positional Information

Paper · 2022 · Adi Haviv, Ori Ram, Ofir Press et al.

Summary

Self-attention in transformers ignores token order, so language models usually receive explicit position information through embeddings or attention biases. Haviv et al. ask whether a causal (left-to-right) LM actually needs this signal. They train models with no positional mechanism (NoPos) and compare them with learned embeddings, sinusoidal embeddings and ALiBi. The comparison covers two settings: WikiText-103 with a 247M-parameter adaptive-input model, and a filtered subset of the Pile with GPT-3-style models from 125M to 1.3B parameters and contexts of 256–2048 tokens.

NoPos performs only slightly worse than the baselines, and the gap narrows as models grow. Layer-wise probing classifiers recover absolute token positions from NoPos hidden states about as accurately as from a model with learned position embeddings. The authors attribute this to the causal mask: each token can attend to a different number of earlier tokens, which implicitly reveals its location. Consistent with this explanation, a bidirectional masked LM without positions fails to train. The study suggests that explicit positional encoding is a useful inductive bias for decoder LMs rather than a strict requirement.

Key points

  • Validation perplexity in the Pile setting (1.3B, 1024 tokens) was NoPos 13.10, Learned 13.05, Sinusoidal 12.93 and ALiBi 12.51. On WikiText-103 (512 tokens) it was 20.97, 20.42, 20.16 and 19.71.
  • Small models benefit noticeably from Sinusoidal and ALiBi, but differences shrink at larger scale. With longer sequences, the NoPos/Learned/Sinusoidal gaps stay roughly constant while ALiBi's advantage grows.
  • The probes were 2-layer ReLU feed-forward networks trained on frozen hidden states to classify each token's position (0–1023). Accuracy is best near the start of a sequence and degrades further along.
  • NoPos really relies on the order it infers: shuffling the prefix before a target token raises its loss from about 4 to about 11.
  • A RoBERTa-large-style MLM on the Pile reached a NoPos perplexity of 147.18, compared with about 4.0–4.07 for position-aware variants. The trained models are released at github.com/adihaviv/NoPos.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States402 works · 212,763 citations🎓Tel Aviv UniversityTel Aviv · Israel89 works · 11,326 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Adi Haviv, Ori Ram, Ofir Press, Peter Izsak and Omer Levy. 2022. Transformer Language Models without Positional Encodings Still Learn Positional Information. arXiv:2203.16634.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2203.16634 · arXiv 2203.16634

@misc{haviv2022transformer,
  title = {{Transformer Language Models without Positional Encodings Still Learn Positional Information}},
  author = {Adi Haviv, Ori Ram, Ofir Press, Peter Izsak and Omer Levy},
  year = {2022},
  doi = {10.48550/arxiv.2203.16634},
  eprint = {2203.16634},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2203.16634}
}

References

24 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexei Baevski and Michael Auli. 2019. Adaptive input representations for neural language modeling. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473.
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — Language Models are Few-Shot Learners
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  5. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. find
  6. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. 2017. Convolutional sequence to sequence learning. In ICML.
  7. Kazuki Irie, Albert Zeyer, Ralf Schlüter, and Hermann Ney. 2019. Language modeling with deep transformers. In INTERSPEECH. find
  8. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
  9. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
  10. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open-Review.net.
  11. Yurii Nesterov. 1983. A method for unconstrained convex minimization problem with the rate of convergence o(1/k 2 ). find
  12. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr…
  13. Ofir Press, Noah Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations. find
  14. Ofir Press, Noah A. Smith, and Omer Levy. 2020. Improving transformer models by reordering their sublayers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2996–3005, Online. Association for Computational Linguistics. find
  15. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  16. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  17. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
  18. Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Jason Phang, Ofir Press, Colin Raffel, Victor Sanh, Sheng Shen, Lintang Sutawika, Jaesung Tae, Zheng Xin Yong, Julien Launay, and Iz Beltagy. 2022. What language model to train if…
  19. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464–468, New…
  20. Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 288…
  21. Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS’15, page 2440–2448, Cambridge, MA, USA. MIT Press. find
  22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  23. Elena Voita, Rico Sennrich, and Ivan Titov. 2019. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Join…
  24. NoPos Performance Across Different Segments of the Input A find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/haviv-2022-transformer-language-models-positional-encodings/