Elamur Research⌕
☰
home / papers / 2022 / Position Information in Transformers: An Overview

Position Information in Transformers: An Overview

Paper · 2022 · Philipp Dufter, Martin Schmitt and Hinrich Schütze

Summary

Self-attention by itself ignores token order. If you permute the input of a plain Transformer, its output is simply permuted the same way, yet word order carries much of the meaning and syntax of language. This survey collects and organizes the many proposals for giving Transformers a sense of position. The authors restate more than 30 position models in one common mathematical notation and compare them on theoretical grounds rather than through experiments. They group the models by the input structure they target (sequences, trees, graphs) and by design properties: absolute or relative positions, how position enters the network, whether the representations are trained, and how many parameters they add. The paper deliberately does not name a winner. The models were tested on too many different tasks for a fair benchmark, and absolute and relative schemes still coexist because neither has been convincingly shown to be better. The survey's value lies in one reference table and a shared vocabulary. These help practitioners pick a position encoding for their application and point researchers to neglected directions.

Key points

  • Published in Computational Linguistics (2022) by Dufter, Schmitt and Schütze (LMU Munich). It formally shows that a Transformer without position signals is permutation-equivariant.
  • Table 1 contrasts over 30 models, including BERT, T5, Transformer XL, DeBERTa, TUPE, Rotary, Shortformer and Graformer. It covers six dimensions:
    • reference point (absolute/relative/both),
    • injection method (APE, which adds embeddings, vs. MAM, which modifies the attention matrix),
    • learnability,
    • recurrence across layers,
    • extrapolation to longer inputs,
    • parameter count.
  • Learnable relative embeddings must cover positions both left and right of a token. They therefore typically need about twice as many parameters as absolute ones.
  • There is no quantitative comparison. Related architectures such as hard-coded Gaussian attention, sliding-window attention (Beltagy et al.) and the Synthesizer are discussed only as limitations.
  • Proposed future work:
    • encoding document structure (tables, headings, links),
    • position schemes for character- and byte-level models,
    • broader empirical comparisons,
    • identifying which tasks truly need order, since Wang et al. (2021) found only marginal drops without position embeddings on some tasks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Ludwig-Maximilians-Universität MünchenMunich · Germany107 works · 56,478 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Philipp Dufter, Martin Schmitt and Hinrich Schütze. 2022. Position Information in Transformers: An Overview. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00445

@article{dufter2022position,
  title = {{Position Information in Transformers: An Overview}},
  author = {Philipp Dufter, Martin Schmitt and Hinrich Schütze},
  journal = {Computational Linguistics},
  year = {2022},
  doi = {10.1162/coli\_a\_00445},
  url = {https://doi.org/10.1162/coli_a_00445}
}

References

31 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Artetxe, Mikel, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, pages 4623–4637. https://doi.org/10 .18653/v1/2020.acl-main.421
  2. Ba, Lei Jimmy, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. CoRR, abs/1607.06450.
  3. Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, Conference Track Proceedings.
  4. Bao, Yu, Hao Zhou, Jiangtao Feng, Mingxuan Wang, Shujian Huang, Jiajun Chen, and Lei Li. 2019. Non-autoregressive transformer by position learning. CoRR, abs/1911.10677. — [arXiv:1911.10677](https://arxiv.org/abs/1911.10677) find
  5. Beltagy, Iz, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. CoRR, abs/2004.05150.
  6. Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Ch… — Language Models are Few-Shot Learners
  7. Cai, Deng and Wai Lam. 2020. Graph transformer for graph-to-sequence learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 34(5):7464–7471. https://doi.org/10 .1609/aaai.v34i05.6243 Chang, Tyler, Yifan Xu, Weijian Xu, and Zhuowen Tu. 2021. Convolutions and self-attention: Re-in…
  8. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
  9. Ding, Liang, Longyue Wang, and Dacheng Tao. 2020. Self-attention with cross-lingual position representation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, pages 1679–1685. https://doi.org/10 .18653/v1/2020.acl-main.153 Dufter, Philipp. 2021. Di… find
  10. Gehring, Jonas, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, pages 1243–1252. Harris, Charles R., K. Jarrod Millman, Stéfan J. van der Walt, Ral…
  11. Howard, Jeremy and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339. https://doi.org/10.18653/v1/P18-1031 Huang, Zhiheng, Davis Liang… ↗
  12. Ke, Guolin, Di He, and Tie-Yan Liu. 2021. Rethinking positional encoding in language pre-training. In 9th International Conference on Learning Representations, ICLR 2021, OpenReview.net. find
  13. Kitaev, Nikita, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In 8th International Conference on Learning Representations, ICLR 2020, OpenReview.net.
  14. Lee, Jason, Kyunghyun Cho, and Thomas Hofmann. 2017. Fully character-level neural machine translation without explicit segmentation. Transactions of the Association for Computational Linguistics., 5:365–378. https://doi.org/10.1162/tacl a 00067 Li, Hailiang, Adele Y. C. Wang, Yang Liu, Du Tang, Zhi… ↗
  15. Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
  16. Liu, Zihan, Genta Indra Winata, Samuel Cahyawijaya, Andrea Madotto, Zhaojiang Lin, and Pascale Fung. 2021b. On the importance of word order information in cross-lingual sequence labeling. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative App… find
  17. Neishi, Masato and Naoki Yoshinaga. 2019. On the relation between position information and sentence length in neural machine translation. In Proceedings of the 23rd Conference on Computational Natural Language Learning, CoNLL 2019, pages 328–338. https://doi.org/10 .18653/v1/K19-1031 Oka, Yui, Katsu… find
  18. Raffel, Colin, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:140:1–140:67.
  19. Rosendahl, Jan, Viet Anh Khoa Tran, Weiyue Wang, and Hermann Ney. 2019. Analysis of positional encodings for neural machine translation. In Proceedings of the 16th International Conference on Spoken Language Translation, IWSLT 2019, 6 pages. Schmitt, Martin, Leonardo F. R. Ribeiro, Philipp Dufter, I…
  20. Shiv, Vighnesh Leonardo and Chris Quirk. 2019. Novel positional encodings to enable tree-based transformers. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, pages 12058–12068. find
  21. Su, Jianlin, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. RoFormer: Enhanced transformer with rotary position embedding. CoRR, abs/2104.09864.
  22. Sutskever, Ilya, James Martens, and Geoffrey E. Hinton. 2011. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011, pages 1017–1024. find
  23. Takase, Sho and Naoaki Okazaki. 2019. Positional encoding to control output sequence length. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3999–4004. https://d… find
  24. Tay, Yi, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021b. Long range arena: A benchmark for efficient transformers. In 9th International Conference on Learning Representations, ICLR 2021, OpenReview.net.
  25. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proce…
  26. Wang, Alex, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355…
  27. Wang, Benyou, Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simonsen. 2020. Encoding word order in complex embeddings. In 8th International Conference on Learning Representations, ICLR 2020, OpenReview.net. find
  28. Wang, Xing, Zhaopeng Tu, Longyue Wang, and Shuming Shi. 2019. Self-attention with structural position representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 201… find
  29. Wang, Yu-An and Yun-Nung Chen. 2020. What do position embeddings learn? An empirical study of pre-trained language model positional encoding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, pages 6840–6849. https://doi.org /10.18653/v1/2020.emnl… ↗
  30. Yang, Baosong, Longyue Wang, Derek F. Wong, Lidia S. Chao, and Zhaopeng Tu. 2019. Assessing the ability of self-attention networks to learn word order. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Volume 1: Long Papers, pages 3635–3644. https://do… find
  31. Zhu, Jie, Junhui Li, Muhua Zhu, Longhua Qian, Min Zhang, and Guodong Zhou. 2019. Modeling graph structure in Transformer for better AMR-to-text generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dufter-2022-position-information-transformers-overview/