Self-Attention with Relative Position Representations
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N18-2074.pdf
Summary
The Transformer uses neither recurrence nor convolution, so it has no built-in sense of token order. The original model compensates by adding sinusoidal encodings of absolute positions to its inputs. Shaw, Uszkoreit and Vaswani instead build order information directly into self-attention. They treat the input as a fully connected, labeled, directed graph. Each edge carries learned vectors that encode the distance between two tokens, with distances clipped at a maximum offset k. These vectors are added to the keys when attention scores are computed and to the values when outputs are formed. They are also shared across heads to limit memory use. A tensor-reshaping strategy keeps training efficient, slowing it by only about 7%. On WMT 2014 English-German and English-French (newstest2014), replacing absolute encodings with relative ones improves BLEU for both base and big Transformers. Adding absolute encodings back brings no further gain. The work offers an influential alternative to absolute position embeddings and points toward attention over arbitrary labeled graphs.
Key points
- Translation results (Transformer big): EN-DE BLEU rises from 27.9 to 29.2 (+1.3) and EN-FR from 41.2 to 41.5 (+0.3). For the base model the gains are +0.3 (26.5→26.8) and +0.5 (38.2→38.7).
- Clipping distance: k=16 was used for the base model and k=8 for the big one. On the newstest2013 dev set, k=0 gives only 12.5 BLEU, while any k≥2 plateaus around 25.8–25.9.
- Ablation: relative representations on the key side alone match the full model (25.8 BLEU). Value-side vectors alone give 25.3, and removing both drops the score to 12.5.
- Efficiency: sharing representations across heads reduces their storage from O(hn²d_a) to O(n²d_a). Model and batch sizes from the original Transformer still fit on P100 GPUs.
- Setup: experiments used the tensor2tensor library, about 4.5M EN-DE and 36M EN-FR sentence pairs, and a 32,768 word-piece vocabulary.
Cited by
- The Falcon Series of Open Language Models
- Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings
- Relational inductive biases, deep learning, and graph networks
- Efficient Training of Language Models to Fill in the Middle
- APE at Scale and Its Implications on MT Evaluation Biases
- DeBERTa: Decoding-enhanced BERT with Disentangled Attention
- DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
- A Survey on Large Language Models for Code Generation
- Challenges and Applications of Large Language Models
- Large Language Models: A Survey
- Scaling Neural Machine Translation
- fairseq: A Fast, Extensible Toolkit for Sequence Modeling
- Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
- Efficient Content-Based Sparse Attention with Routing Transformers
- MPNet: Masked and Permuted Pre-training for Language Understanding
- Multimodal Transformer for Unaligned Multimodal Language Sequences
- Multimodal Few-Shot Learning with Frozen Language Models
- Graph-to-Graph Transformer for Transition-based Dependency Parsing
- Recursive Non-Autoregressive Graph-to-Graph Transformer for Dependency Parsing with Iterative Refinement
- Learning Deep Transformer Models for Machine Translation
- Pay Less Attention with Lightweight and Dynamic Convolutions
- LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding
- Big Bird: Transformers for Longer Sequences
- A Survey of Large Language Models
Cite
@inproceedings{shaw2018self,
title = {{Self-Attention with Relative Position Representations}},
author = {Shaw, Peter and Uszkoreit, Jakob and Vaswani, Ashish},
booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)},
year = {2018},
doi = {10.18653/v1/n18-2074},
eprint = {1803.02155},
archivePrefix = {arXiv},
url = {https://doi.org/10.18653/v1/n18-2074}
}References
- Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 . find
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 . find our card
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 . find our card
- Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 . find
- Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122 . find our card
- Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Koray Kavukcuoglu. 2016. Neural machine translation in linear time. arXiv preprint arXiv:1610.10099 . — [arXiv:1610.10099](https://arxiv.org/abs/1610.10099) find
- Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 . find