Elamur Research⌕
☰
home / papers / 2018 / Modeling Localness for Self-Attention Networks

Modeling Localness for Self-Attention Networks

Paper · 2018 · Baosong Yang, Zhaopeng Tu, Derek F. Wong et al.

Summary

Self-attention in the Transformer links every token directly to all other tokens. This handles long-distance dependencies well, but it spreads attention weight thinly, so relations between neighbouring words can get lost. The authors add a learnable Gaussian bias to the attention logits before the softmax. For each query, the model predicts a centre position and a window size, which sets the standard deviation. Weights near the centre are boosted and distant ones are damped.

The window can take three forms:

  • a fixed constant,
  • one value per layer, derived from the mean of the keys,
  • a separate value for every query.

Each attention head receives its own bias. The bias is applied only to the lower encoder layers, where short-range patterns dominate, so the upper layers stay free to model global context.

On WMT14 English⇒German and WMT17 Chinese⇒English, the method consistently improves over a strong Transformer baseline. It offers a cheap, task-independent way to inject locality into attention, and it combines well with relative position encoding.

Key points

  • The Gaussian bias is added to the dot-product energy. Its centre and window are scaled by a sigmoid to the sentence length. The fixed-window variant uses D = 10, following Luong et al. (2015).
  • The query-specific window gave the best results. It shares the projection W_p with the centre-position predictor, which keeps it lighter than the layer-specific variant.
  • The baseline Transformer has 6 encoder and 6 decoder layers. Restricting localness to the lower layers preserves long-range modelling. Localness helps more with multi-head attention than with a single head.
  • The method is complementary to the relative position encoding of Shaw et al. (2018), and combining the two adds further gains.
  • N-gram analysis shows slight losses on unigrams but gains on longer n-grams, with the query-specific variant strongest on 4–8-grams. Visualisation shows that learned windows widen in higher layers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢TencentShenzhen · China66 works · 17,945 citations🎓University of MacauMacao · China10 works · 1,905 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Baosong Yang, Zhaopeng Tu, Derek F. Wong, Fandong Meng, Lidia S. Chao and Tong Zhang. 2018. Modeling Localness for Self-Attention Networks. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1475

@inproceedings{yang2018modeling,
  title = {{Modeling Localness for Self-Attention Networks}},
  author = {Baosong Yang, Zhaopeng Tu, Derek F. Wong, Fandong Meng, Lidia S. Chao and Tong Zhang},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1475},
  url = {https://doi.org/10.18653/v1/d18-1475}
}

References

27 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Antonios Anastasopoulos and David Chiang. 2018. Tied Multitask Learning for Neural Speech Translation. In NAACL. our card
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In ICLR. our card
  3. Michał Daniluk, Tim Rocktäschel, Johannes Welbl, and Sebastian Riedel. 2017. Frustratingly short attention spans in neural language modeling. In ICLR. find
  4. Ziyi Dou, Zhaopeng Tu, Xing Wang, Shuming Shi, and Tong Zhang. 2018. Exploiting Deep Representations for Neural Machine Translation. In EMNLP. find
  5. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional Sequence to Sequence Learning. In ICML. our card
  6. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, et al. 2018. Achieving Human Parity on Automatic Chinese to English News Translation. arXiv:1803.05567. our card
  7. Po Sen Huang, Chong Wang, Sitao Huang, Dengyong Zhou, and Li Deng. 2018. Towards Neural Phrase-based Machine Translation. In ICLR. find
  8. Jinbae Im and Sungzoon Cho. 2017. Distance-based Self-Attention Network for Natural Language Inference. arXiv:1712.02047. — [arXiv:1712.02047](https://arxiv.org/abs/1712.02047) find
  9. Philipp Koehn. 2004. Statistical Significance Tests for Machine Translation Evaluation. In EMNLP. our card
  10. Jian Li, Zhaopeng Tu, Baosong Yang, Michael R. Lyu, and Tong Zhang. 2018. Multi-Head Attention with Disagreement Regularization. In EMNLP. find
  11. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. In EMNLP. our card
  12. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In ACL. our card
  13. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In NAACL. our card
  14. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In ACL. our card
  15. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-Attention with Relative Position Representations. In NAACL. our card
  16. Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. 2018a. DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding. In AAAI. find
  17. Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. 2018b. Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling. In ICLR. find
  18. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does String-based Neural MT Learn Source Syntax? In EMNLP. our card
  19. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s Neural Machine Translation System: Bridging the Gap Between Human and Machine Translation. arXiv:1609.08144. find
  20. Chang Xu, Cecile Paris, Surya Nepal, and Ross Sparks. 2018. Cross-Target Stance Classification with Self-Attention Networks. In ACL. find
  21. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, and Yoshua Bengio. 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In ICML. our card
  22. Matthias Sperber, Jan Niehues, Graham Neubig, Sebastian Stüker, and Alex Waibel. 2018. Self-Attentional Acoustic Models. In Interspeech. find
  23. Baosong Yang, Derek F. Wong, Tong Xiao, Lidia S. Chao, and Jingbo Zhu. 2017. Towards Bidirectional Hierarchical Representations for Attention-based Neural Machine Translation. In EMNLP. find
  24. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. In NIPS. our card
  25. Biao Zhang, Deyi Xiong, and Jinsong Su. 2018. Accelerating Neural Transformer via an Average Attention Network. In ACL. find
  26. Xing Wang, Zhaopeng Tu, Deyi Xiong, and Min Zhang. 2017. Translating Phrases in Neural Machine Translation. In EMNLP. find
  27. Jiacheng Zhang, Yanzhuo Ding, Shiqi Shen, Yong Cheng, Maosong Sun, Huanbo Luan, and Yang Liu. 2017. THUMT: An Open Source Toolkit for Neural Machine Translation. arXiv:1706.06415. — [arXiv:1706.06415](https://arxiv.org/abs/1706.06415) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2018-modeling-localness-self-attention-networks/