Elamur Research⌕
☰
home / papers / 2026 / Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks

Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks

Paper · 2026 · Yuval Ran-Milo

Summary

Softmax Transformers often put almost all attention on one fixed, content-independent position, usually BOS. This "attention sink" matters in practice because it affects quantization, interpretability and streaming inference. The paper asks whether sinks are simply a habit picked up during training, or whether some computations actually require them.

To answer this, Ran-Milo designs a minimal synthetic task based on real "fire or stay dormant" heads seen in Gemma 7B and Llama 2–7B. At one designated trigger token, the model must output the mean of all earlier non-BOS tokens. Everywhere else it must output the zero vector.

The paper proves three results:

  • Single-layer softmax: any model with vanishing error must put attention arbitrarily close to 1 on BOS at every non-trigger position.
  • Multi-layer softmax: at least one layer must show sink behaviour somewhere.
  • ReLU attention: a model without normalization can solve the task exactly while giving BOS zero attention.

Small trained models confirm these predictions. The conclusion is that normalization onto the probability simplex is what forces sinks. Removing them therefore means changing how attention represents an "off" state, for example through gating or non-normalized attention.

Key points

  • Task setup: each token carries a BOS flag, a trigger flag, a flag for ordinary tokens, and i.i.d. continuous content values. The trigger position is drawn uniformly from {2,…,L}, with n ≥ 5 dimensions and L ≥ 4 positions. Error is measured with a worst-case ℓ∞ loss.
  • Theorems: Theorem 1 covers single-layer softmax and requires a near-total BOS sink at all non-trigger positions. Theorem 2 covers multi-layer models and guarantees only that a sink exists somewhere. Theorem 3 gives an explicit sink-free ReLU construction.
  • Single-layer experiments: across 1000 test examples with the trigger at position 8, softmax models put almost all attention mass on BOS, with negligible variance. ReLU models reach comparable accuracy while BOS attention stays close to zero.
  • Deeper models: a 2-layer, 2-head softmax model shows strong sinks in every head on a random input. Elsewhere the paper notes that sinks do not form in all positions and layers, which fits the existence-only claim of Theorem 2.
  • Venue: ACL 2026 Short Papers, pp. 70–89. An extended version is available as arXiv 2603.11487.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tel Aviv UniversityTel Aviv · Israel82 works · 10,655 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Yuval Ran-Milo. 2026. Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-short.8

@inproceedings{ranmilo2026attention,
  title = {{Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks}},
  author = {Yuval Ran-Milo},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-short.8},
  url = {https://doi.org/10.18653/v1/2026.acl-short.8}
}

References

31 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Victoria Hankemeier and Malte Schilling. 2026. Stochastic parroting in temporal attention – regulating the diagonal sink. Preprint, arXiv:2602.10956. — [arXiv:2602.10956](https://arxiv.org/abs/2602.10956) find
  2. Anand, Umberto Cappellazzo, Stavros Petridis, and Maja Pantic. 2026. Mitigating attention sinks and Jonghyun Hong and Sungyoon Lee. 2025. Variance sensitivity induces attention entropy collapse and instability in transformers. In Proceedings of the 2025 Conference on Empirical Methods in Natural Lan… find
  3. Ofir Press, Noah A. Smith, and Mike Lewis. 2021. Train short, test long: Attention with linear biases enables input length extrapolation. Preprint, arXiv:2108.12409. find
  4. Zihan Qiu, Zeyu Huang, Kaiyue Wen, Peng Jin, Bo Zheng, Yuxin Zhou, Haofeng Huang, Zekun Wang, Xiao Li, Huaqing Zhang, Yang Xu, Haoran Lian, Siqi Zhang, Rui Men, Jianwei Zhang, Ivan Titov, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2026. A unified view of attention and residual sinks: Outlier-drive… — [arXiv:2601.22966](https://arxiv.org/abs/2601.22966) find
  5. Sayed Mohammadreza Tayaranian Hosseini, Amir Ardakani, and Warren J. Gross. 2026. In-nerq: Hardware-aware tuning-free quantization of kv cache for large language models. Preprint, arXiv:2602.23200. find
  6. Xingyue Huang, Xueying Ding, Mingxuan Ju, Yozen Liu, Neil Shah, and Tong Zhao. 2026. Threshold differential attention for sink-free, ultra-sparse, and non-dispersive language modeling. Preprint, arXiv:2601.12145. find
  7. Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. Preprint, arXiv:2505.06708. — [arXiv:2505.06708](https://arxiv.org/abs/2505.06708) find
  8. Mingyu Jin, Kai Mei, Wujiang Xu, Mingjie Sun, Ruixiang Tang, Mengnan Du, Zirui Liu, and Yongfeng Zhang. 2025. Massive values in self-attention modules are the key to contextual knowledge understanding. Preprint, arXiv:2502.01563. — [arXiv:2502.01563](https://arxiv.org/abs/2502.01563) find
  9. Yuval Ran-Milo, Yotam Alexander, Shahar Mendel, and Nadav Cohen. 2026. Outcome-based rl provably leads transformers to reason, but only with the right data. Preprint, arXiv:2601.15158. — [arXiv:2601.15158](https://arxiv.org/abs/2601.15158) find
  10. Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. 2025. See what you are told: Visual attention sink in large multimodal models. ArXiv, abs/2503.03321. — [arXiv:2503.03321](https://arxiv.org/abs/2503.03321) find
  11. Oliver Richter and Roger Wattenhofer. 2020. Normalized attention without probability cage. Preprint, arXiv:2005.09561. — [arXiv:2005.09561](https://arxiv.org/abs/2005.09561) find
  12. Maximo Eduardo Rulli, Simone Petruzzi, Edoardo Michielon, Fabrizio Silvestri, Simone Scardapane, and Alessio Devoto. 2025. Attention sinks in diffusion language models. Preprint, arXiv:2510.15731.
  13. Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. 2024. Duquant: Distributing outliers via dual transformation makes stronger quantized llms. Preprint, arXiv:2406.01721. — [arXiv:2406.01721](https://arxiv.org/abs/2406.01721) find
  14. Valeria Ruscio, Umberto Nanni, and Fabrizio Silvestri. 2025. What are you sinking? a geometric approach on attention sink. Preprint, arXiv:2508.02546. — [arXiv:2508.02546](https://arxiv.org/abs/2508.02546) find
  15. Ziyong Lin, Haoyi Wu, Shu Wang, Kewei Tu, Zilong Zheng, and Zixia Jia. 2025. Look both ways and no sink: Converting LLMs into text encoders without training. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 22839–22853, Vienna,… find
  16. Pedro Sandoval-Segura, Xijun Wang, Ashwinee Panda, Micah Goldblum, Ronen Basri, Tom Goldstein, and David Jacobs. 2025. Using attention sinks to identify and evaluate dormant heads in pretrained llms. arXiv preprint arXiv:2504.03889. — [arXiv:2504.03889](https://arxiv.org/abs/2504.03889) find
  17. Bingqi Shang, Yiwei Chen, Yihua Zhang, Bingquan Shen, and Sijia Liu. 2025. Forgetting to forget: Attention sink as a gateway for backdooring llm unlearning. Preprint, arXiv:2510.17021. — [arXiv:2510.17021](https://arxiv.org/abs/2510.17021) find
  18. Guozhi Liu, Weiwei Lin, Tiansheng Huang, Ruichao Mo, Qi Mu, Xiumin Wang, and Li Shen. 2026. Surgery: Mitigating harmful fine-tuning for large language models via attention sink. Preprint, arXiv:2602.05228. — [arXiv:2602.05228](https://arxiv.org/abs/2602.05228) find
  19. Jaewon Sok, Jewon Yeom, Seonghyeon Park, Jeongjae Park, and Taesup Kim. 2026. Garbage attention in large language models: Bos sink heads and sink-aware pruning. Preprint, arXiv:2601.06787. — [arXiv:2601.06787](https://arxiv.org/abs/2601.06787) find
  20. Andrew Lu, Wentinn Liao, Liuhui Wang, Huzheng Yang, and Jianbo Shi. 2025. Artifacts and attention sinks: Structured approximations for efficient vision transformers. Preprint, arXiv:2507.16018. — [arXiv:2507.16018](https://arxiv.org/abs/2507.16018) find
  21. Jiayun Luo, Wan-Cyuan Fan, Lyuyang Wang, Xiangteng He, Tanzila Rahman, Purang Abolmaesumi, and Leonid Sigal. 2025. To sink or not to sink: Visual information pathways in large vision-language models. Preprint, arXiv:2510.08510. — [arXiv:2510.08510](https://arxiv.org/abs/2510.08510) find
  22. Seungwoo Son, Wonpyo Park, Woohyun Han, Kyuyeun Kim, and Jaeho Lee. 2024. Prefixing attention sinks can mitigate activation outliers for large language model quantization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2242–2252, Miami, Florida, USA.… find
  23. Aidar Myrzakhan, Tianyi Li, Bowei Guo, Shengkun Tang, and Zhiqiang Shen. 2026. Sink-aware pruning for diffusion language models. Preprint, arXiv:2602.17664. — [arXiv:2602.17664](https://arxiv.org/abs/2602.17664) find
  24. Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. 2024. The implicit bias of gradient descent on separable data. Preprint, arXiv:1710.10345. — [arXiv:1710.10345](https://arxiv.org/abs/1710.10345) find
  25. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2021. Roformer: Enhanced transformer with rotary position embedding. Preprint, arXiv:2104.09864.
  26. Xiaofeng Zhang, Yihao Quan, Chaochen Gu, Chen Shen, Xiaosong Yuan, Shaotian Yan, Hao Cheng, Kaijie Wu, and Jieping Ye. 2024. Seeing clearly by layer two: Enhancing attention heads to alleviate hallucination in lvlms. Preprint, arXiv:2411.09968. — [arXiv:2411.09968](https://arxiv.org/abs/2411.09968) find
  27. Zunhai Su and Kehong Yuan. 2025. Kvsink: Understanding and enhancing the preservation of attention sinks in kv cache quantization for llms. Preprint, arXiv:2508.04257. — [arXiv:2508.04257](https://arxiv.org/abs/2508.04257) find
  28. Zihou Zhang, Zheyong Xie, Li Zhong, Haifeng Liu, Yao Hu, and Shaosheng Cao. 2026. One token is enough: Improving diffusion language models with a sink token. Preprint, arXiv:2601.19657.
  29. Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. 2024. Massive activations in large language models. Preprint, arXiv:2402.17762. find
  30. Zayd M. K. Zuhri, Erland Hilman Fuadi, and Alham Fikri Aji. 2026. Softpick: No attention sink, no massive activations with rectified softmax. Preprint, arXiv:2504.20966.
  31. Shangwen Sun, Alfredo Canziani, Yann LeCun, and Jiachen Zhu. 2026. The spike, the sparse and the sink: Anatomy of massive activations and attention sinks. Preprint, arXiv:2603.05498. — [arXiv:2603.05498](https://arxiv.org/abs/2603.05498) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/milo-2026-attention-sinks-are-provably-necessary/