Elamur Research⌕
☰
home / papers / 2026 / Softpick: No Attention Sink, No Massive Activations with Rectified Softmax

Softpick: No Attention Sink, No Massive Activations with Rectified Softmax

Paper · 2026 · Zayd Muhammad Kawakibi Zuhri, Erland Hilman Fuadi and Alham Fikri Aji

Summary

Softmax attention forces each head's scores to add up to one. This pushes heads to dump weight on uninformative tokens such as BOS, a pattern known as attention sinks. It also produces extremely large hidden-state values, called massive activations, which make quantization and low-precision training difficult. The authors propose softpick, which swaps softmax for a rectified function. Its numerator is ReLU(e^x − 1) and its denominator sums the absolute values |e^x − 1|. Because of this asymmetry, outputs can be exactly zero and no longer need to total one. Negative scores still pass gradients, and the function adds no parameters and works with FlashAttention-2. Llama-style models with 340M and 1.8B parameters were trained from scratch on fineweb-edu, each with a softmax and a softpick variant. Softpick removed sinks entirely and gave sparser attention maps and hidden states with much lower kurtosis. Its quantized models beat softmax, especially at low bit widths. At 340M it roughly matched softmax quality, but at 1.8B it fell behind. That gap matters for anyone weighing it for quantization, pruning or interpretability work.

Key points

  • Training setup: the 340M models saw 52B tokens and the 1.8B models 104B tokens from the fineweb-edu 100B subset, trained on 8×H100 GPUs with the flash-linear-attention / Flame framework.
  • Sinks and loss at 340M: the sink rate fell from 63.41% with softmax to 0.00% with softpick. The final training loss differed by only 0.004.
  • Quantization: quantized softpick models outperformed softmax, with the largest advantage at the lowest precisions, down to 2-bit.
  • Weakness at 1.8B:
    • Wikitext word perplexity rose from 15.10 to 17.87.
    • Arc Easy accuracy dropped from 67.21 to 62.04.
    • The authors offer two explanations: long-context underscoring and "dead" heads.
  • Dead heads: their share declined over training, from 17.19% to 9.11% at 340M and from 42.87% to 20.90% at 1.8B. The authors suggest training on more tokens could help. Passkey-style long-context retrieval did not improve.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Zayd Muhammad Kawakibi Zuhri, Erland Hilman Fuadi and Alham Fikri Aji. 2026. Softpick: No Attention Sink, No Massive Activations with Rectified Softmax. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.905

@inproceedings{zuhri2026softpick,
  title = {{Softpick: No Attention Sink, No Massive Activations with Rectified Softmax}},
  author = {Zayd Muhammad Kawakibi Zuhri, Erland Hilman Fuadi and Alham Fikri Aji},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.905},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.905}
}

References

49 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samira Abnar and Willem Zuidema. 2020. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190–4197, Online. Association for Computational Linguistics.
  2. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, 18187 9 Che Chang, Kai Chen, and 106 others. 2025. gpt-oss-120b &…
  3. Hicham Badri and Appu Shaji. 2023. HQQ: Half-quadratic quantization for large language models (blog post). https://mobiusml.github.io/hqq_ blog/. Accessed: 2025-12-30. ↗
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
  5. Federico Barbero, Álvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, and Razvan Pascanu. 2025. Why do LLMs attend to the first token? Preprint, arXiv:2504.02732. — [arXiv:2504.02732](https://arxiv.org/abs/2504.02732) find
  6. Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference,…
  7. Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. 2023. Quantizable transformers: Removing outliers by helping attention heads do nothing. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA,… find
  8. David Chiang. 2025. Transformers in uniform tc 0 . Preprint, arXiv:2409.13629. — [arXiv:2409.13629](https://arxiv.org/abs/2409.13629) find
  9. Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. 2024. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. find
  10. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. LLM.int8(): 8-bit matrix multiplication for transformers at scale. find
  11. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient fine-tuning of quantized llms. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 20… find
  12. Maxim Fishman, Brian Chmiel, Ron Banner, and Daniel Soudry. 2024. Scaling FP8 training to trillion-token llms. ArXiv preprint, abs/2409.12517. find
  13. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. GPTQ: Accurate post-training quantization for generative pre-trained transformers. find
  14. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… find
  15. Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. 2024. When attention sink emerges in language models: An empirical view. Preprint, arXiv:2410.10781. — [arXiv:2410.10781](https://arxiv.org/abs/2410.10781) find
  16. Tianyu Guo, Druv Pai, Yu Bai, Jiantao Jiao, Michael I. Jordan, and Song Mei. 2024. Active-dormant attention heads: Mechanistically demystifying extreme-token phenomena in llms. Preprint, arXiv:2410.13835. — [arXiv:2410.13835](https://arxiv.org/abs/2410.13835) find
  17. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try ARC, the AI2 reasoning challenge. Preprint, arXiv:1803.05457.
  18. Prannay Kaul, Chengcheng Ma, Ismail Elezi, and Jiankang Deng. 2024. From attention to activation: Unravelling the enigmas of large language models. Preprint, arXiv:2410.17174. — [arXiv:2410.17174](https://arxiv.org/abs/2410.17174) find
  19. Tri Dao. 2024. FlashAttention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
  20. Shashi Kumar, Srikanth R. Madikeri, Juan Pablo Zuluaga, Esaú Villatoro-Tello, Iuliia Nigmatulina, Petr Motlícek, E ManjunathK, and Aravind Ganapathiraju. 2024. XLSR-Transducer: Streaming asr for self-supervised pretrained models. find
  21. ICASSP 2025 -2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). find
  22. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans… find
  23. Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjie Wang, Sanket Purandare, Gokul Nadathur, and Stratos Idreos. 2025. TorchTitan: One-stop pytorch native solution for production ready LLM pretraining. In The Thirteenth 18188… find
  24. International Conference on Learning Representations. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, and 180 others. 20… find
  25. Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. 2024. FineWeb-Edu: the finest collection of educational content. find
  26. Yi Lu, Xin Zhou, Wei He, Jun Zhao, Tao Ji, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. LongHeads: Multi-head attention is secretly a long context processor. Preprint, arXiv:2402.10685. find
  27. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open-Review.net.
  28. Meta. 2025. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. find
  29. Evan Miller. 2023. Attention is off by one. find
  30. Ken M. Nakanishi. 2025. Scalable-softmax is superior for attention. Preprint, arXiv:2501.19399. — [arXiv:2501.19399](https://arxiv.org/abs/2501.19399) find
  31. Louis Owen, Nilabhra Roy Chowdhury, Abhay Kumar, and Fabian Güra. 2025. A refined analysis of massive activations in llms. Preprint, arXiv:2503.22329. — [arXiv:2503.22329](https://arxiv.org/abs/2503.22329) find
  32. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association fo…
  33. Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. In The Thirty-ninth Annual Conference on Neu… find
  34. Hemanth Saratchandran, Jianqiao Zheng, Yiping Ji, Wenbo Zhang, and Simon Lucey. 2024. Rethinking softmax: Self-attention with polynomial activations. Preprint, arXiv:2410.18613. — [arXiv:2410.18613](https://arxiv.org/abs/2410.18613) find
  35. Kyuhong Shim, Iksoo Choi, Wonyong Sung, and Jungwook Choi. 2021. Layer-wise pruning of transformer attention heads for efficient language modeling. In 2021 18th International SoC Design Conference (ISOCC), page 357–358. IEEE. find
  36. Ethan Smith. 2025. Softmax attention is a fluke. find
  37. Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. find
  38. Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2440–2448. find
  39. Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. 2024. Massive activations in large language models. Preprint, arXiv:2402.17762.
  40. Philippe Tillet, H. T. Kung, and David Cox. 2019. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA. Association for…
  41. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017,…
  42. Hanrui Wang, Zhekai Zhang, and Song Han. 2021. SpAtten: Efficient sparse attention architecture with cascade token and head pruning. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE.
  43. Hongyu Wang, Shuming Ma, and Furu Wei. 2025. Bit-Net v2: Native 4-bit activations with hadamard transformation for 1-bit llms. find
  44. Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106, Copenhagen, Denmark. Association for Computational Linguistics.
  45. Yuxin Wen, Jim Wu, Ajay Jain, Tom Goldstein, and Ashwinee Panda. 2025. Analysis of attention in video diffusion transformers. Preprint, arXiv:2504.10317. — [arXiv:2504.10317](https://arxiv.org/abs/2504.10317) find
  46. Guangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. SmoothQuant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings… find
  47. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. In The Twelfth 18189 11 find
  48. International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. find
  49. Songlin Yang and Yu Zhang. 2024. FLA: A triton-based library for hardware-efficient implementations of linear attention mechanism. 18190 12 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zuhri-2026-softpick-no-attention-sink-no/