Elamur Research⌕
☰
home / papers / 2026 / HTMuon: Improving Muon via Heavy-Tailed Spectral Correction

HTMuon: Improving Muon via Heavy-Tailed Spectral Correction

Paper · 2026 · Tianyu Pang, Yujie Fang, Zihang Liu et al.

Summary

Muon is a matrix-based optimizer now used in large-scale LLM training. It orthogonalizes the momentum matrix, which forces every singular value to one. The authors argue this has two costs. It gives as much weight to noisy, small-singular-value directions as to informative ones. It also keeps the weight spectra light-tailed, while Heavy-Tailed Self-Regularization (HT-SR) theory links heavier-tailed empirical spectral densities with better-trained models.

Their fix, HTMuon, raises the momentum's singular values to a power p between 0 and 1. Setting p = 0 recovers Muon and p = 1 gives SGDM. This keeps Muon's ability to model parameter coupling while producing heavier-tailed updates and weights.

They evaluate HTMuon on:

  • LLaMA pretraining on C4
  • GPT-2 on OpenWebText
  • ResNet and ViT image classification

Across these tasks it beats strong optimizers and also improves existing Muon variants when added on top of them. On the theory side, HTMuon is shown to be steepest descent under a Schatten-q norm constraint, with a convergence guarantee matching Muon's. This matters because it offers a simple, theory-backed upgrade to a widely adopted optimizer.

Key points

  • Default setting: p = 0.125, chosen by grid search. Compared with Muon, HTMuon lowers perplexity by 0.92 on LLaMA-60M and by 0.98 on LLaMA-135M trained on C4.
  • Baselines beaten: Adam, Muon, Cautious, GaLore, Sophia, Mars, SOAP and COSMOS. It also works as a plug-in for NorMuon and AdaMuon.
  • Motivating observation: the approximate Muon_NS (Newton–Schulz) outperforms the exact Muon_SVD, which suggests that uniform unit singular values are suboptimal.
  • Theory: HTMuon generalizes Muon's Schatten-∞ steepest-descent view to Schatten-q. In smooth non-convex settings its sample complexity is O(ε⁻⁴), the same as Muon and SGDM.
  • Scaling and limitations: two accelerated variants let HTMuon outperform Muon on LLaMA-1B. Per-step runtime remains slightly higher, and nothing beyond 1B parameters was tested. Code is at github.com/TDCSZ327/HTmuon.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations🎓University of California, BerkeleyBerkeley · United States175 works · 227,440 citations🔬International Computer Science InstituteBerkeley · United States15 works · 11,471 citations🎓Dartmouth CollegeHanover · United States37 works · 42,625 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Tianyu Pang, Yujie Fang, Zihang Liu, Shenyang Deng, Lei Hsiung, Shuhua Yu and Yaoqing Yang. 2026. HTMuon: Improving Muon via Heavy-Tailed Spectral Correction. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1819

@inproceedings{pang2026htmuon,
  title = {{HTMuon: Improving Muon via Heavy-Tailed Spectral Correction}},
  author = {Tianyu Pang, Yujie Fang, Zihang Liu, Shenyang Deng, Lei Hsiung, Shuhua Yu and Yaoqing Yang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1819},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1819}
}

References

72 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. 2023. Lower bounds for non-convex stochastic optimization. Mathematical Programming, 199(1):165–214. find
  2. Jeremy Bernstein and Laker Newhouse. 2024. Modular duality in deep learning. arXiv preprint arXiv:2410.21265. find
  3. Da Chang, Yongxiang Liu, and Ganzhao Yuan. 2025. On the convergence of muon and beyond. arXiv preprint arXiv:2509.15816. — [arXiv:2509.15816](https://arxiv.org/abs/2509.15816) find
  4. Lei Chen, Joan Bruna, and Alberto Bietti. 2024. Distributional associations vs in-context reasoning: A study of feed-forward and attention layers. arXiv preprint arXiv:2406.03068. — [arXiv:2406.03068](https://arxiv.org/abs/2406.03068) find
  5. Lizhang Chen, Jonathan Li, and Qiang Liu. 2025. Muon optimizes under spectral norm constraints. arXiv preprint arXiv:2506.15054. — [arXiv:2506.15054](https://arxiv.org/abs/2506.15054) find
  6. Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V Le. 2023. Symbolic discovery of optimization algorithms. In Thirty-seventh Conference on Neural Information Processing Systems. find
  7. Yatin Dandi, Luca Pesce, Hugo Cui, Florent Krzakala, Yue M Lu, and Bruno Loureiro. 2024. A random matrix theory perspective on the spectrum of learned features and asymptotic generalization capabilities. arXiv preprint arXiv:2410.18938. — [arXiv:2410.18938](https://arxiv.org/abs/2410.18938) find
  8. Damek Davis and Dmitriy Drusvyatskiy. 2025. When do spectral gradient updates help in deep learning? arXiv preprint arXiv:2512.04299. — [arXiv:2512.04299](https://arxiv.org/abs/2512.04299) find
  9. Leonardo Defilippis, Yizhou Xu, Julius Girardin, Emanuele Troiani, Vittorio Erba, Lenka Zdeborová, Bruno Loureiro, and Florent Krzakala. 2025. Scaling laws and spectra of shallow neural networks in the feature learning regime. arXiv preprint arXiv:2509.24882. — [arXiv:2509.24882](https://arxiv.org/abs/2509.24882) find
  10. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee. find
  11. Shenyang Deng, Boyao Liao, Zhuoli Ouyang, Tianyu Pang, Minhak Song, and Yaoqing Yang. 2026. Suspicious alignment of sgd: A fine-grained step size condition analysis. arXiv preprint arXiv:2601.11789. — [arXiv:2601.11789](https://arxiv.org/abs/2601.11789) find
  12. Alexey Dosovitskiy. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
  13. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7). find
  14. Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. 2020. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412. — [arXiv:2010.01412](https://arxiv.org/abs/2010.01412) find
  15. Guillaume Garrigos and Robert M Gower. 2023. Handbook of convergence theorems for (stochastic) gradient methods. arXiv preprint arXiv:2301.11235. — [arXiv:2301.11235](https://arxiv.org/abs/2301.11235) find
  16. Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus. http://Skylion007.github.io/ OpenWebTextCorpus. ↗
  17. Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, and 24 others. 2024. Olmo: A… find
  18. Vineet Gupta, Tomer Koren, and Yoram Singer. 2018. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pages 1842–1850. PMLR. find
  19. Di He, Songjun Tu, Ajay Jaiswal, Li Shen, Ganzhao Yuan, Shiwei Liu, and Lu Yin. 2025. Alphadecay: Module-wise weight decay for heavy-tailed balancing in llms. arXiv preprint arXiv:2506.14562. — [arXiv:2506.14562](https://arxiv.org/abs/2506.14562) find
  20. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778.
  21. Liming Liu, Zhenghao Xu, Zixuan Zhang, Hao Kang, Zichong Li, Chen Liang, Weizhu Chen, and Tuo Zhao. 2025b. Cosmos: A hybrid adaptive optimizer for memory-efficient training of llms. arXiv preprint arXiv:2502.17410. — [arXiv:2502.17410](https://arxiv.org/abs/2502.17410) find
  22. Nicholas J Higham and Robert S Schreiber. 1990. Fast polar decomposition of an arbitrary matrix. SIAM Journal on Scientific and Statistical Computing, 11(4):648–655. find
  23. Xuyuan Liu, Lei Hsiung, Yaoqing Yang, and Yujun Yan. 2025c. Spectral insights into data-oblivious critical layers in large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 4860–4877, Vienna, Austria. Association for Computational Linguistics. find
  24. Liam Hodgkinson, Umut Simsekli, Rajiv Khanna, and Michael Mahoney. 2022. Generalization bounds using lower tail exponents in stochastic optimizers. In International Conference on Machine Learning, pages 8774–8795. PMLR. find
  25. Liam Hodgkinson, Zhichao Wang, and Michael W Mahoney. 2025. Models of heavy-tailed mechanistic universality. arXiv preprint arXiv:2506.03470. — [arXiv:2506.03470](https://arxiv.org/abs/2506.03470) find
  26. Yuanzhe Hu, Kinshuk Goel, Vlad Killiakov, and Yaoqing Yang. 2025. Eigenspectrum analysis of neural networks without aspect ratio bias. In Forty-second International Conference on Machine Learning. find
  27. Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. 2024. Muon: An optimizer for hidden layers in neural networks. find
  28. Jean Kaddour, Linqing Liu, Ricardo Silva, and Matt J Kusner. 2022. When do flat minima optimizers work? Advances in Neural Information Processing Systems, 35:16577–16595. find
  29. Diederik P Kingma. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  30. Vignesh Kothapalli, Tianyu Pang, Shenyang Deng, Zongmin Liu, and Yaoqing Yang. 2025. From spikes to heavy tails: Unveiling the spectral evolution of neural networks. Transactions on Machine Learning Research. find
  31. Alex Krizhevsky and Geoffrey Hinton. 2009. Learning multiple layers of features from tiny images. find
  32. Zichong Li, Liming Liu, Chen Liang, Weizhu Chen, and Tuo Zhao. 2025. Normuon: Making muon more efficient and scalable. arXiv preprint arXiv:2510.05491. — [arXiv:2510.05491](https://arxiv.org/abs/2510.05491) find
  33. Kaizhao Liang, Lizhang Chen, Bo Liu, and Qiang Liu. 2024. Cautious optimizers: Improving training with one line of code. arXiv preprint arXiv:2411.16085. — [arXiv:2411.16085](https://arxiv.org/abs/2411.16085) find
  34. Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. 2024a. Sophia: A scalable stochastic second-order optimizer for language model pre-training. Preprint, arXiv:2305.14342. find
  35. Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, and 9 others. 2025a. Muon is scalable for llm training. arXiv preprint… find
  36. Zihang Liu, Yuanzhe Hu, Tianyu Pang, Yefan Zhou, Pu Ren, and Yaoqing Yang. 2024b. Model balancing helps low-data training and fine-tuning. arXiv preprint arXiv:2410.12178. find
  37. Zihang Liu, Tianyu Pang, Oleg Balabanov, Chaoqun Yang, Tianjin Huang, Lu Yin, Yaoqing Yang, and Shiwei Liu. 2025d. Lift the veil for the truth: Principal weights emerge after rank reduction for reasoning-focused supervised fine-tuning. arXiv preprint arXiv:2506.00772. — [arXiv:2506.00772](https://arxiv.org/abs/2506.00772) find
  38. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. find
  39. Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W Mahoney, and Yaoqing Yang. 2024. Al-phapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models. Advances in neural information processing systems, 37:9117–9152. find
  40. Jianhao Ma, Yu Huang, Yuejie Chi, and Yuxin Chen. 2026. Preconditioning benefits of spectral orthogonalization in muon. Preprint, arXiv:2601.13474. — [arXiv:2601.13474](https://arxiv.org/abs/2601.13474) find
  41. Charles H Martin and Christopher Hinrichs. 2025. Setol: A semi-empirical theory of (deep) learning. arXiv preprint arXiv:2507.17912. — [arXiv:2507.17912](https://arxiv.org/abs/2507.17912) find
  42. Charles H Martin and Michael W Mahoney. 2021. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1–73. find
  43. Charles H Martin, Tongsu Peng, and Michael W Mahoney. 2021. Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data. Nature Communications, 12(1):4122. find
  44. Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. 2018. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957. — [arXiv:1802.05957](https://arxiv.org/abs/1802.05957) find
  45. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, and 1 others. 2024. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656. find
  46. Thomas Pethick, Wanyun Xie, Kimon Antonakopou-los, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. 2025. Training deep learning models with norm-constrained lmos. arXiv preprint arXiv:2502.07529. — [arXiv:2502.07529](https://arxiv.org/abs/2502.07529) find
  47. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  48. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.
  49. Andrei Semenov, Matteo Pagliardini, and Martin Jaggi. 2025. Benchmarking optimizers for large language model pretraining. arXiv preprint arXiv:2509.01440. — [arXiv:2509.01440](https://arxiv.org/abs/2509.01440) find
  50. Ishaan Shah, Anthony M. Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J Shah, Khoi Nguyen, Kurt Smith, Michael Callahan, Michael Pust, Mohit Parmar, Peter Rushton, Platon Mazarakis, Ritvik Kapila, Saurabh Srivastava, and 4 others… find
  51. Pratyusha Sharma, Jordan T Ash, and Dipendra Misra. 2023. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. arXiv preprint arXiv:2312.13558. — [arXiv:2312.13558](https://arxiv.org/abs/2312.13558) find
  52. Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, and Jiawei Zhang. 2025. On the convergence analysis of muon. arXiv preprint arXiv:2505.23737. — [arXiv:2505.23737](https://arxiv.org/abs/2505.23737) find
  53. Chongjie Si, Debing Zhang, and Wei Shen. 2025. Adamuon: Adaptive muon optimizer. arXiv preprint arXiv:2507.11005. — [arXiv:2507.11005](https://arxiv.org/abs/2507.11005) find
  54. James B Simon, Madeline Dickens, Dhruva Karkada, and Michael R DeWeese. 2023. The eigenlearning framework: A conservation law perspective on kernel ridge regression and wide neural networks. Transactions on Machine Learning Research. find
  55. Umut Simsekli, Ozan Sener, George Deligiannidis, and Murat A Erdogdu. 2020. Hausdorff dimension, heavy tails, and generalization in neural networks. Advances in Neural Information Processing Systems, 33:5138–5151. find
  56. Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. 2013. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139– 1147. pmlr. find
  57. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, and 1 others. 2025. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534.
  58. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  59. Nikhil Vyas, Depen Morwani, Rosie Zhao, Mujin Kwun, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. 2024. Soap: Improving and stabilizing shampoo using adam. arXiv preprint arXiv:2409.11321. find
  60. Jinbo Wang, Mingze Wang, Zhanpeng Zhou, Junchi Yan, and Lei Wu. 2025a. The sharpness disparity principle in transformers for accelerating language model pre-training. arXiv preprint arXiv:2502.19002. — [arXiv:2502.19002](https://arxiv.org/abs/2502.19002) find
  61. Shuche Wang, Fengzhuo Zhang, Jiaxiang Li, Cunxiao Du, Chao Du, Tianyu Pang, Zhuoran Yang, Mingyi Hong, and Vincent YF Tan. 2025b. Muon outperforms adam in tail-end associative memory learning. arXiv preprint arXiv:2509.26030. — [arXiv:2509.26030](https://arxiv.org/abs/2509.26030) find
  62. Zhichao Wang, Andrew Engel, Anand D Sarwate, Ioana Dumitriu, and Tony Chiang. 2023. Spectral evolution and invariance in linear-width neural networks. Advances in neural information processing systems, 36:20695–20728. find
  63. Kaiyue Wen, David Hall, Tengyu Ma, and Percy Liang. 2025. Fantastic pretraining optimizers and where to find them. arXiv preprint arXiv:2509.02046. find
  64. Yaoqing Yang, Ryan Theisen, Liam Hodgkinson, Joseph E Gonzalez, Kannan Ramchandran, Charles H Martin, and Michael W Mahoney. 2023. Test accuracy vs. generalization gap: Model selection in nlp without accessing training or testing data. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Di… find
  65. Huizhuo Yuan, Yifeng Liu, Shuang Wu, Xun Zhou, and Quanquan Gu. 2024. Mars: Unleashing the power of variance reduction for training large models. arXiv preprint arXiv:2411.10438. — [arXiv:2411.10438](https://arxiv.org/abs/2411.10438) find
  66. Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, and 1 others. 2025. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.06471.
  67. Chunhui Zhang, Yiren Jian, Zhongyu Ouyang, and Soroush Vosoughi. 2024a. Working memory identifies reasoning limits in language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16896–16922. find
  68. Chunhui Zhang, Yiren Jian, Zhongyu Ouyang, and Soroush Vosoughi. 2025. Pretrained image-text models are secretly video captioners. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies. find
  69. Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Diederik P Kingma, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. 2024b. Adam-mini: Use fewer learning rates to gain more. arXiv preprint arXiv:2406.16793. find
  70. Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. 2024. Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507. find
  71. Beitong Zhou, Jun Liu, Weigao Sun, Ruijuan Chen, Claire J Tomlin, and Ye Yuan. 2020. pbsgd: Powered stochastic gradient descent methods for accelerated non-convex optimization. In IJCAI, pages 3258–3266. find
  72. Yefan Zhou, Tianyu Pang, Keqin Liu, Michael W Mahoney, and Yaoqing Yang. 2023. Temperature balancing, layer-wise weight analysis, and neural network training. Advances in Neural Information Processing Systems, 36:63542–63572. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pang-2026-htmuon-improving-muon-heavy-tailed/