Elamur Research⌕
☰
home / papers / 2026 / On the Mathematical Relationship Between Layer Normalization and Dynamic Activation Functions

On the Mathematical Relationship Between Layer Normalization and Dynamic Activation Functions

Paper · 2026 · Felix Stollenwerk

Summary

Layer normalization (LN) and RMSNorm are standard in transformers. Zhu et al. (2025) proposed Dynamic Tanh (DyT), an element-wise function with a learnable parameter, as a drop-in substitute. DyT works well empirically, but nobody had explained theoretically why it mimics normalization.

This short theoretical paper fills that gap starting from RMSNorm:

  • DyT derivation: The author computes the Jacobian of RMSNorm and decouples it. Off-diagonal entries are discarded and the input norm is swapped for a global constant. Solving the resulting single differential equation gives a scaled DyT, so DyT turns out to be an approximation.
  • DyISRU derivation: Applying the same decoupling directly to a rearranged RMSNorm formula removes the need for any approximation. The result is an exact element-wise counterpart called Dynamic Inverse Square Root Unit (DyISRU).

A synthetic outlier simulation shows that DyISRU reproduces how RMSNorm squashes outliers far more faithfully than DyT. The work gives dynamic activation functions a mathematical grounding and points to a potentially better replacement for normalization layers. However, no model-training experiments were run.

Key points

  • Scaled DyT: √C · tanh(αx_i). It follows from neglecting off-diagonal Jacobian terms and fixing the norm of x as a learnable constant.
  • DyISRU: √C · x_i / √(β + x_i²). Here β globally stands in for the squared norm of all channels except the transformed one. The function is proportional to the ISRU activation (Carlile et al., 2017).
  • Simulation setup: C = 100 channels drawn from N(0, 2²). The largest value was raised in steps of 5, up to S = 9 steps.
  • Fit on outliers: Optimal parameters were α = 0.049 and β = 301.1. Mean absolute residuals were 0.33 for DyT versus < 0.01 for DyISRU.
  • Venue, code and limitations: EACL 2026 short paper. Code is at github.com/flxst/dynamic-activation-functions. Effects on downstream performance and on sensitivity to parameter initialization remain untested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: AI Sweden.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Felix Stollenwerk. 2026. On the Mathematical Relationship Between Layer Normalization and Dynamic Activation Functions. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-short.48

@inproceedings{stollenwerk2026mathematical,
  title = {{On the Mathematical Relationship Between Layer Normalization and Dynamic Activation Functions}},
  author = {Felix Stollenwerk},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-short.48},
  url = {https://doi.org/10.18653/v1/2026.eacl-short.48}
}

References

14 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization.
  2. Andrew Brock, Soham De, and Samuel L. Smith. 2021a. Characterizing signal propagation to close the performance gap in unnormalized resnets. find
  3. Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. 2021b. High-performance large-scale image recognition without normalization. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1059–1071. PMLR. find
  4. Brad Carlile, Guy Delamarter, Paul Kinney, Akiko Marti, and Brian Whitney. 2017. Improving deep learning by inverse square root linear units (isrlus). find
  5. Soham De and Sam Smith. 2020. Batch normalization biases residual blocks towards the identity function in deep networks. In Advances in Neural Information Processing Systems, volume 33, pages 19964–19975. Curran Associates, Inc. find
  6. Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. 2000. Incorporating second-order functional knowledge for better option pricing. Advances in neural information processing systems, 13. find
  7. Bobby He and Thomas Hofmann. 2024. Simplifying transformer blocks. find
  8. Lei Huang, Jie Qin, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. 2023. Normalization techniques in training dnns: Methodology, analysis and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8):10173–10196. find
  9. Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. 2017. Self-normalizing neural networks. In Advances in Neural Information Yunhao Ni, Yuxin Guo, Junlong Jia, and Lei Huang. 2024. On the nonlinearity of layer normalization. In Proceedings of the 41st International Conference o… find
  10. Sho Takase, Shun Kiyono, Sosuke Kobayashi, and Jun Suzuki. 2025. Spike no more: Stabilizing the pre-training of large language models. find
  11. Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. 2020. On layer normalization in the transformer architecture. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine… find
  12. Biao Zhang and Rico Sennrich. 2019. Root mean square layer normalization. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. find
  13. Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. 2019. Fixup initialization: Residual learning without normalization. find
  14. Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. 2025. Transformers without normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/stollenwerk-2026-mathematical-relationship-between-layer-normalization/