Elamur Research⌕
☰
home / papers / 2026 / Representation Collapse in Machine Translation Through the Lens of Angular Dispersion

Representation Collapse in Machine Translation Through the Lens of Angular Dispersion

Paper · 2026 · Evgeniia Tokarchuk, Maya K. Nachesa, Sergey Troshin et al.

Summary

Transformer models for neural machine translation trained with standard next-token prediction can suffer from representation collapse. In dimensional collapse, vectors occupy only a low-dimensional part of the space. In complete collapse, all vectors become the same point, a trivial optimum that threatens continuous-output NMT (CoNMT) when target embeddings are learned end-to-end. The authors track collapse during training of discrete and continuous Transformer-base and Transformer-big models on WMT19 English–German. They use three diagnostics: average cosine similarity, matrix Rényi entropy of the Gram spectrum, and spherical variance. As a remedy, they add an existing angular-dispersion regularizer, sliced dispersion, to individual representation layers. This method keeps directions spread on the sphere without pairwise quadratic cost. The regularizer improves translation quality over a strong baseline, and the gain survives post-training quantization. For CoNMT, it prevents complete collapse when embeddings are trainable. The work matters because it offers a cheap, contrastive-free way to stabilize representation learning, which is useful for compact and edge-deployable MT systems.

Key points

  • Collapse metrics:
    • Average cosine similarity.
    • Matrix (Shannon, α = 1) Rényi entropy of eigenvalues.
    • Spherical variance, which is cheap enough to log during training.
  • Setup: WMT19 en–de is the benchmark, with newstest17 used for development and newstest2019 for test. Quality is measured with BLEU and COMET.
  • CoNMT with trainable embeddings (de-en, newstest2019):
    • Without regularization, the model collapses to 0.0 BLEU and 0.000 COMET. Spherical variance and entropy fall to 0, and cosine similarity reaches 1.
    • With dispersion regularization, randomly initialized embeddings reach 33.2 BLEU and 0.708 COMET.
  • CoNMT with frozen embeddings: The best CoNMT result comes from frozen target embeddings taken from a regularized NMT model, at 36.6 BLEU and 0.749 COMET. Frozen embeddings from an unregularized model score 32.9 BLEU.
  • Quantization: float16 training and post-training quantization roughly match full precision. int8_float16 causes a large quality drop, but the advantage of the dispersion-regularized model remains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands144 works · 173,906 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 1,087 works

Cite

Cite as Evgeniia Tokarchuk, Maya K. Nachesa, Sergey Troshin and Vlad Niculae. 2026. Representation Collapse in Machine Translation Through the Lens of Angular Dispersion. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.126

@inproceedings{tokarchuk2026representation,
  title = {{Representation Collapse in Machine Translation Through the Lens of Angular Dispersion}},
  author = {Evgeniia Tokarchuk, Maya K. Nachesa, Sergey Troshin and Vlad Niculae},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.126},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.126}
}

References

47 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Duarte Miguel Alves, José Pombal, Nuno M Guerreiro, Pedro Henrique Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and Andre Martins. 2024. Tower: An open multilingual large language model for translation-related tas…
  2. Md Rifat Arefin, Gopeshh Subbaraj, Nicolas Gontier, Yann LeCun, Irina Rish, Ravid Shwartz-Ziv, and Christopher Pal. 2025. Seq-VCR: Preventing collapse in intermediate transformer representations for enhanced reasoning. In The Thirteenth International Conference on Learning Representations. find
  3. Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, João Guilherme Madeira Araújo, Alex Vitvitskyi, Razvan Pascanu, and Petar Veličković. 2024. Transformers need glasses! information over-squashing in language tasks. In The Thirty-eighth Annual Conference on Neural Information P… find
  4. Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (…
  5. Clément Bonet, Paul Berg, Nicolas Courty, François Septier, Lucas Drumetz, and Minh Tan Pham. 2023. Spherical Sliced-Wasserstein. In The Eleventh International Conference on Learning Representations. find
  6. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597–1607. PMLR.
  7. Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. 2019. Representation degeneration problem in training natural language generation models. In International Conference on Learning Representations.
  8. Gleb Gerasimov, Yaroslav Aksenov, Nikita Balagansky, Viacheslav Sinii, and Daniil Gavrilov. 2025. You do not fully utilize transformer’s representation capacity. Preprint, arXiv:2502.09245. — [arXiv:2502.09245](https://arxiv.org/abs/2502.09245) find
  9. Nathan Godey, Éric Clergerie, and Benoît Sagot. 2024. Anisotropy is inherent to self-attention in transformers. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 35–48, St. Julian’s, Malta. Association for Co… find
  10. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  11. Zhaohan Daniel Guo, Bernardo Avila Pires, Khimya Khetarpal, Dale Schuurmans, and Bo Dai. 2025. Representation learning via non-contrastive mutual information. Preprint, arXiv:2504.16667. — [arXiv:2504.16667](https://arxiv.org/abs/2504.16667) find
  12. Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. 2021. On feature decorrelation in self-supervised learning. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9578–9588. find
  13. S Rao Jammalamadaka and Ambar Sengupta. 2001. Topics in circular statistics. Series on multivariate analysis; vol. 5. World Scientific, Singapore. find
  14. Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. 2022. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations. find
  15. Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2021. Nearest neighbor machine translation. In International Conference on Learning Representations. find
  16. Diederik P. Kingma and Jimmy Ba. 2014. A method for stochastic optimization. abs/1412.6980.
  17. Adam: CoRR, Guillaume Klein, Dakun Zhang, Clément Chouteau, Josep M Crego, and Jean Senellart. 2020. Efficient and high-quality neural machine translation with opennmt. In Proceedings of the fourth workshop on neural generation and translation, pages 211–217. find
  18. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ…
  19. Sachin Kumar and Yulia Tsvetkov. 2019. Von Mises-Fisher loss for training sequence to sequence models with continuous outputs. In International Conference on Learning Representations. find
  20. Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. ArXiv, abs/2005.11401.
  21. Weiyang Liu, Rongmei Lin, Z. Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. 2018. Learning towards minimum hyperspherical energy. In Neural Information Processing Systems. find
  22. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. find
  23. Kantilal Varichand Mardia. 1975. Statistics of directional data. Journal of the Royal Statistical Society Series B: Statistical Methodology, 37(3):349–371. find
  24. Gaurav Menghani. 2023. Efficient deep learning: A survey on making deep learning models smaller, faster, and better. ACM Computing Surveys, 55(12):1–37. find
  25. Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, Xinyi Wang, and John Wieting. 2019. compare-mt: A tool for holistic comparison of language generation systems. CoRR, abs/1903.07926.
  26. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
  27. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Belgium, Brussels. Association for Computational Linguistics.
  28. Anastasia Razdaibiedina, Ashish Khetan, Zohar Karnin, Daniel Khashabi, and Vivek Madan. 2023. Representation projection invariance mitigates representation collapse. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14638– 14664, Singapore. Association for Computational… find
  29. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.
  30. Alfréd Rényi. 1961. On measures of entropy and information. In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume 1: contributions to the theory of statistics, volume 4, pages 547–562. University of California Press. find
  31. Luis Gonzalo Sanchez Giraldo, Murali Rao, and Jose C. Principe. 2015. Measures of entropy from data using infinitely divisible kernels. 61(1):535–548. Publisher Copyright: © 2014 IEEE. find
  32. Copyright: Copyright 2015 Elsevier B.V., All rights reserved. find
  33. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin…
  34. Dinghan Shen, Ming Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. 2020. A simple but tough-to-beat data augmentation approach for natural language understanding and generation. ArXiv, abs/2009.13818. — [arXiv:2009.13818](https://arxiv.org/abs/2009.13818) find
  35. Dimitar Shterionov and Eva Vanmassenhove. 2023. The ecological footprint of neural machine translation systems. In Towards Responsible Machine Translation: Ethical and Legal Considerations in Machine Translation, pages 185–213. Springer. find
  36. Oscar Skean, Md Rifat Arefin, and Ravid Shwartz-Ziv. 2024. Does representation matter? exploring intermediate layers in large language models. In Workshop on Machine Learning and Compression, NeurIPS 2024. find
  37. Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for better semantics and faster retrieval. ArXiv, abs/2103.15316. find
  38. Pieter Merkus Lambertus Tammes. 1930. On the origin of number and arrangement of the places of exit on the surface of pollen-grains. Ph.D. thesis, University of Groningen. Relation: http://www.rug.nl/ Rights: De Bussy. ↗
  39. Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhijun Tu, Kai Han, Hailin Hu, and Dacheng Tao. 2024. A survey on transformer compression. arXiv preprint arXiv:2402.05964. — [arXiv:2402.05964](https://arxiv.org/abs/2402.05964) find
  40. Evgeniia Tokarchuk, Hua Chang Bakker, and Vlad Niculae. 2025a. Keep your distance: learning dispersed embeddings on $\\mathbb{S}_{m}$. Transactions on Machine Learning Research. find
  41. Evgeniia Tokarchuk and Vlad Niculae. 2022. On target representation in continuous-output neural machine translation. In Proceedings of the 7th Workshop on Representation Learning for NLP, pages 227–235, Dublin, Ireland. Association for Computational Linguistics. find
  42. Evgeniia Tokarchuk and Vlad Niculae. 2024. The unreasonable effectiveness of random target embeddings for continuous-output neural machine translation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (V… find
  43. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, Red Hook, NY, USA. Curran Associates Inc.
  44. Elena Voita, Rico Sennrich, and Ivan Titov. 2019. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Join…
  45. Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9929–9939. PMLR. find
  46. Zhennan Wang, Canqun Xiang, Wenbin Zou, and Chen Xu. 2020. MMA regularization: Decorrelating weights of neural networks by maximizing the minimal angles. In Advances in Neural Information Processing Systems, volume 33, pages 19099–19110. Curran Associates, Inc. find
  47. Jason Wei and Kai Zou. 2019. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tokarchuk-2026-representation-collapse-machine-translation-through/