Elamur Research⌕
☰
home / papers / 2026 / Map of Encoders – Mapping Sentence Encoders using Quantum Relative Entropy

Map of Encoders – Mapping Sentence Encoders using Quantum Relative Entropy

Paper · 2026 · Gaifan Zhang and Danushka Bollegala

Summary

Thousands of sentence encoders are hosted on Hugging Face Hub, but there is no principled way to see how they relate to each other. Comparing them is hard because their outputs are multivariate vectors, their spaces are not aligned, and their sizes range from 384 to 4096 dimensions.

The authors embed a fixed set of 10,000 English sentences with each encoder and form a Pairwise Inner Product (PIP) matrix. This matrix has the same shape for every model, whatever its embedding dimensionality. Dividing it by its trace turns it into a density matrix. Quantum Relative Entropy (QRE), measured against a unit base encoder, then gives each encoder a feature vector in a common space, and t-SNE projects these vectors into a 2D map covering 1101 encoders.

Models that share an architecture, training data or fine-tuning tasks end up near each other on the map. The feature vectors also predict downstream MTEB scores well, so the map can help with choosing an encoder or estimating how a new one will behave.

Key points

  • Encoder selection: the authors took the 2000 most downloaded of 17,515 Hub encoders (as of December 2025). They removed models that SentenceTransformer cannot load or run correctly, such as cross encoders, which left 1101 mapped encoders.
  • Pipeline: embedding matrix → PIP matrix → trace-normalised density matrix → QRE-based feature vector → t-SNE. The eigenspace comes from an SVD of the embedding matrix, so the N×N PIP matrix never has to be built explicitly, which keeps the method scalable.
  • Data: the sentence set comes from the high-quality-english-sentences dataset. Experiments indicate that 10,000 sentences are enough for a good map.
  • Predicting performance: Spearman correlation between predicted and actual MTEB scores exceeds 0.75 for retrieval, reranking and clustering. It is weaker for classification and moderate for STS overall, though it reaches at least 0.75 on STS14 and SICK-R.
  • Limitations and code: the sentence set is English-only, and selecting by downloads introduces a popularity bias against low-resource-language encoders. Code is available at github.com/LivNLP/Map-of-Encoders.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPhysics and Astronomy · 52 worksAtomic and Molecular Physics, and Optics · 4 works

Cite

Cite as Gaifan Zhang and Danushka Bollegala. 2026. Map of Encoders – Mapping Sentence Encoders using Quantum Relative Entropy. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.645

@inproceedings{zhang2026map,
  title = {{Map of Encoders – Mapping Sentence Encoders using Quantum Relative Entropy}},
  author = {Gaifan Zhang and Danushka Bollegala},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.645},
  url = {https://doi.org/10.18653/v1/2026.acl-long.645}
}

References

48 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. agentlans. 2024. High-quality english sentences. https://huggingface.co/datasets/agentlans/high-quali ty-english-sentences . Accessed: 2025-12-13. ↗
  2. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. International Conference on Learning Representations. find
  3. Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404.05961.
  4. Danushka Bollegala. 2022. Learning meta word embeddings by unsupervised weighted concatenation of source embeddings. Preprint, arXiv:2204.12386. — [arXiv:2204.12386](https://arxiv.org/abs/2204.12386) find
  5. Qian Chen, Wen Wang, Qinglin Zhang, Siqi Zheng, Chong Deng, Hai Yu, Jiaqing Liu, Yukun Ma, and Chong Zhang. 2023. Ditto: A simple and efficient approach to improve sentence embeddings. Empir Method Nat Lang Process, abs/2305.10786:5868– 5875. find
  6. Manlio De Domenico and Jacob Biamonte. 2016. Spectral entropies as information-theoretic tools for complex network comparison. Physical Review X, 6(4). find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019a. Bert: Pre-training of deep bidirectional transformers for language understanding. Preprint, arXiv:1810.04805.
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019b. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
  9. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Stroudsburg, PA, USA. Association for Computational Linguistics.
  10. Rita González-Márquez, Philipp Berens, and Dmitry Kobak. 2022. Two-dimensional visualization of large document libraries using t-sne. In ICLR 2022 workshop on geometrical and topological representation learning. find
  11. Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. 2005. Measuring statistical dependence with hilbert-schmidt norms. In International Conference on Algorithmic Learning Theory, pages 63–77. Springer. find
  12. Nicholas J Higham. 2008. Functions of matrices: theory and computation. SIAM. find
  13. Wassily Hoeffding. 1948. A class of statistics with asymptotically normal distribution. The Annals of Mathematical Statistics, 19(3):293–325. find
  14. N Houlsby, A Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and S Gelly. 2019. Parameter-efficient transfer learning for NLP. ICML, abs/1902.00751:2790–2799. find
  15. Hugging Face. 2025. Hugging face models: Sentence transformers. https://huggingface.co/models? library=sentence-transformers&sort=trending . Accessed: 2025-12-13. ↗
  16. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. Preprint, arXiv:1905.00414.
  17. Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9119–9130, Stroudsburg, PA, USA. Association for Computa…
  18. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in nlp. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 681–691.
  19. Qiuchi Li, Sagar Uprety, Benyou Wang, and Dawei Song. 2018. Quantum-inspired complex word embedding. In Proceedings of the Third Workshop on Representation Learning for NLP, pages 50–57, Melbourne, Australia. Association for Computational Linguistics. find
  20. Luyang Lin, Lingzhi Wang, Jinsong Guo, and Kam-Fai Wong. 2025. Investigating bias in llm-based bias detection: Disparities between llms and human perception. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10634– 10649.
  21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  22. Nathan Mantel. 1967. The detection of disease clustering and a generalized regression approach. Cancer Research, 27(2 Part 1):209–220. find
  23. Chandler May, Alex Wang, Shikha Bordia, Samuel Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Sho…
  24. Ari S Morcos, Maithra Raghu, and Samy Bengio. 2018. Insights on representational similarity in neural networks with canonical correlation. arXiv [stat.ML]. find
  25. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2023. Mteb: Massive text embedding benchmark. Preprint, arXiv:2210.07316.
  26. Michael A. Nielsen and Isaac L. Chuang. 2010. Quantum Computation and Quantum Information: 10th Anniversary Edition. Cambridge University Press. find
  27. Momose Oyama, Ryo Kishino, Hiroaki Yamagiwa, and Hidetoshi Shimodaira. 2025a. Likelihood variance as text importance for resampling texts to map language models. arXiv [cs.CL]. find
  28. Momose Oyama, Hiroaki Yamagiwa, Yusuke Takase, and Hidetoshi Shimodaira. 2025b. Mapping 1,000+ language models via the log-likelihood vector. arXiv [cs.CL]. find
  29. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. Preprint, arXiv:2406.17557.
  30. Fernando Pérez-Cruz. 2008. Kullback-leibler divergence estimation of continuous distributions. In 2008 IEEE international symposium on information theory, pages 1666–1670. IEEE. find
  31. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Improving text embeddings with large language models. Preprint, arXiv:2401.00368.
  32. Wenbo Qiao, Peng Zhang, and ZengLai Ma. 2024. A quantum-inspired matching network with linguistic theories for metaphor detection. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 1435–1445, Torino, It… find
  33. Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, Ed Chi, and Slav Petrov. 2021. Measuring and reducing gendered correlations in pre-trained models.
  34. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.
  35. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. Exploring the limits of transfer learning with a unified text-to-text transformer. Preprint, arXiv:1910.10683.
  36. Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017. SVCCA: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. find
  37. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992.…
  38. Robert J Serfling. 1980. Approximation Theorems of Mathematical Statistics. John Wiley & Sons. find
  39. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted pre-training for language understanding. Preprint, arXiv:2004.09297.
  40. Gilbert Strang. 2022. Introduction to linear algebra. SIAM. find
  41. Peter D Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37:141–188.
  42. Hisaharu Umegaki. 1962. Conditional expectation in an operator algebra, iv (entropy and information). Technical Report 2, Department of Mathematics, Tokyo Institute of Technology. find
  43. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11):2579–2605. find
  44. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transf…
  45. Sixuan Wu, Jian Li, Peng Zhang, and Yue Zhang. 2021. Natural language processing meets quantum physics: A survey and categorization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3172–3182, Online and Punta Cana, Dominican Republic. Association for… find
  46. Jiahao Xu, Wei Shao, Lihui Chen, and Lemao Liu. 2023. SimCSE++: Improving contrastive learning for sentence embeddings from two perspectives. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12028–12040, Stroudsburg, PA, USA. Association for Computatio… find
  47. Zi Yin and Yuanyuan Shen. 2018. On the dimensionality of word embedding. Advances in neural information processing systems, 31. find
  48. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. Preprint, arXiv:2506.05176.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-map-encoders-mapping-sentence-encoders/