Elamur Research⌕
☰
home / papers / 2025 / EMO: Embedding Model Distillation via Intra-Model Relation and Optimal Transport Alignments

EMO: Embedding Model Distillation via Intra-Model Relation and Optimal Transport Alignments

Paper · 2025 · Minh-Phuc Truong, Hai An Vu, Tu Vu et al.

Summary

Large text embedding models such as LLM2Vec are expensive to deploy. Knowledge distillation can compress them, but it gets harder when the teacher and the student split text with different tokenizers. Earlier cross-tokenizer KD (CTKD) methods mostly match final outputs or logits. They ignore the relational knowledge stored in attention maps and hidden states.

The authors propose EMO, a CTKD framework designed for embedding models, with three steps:

  • Student and teacher tokens are first paired one-to-one using Minimum Edit Distance (MinED).
  • From these pairs, the top-m most salient tokens are kept, and their attention patterns are matched with Centered Kernel Alignment (the IRA component).
  • In parallel, final hidden states are aligned through Optimal Transport, with each token's mass set by an importance score derived from attention (the OTIS component).

LLM2Vec-Mistral 7B and BGE-M3 teachers are distilled into Bert-base-uncased. The students are tested on three kinds of tasks: categorizing single texts, categorizing pairs of sentences, and scoring how similar two texts are in meaning. EMO consistently beats the ULD, MinED, DSKD and MultiLevelOT baselines. This suggests compact embedding models can keep much of a teacher's quality despite tokenizer mismatch.

Key points

  • On STS with the LLM2Vec-Mistral 7B teacher, EMO improves substantially over DSKD:
    • STS-B: 81.3 vs 78.3 Spearman (Bert SFT: 75.1; teacher: 90.8).
    • STS12: 75.3 vs 65.3 (Bert SFT: 49.7).
    • SICK: 80.1 vs 78.7 (Bert SFT: 61.1).
  • With BGE-M3 as teacher on SciTail, EMO reaches 92.7 accuracy, against:
    • DSKD: 91.5
    • MultilevelOT: 91.4
    • ULD: 91.2
    • MinED: 89.5
    • Bert SFT: 88.1
  • The ablation on SciTail shows the two components work together. Full EMO scores 90.9 accuracy, versus 88.8 without OTIS and 87.2 without IRA.
  • Applying CKA to only a subset of aligned tokens works best. With m = ⌊n/3⌋, scores are 81.3 on STS-B and 80.1 on SICK, compared with 80.2 and 76.9 when all n tokens are used.
  • Acknowledged limitation: tokens without a 1-to-1 MinED match, or outside the top-m, are left out of the attention-alignment step.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Monash UniversityMelbourne · Australia60 works · 54,866 citations🏢ByteDanceBeijing · China41 works · 4,180 citations🎓University of OregonEugene · United States23 works · 3,272 citations🎓Hanoi University of Science and TechnologyHanoi · Vietnam10 works · 89 citations🎓Trường Đại học Khoa học và Công nghệ Hà NộiHanoi · Vietnam8 works · 88 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Minh-Phuc Truong, Hai An Vu, Tu Vu, Nguyen Thi Ngoc Diep, Linh Ngo Van, Thien Huu Nguyen and Trung Le. 2025. EMO: Embedding Model Distillation via Intra-Model Relation and Optimal Transport Alignments. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.385

@inproceedings{truong2025emo,
  title = {{EMO: Embedding Model Distillation via Intra-Model Relation and Optimal Transport Alignments}},
  author = {Minh-Phuc Truong, Hai An Vu, Tu Vu, Nguyen Thi Ngoc Diep, Linh Ngo Van, Thien Huu Nguyen and Trung Le},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.385},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.385}
}

References

40 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nguyen Hoang Anh, Quyen Tran, Thanh Xuan Nguyen, Nguyen Thi Ngoc Diep, Linh Ngo Van, Thien Huu Nguyen, and Trung Le. 2025. Mutual-pairing data augmentation for fewshot continual relation extraction. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for C… find
  2. Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. LLM2Vec: Large language models are secretly powerful text encoders. In First Conference on Language Modeling.
  3. Nicolas Boizard, Kevin El Haddad, Céline Hudelot, and Pierre Colombo. 2025. Towards cross-tokenizer distillation: the universal logit distillation loss for llms.
  4. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. find
  5. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does bert look at? an analysis of bert’s attention. find
  6. Xiao Cui, Mo Zhu, Yulei Qin, Liang Xie, Wengang Zhou, and Houqiang Li. 2024. Multi-level optimal transport for universal cross-tokenizer knowledge distillation on language models. arXiv preprint arXiv:2412.14528. — [arXiv:2412.14528](https://arxiv.org/abs/2412.14528) find
  7. Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transportation distances. find
  8. Sayantan Dasgupta and Trevor Cohn. 2025. Improving language model distillation through hidden state matching. In The Thirteenth International Conference on Learning Representations. find
  9. Chaochen Gao, Xing Wu, Peng Wang, Jue Wang, Liangjun Zang, Zhongyuan Wang, and Songlin Hu. 2023. Distilcse: Effective knowledge distillation for contrastive sentence embeddings. find
  10. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. Minillm: Knowledge distillation of large language models. find
  11. David R. Hardoon, Sandor Szedmak, and John Shawe-Taylor. 2004. Canonical correlation analysis: An overview with application to learning methods. Neural Computation, 16(12):2639–2664. find
  12. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network.
  13. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling BERT for natural language understanding. CoRR, abs/1909.10351.
  14. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. SciTail: A textual entailment dataset from science question answering. In AAAI. find
  15. Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation.
  16. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited.
  17. Anh Duc Le, Nam Le Hai, Thanh Xuan Nguyen, Linh Ngo Van, Nguyen Thi Ngoc Diep, Sang Dinh, and Thien Huu Nguyen. 2025. Enhancing discriminative representation in similar relation clusters for few-shot continual relation extraction. In Proceedings of the 2025 Conference of the Nations of the Americas… find
  18. Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2025. Nv-embed: Improved techniques for training llms as generalist embedding models.
  19. Junyan Li, Li Lyna Zhang, Jiahang Xu, Yujing Wang, Shaoguang Yan, Yunqing Xia, Yuqing Yang, Ting Cao, Hao Sun, Weiwei Deng, Qi Zhang, and Mao Yang. 2023. Constraint-aware and ranking-distilled token pruning for efficient transformer inference. In Proceedings of the 29th ACM SIGKDD Conference on Know… find
  20. Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. 2023. Less is more: Task-aware layer-wise distillation for language model compression. find
  21. Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, and Weizhu Chen. 2025. Rho-1: Not all tokens are what you need. find
  22. Hanmeng Liu, Leyang Cui, Jian Liu, and Yue Zhang. 2021. Natural language inference in context - investigating contextual reasoning over long texts. Proceedings of the AAAI Conference on Artificial Intelligence, 35(15):13388–13396. find
  23. Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014. SemEval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of the 8th Internatio…
  24. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2023. Mteb: Massive text embedding benchmark.
  25. Khai Nguyen. 2025. An introduction to sliced optimal transport. arXiv preprint arXiv:2508.12519. — [arXiv:2508.12519](https://arxiv.org/abs/2508.12519) find
  26. Toan Ngoc Nguyen, Nam Le Hai, Nguyen Doan Hieu, Dai An Nguyen, Linh Ngo Van, Thien Huu Nguyen, and Sang Dinh. 2025. Improving vietnamese-english cross-lingual retrieval for legal and general domains. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for… find
  27. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
  28. Thanh Duc Pham, Nam Le Hai, Linh Ngo Van, Nguyen Thi Ngoc Diep, Sang Dinh, and Thien Huu Nguyen. 2025. Mitigating non-representative prototypes and representation bias in few-shot continual relation extraction. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistic… find
  29. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. find
  30. Eva Sharma, Chen Li, and Lu Wang. 2019. BIG-PATENT: A large-scale dataset for abstractive and coherent summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2204–2213, Florence, Italy. Association for Computational Linguistics. find
  31. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for bert model compression.
  32. Cédric Villani. 2008. Optimal transport – Old and new, volume 338, pages xxii+973. find
  33. Tu Vu, Manh Do, Tung Nguyen, Linh Ngo Van, Sang Dinh, and Thien Huu Nguyen. 2025. Topic modeling for short texts via optimal transport-based clustering. In Findings of the Association for Computational Linguistics: ACL 2025, pages 7666–7680. find
  34. Hoang Tran Vuong, Tue Le, Tu Vu, Tung Nguyen, Linh Ngo Van, Sang Dinh, and Thien Huu Nguyen. 2025. Hicot: Improving neural topic models via optimal transport and contrastive learning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 13894–13920. find
  35. Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. 2024. Knowledge fusion of large language models. find
  36. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. find
  37. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings.
  38. Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. 2025. Jasper and stella: distillation of sota embedding models. find
  39. Songming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen, and Jinan Xu. 2024. Dual-space knowledge distillation for large language models.
  40. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2025. A survey of l…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/truong-2025-emo-embedding-model-distillation-intra/