Elamur Research⌕
☰
home / papers / 2026 / Disentangling Meaning and Language Components in Diverse Multilingual Sentence Embeddings

Disentangling Meaning and Language Components in Diverse Multilingual Sentence Embeddings

Paper · 2026 · Kanade Nonomura, Keita Fukushima, Risa Kondo et al.

Summary

Multilingual sentence embeddings often carry language-identity signals, so vectors group by language instead of by meaning, which weakens cross-lingual similarity estimation. Earlier disentanglement methods split an embedding into a meaning part and a language part, but they were tested mainly on encoder models that receive only the raw sentence. This student-workshop paper tests whether disentanglement still helps for newer decoder-based and prompt-based embeddings. The authors classify embeddings along four axes: architecture, whether the model was fine-tuned for embeddings, extraction method (input only or prompt), and disentanglement method. They train DREAM, MEAT and SEED as single-layer MLPs on frozen models, using parallel corpora. Evaluation is unsupervised, on the WMT20 QE task across six language pairs. Disentanglement improved Pearson correlation with human scores in almost every configuration. Gains were largest for pretrained models without embedding fine-tuning and smaller for decoder models. An Alignment/Uniformity analysis links the size of the gain to whether the model was fine-tuned. This matters because cheap post-hoc disentanglement can upgrade even raw LLMs for cross-lingual QE.

Key points

  • Models: mBERT/LaBSE and XLM-R/mE5 (encoders); Llama3.1-8B/Llama-Embed-Nemotron-8B and Qwen3-8B/Qwen3-Embedding-8B (decoders).
  • Prompting: raw models were prompted with PromptBERT (encoders) or PromptEOL (decoders).
  • Training data: parallel pairs sampled from the WMT20 QE resources:
    • 1,000k each for en–de and en–zh;
    • 200k each for ro–en and et–en;
    • 50k each for ne–en and si–en.
  • Gains for raw models: the boosts were dramatic. Input-only Llama3.1 rose from 0.013 to 0.311 with DREAM. Prompted Qwen3 rose from 0.156 to 0.412 with SEED.
  • Gains for embedding models: improvements were modest. The best overall score was 0.531, from Llama-Emb. with prompt and MEAT, against a baseline of 0.524. DREAM actually hurt both decoder embedding models.
  • Geometric analysis: Alignment and Uniformity (Wang and Isola, 2020) improved more for pretrained models than for fine-tuned ones. For mBERT, a large Uniformity gain outweighed a slight loss in Alignment.
  • Limitations: only the QE task was evaluated, and embeddings were split into just two factors (meaning and language).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of OsakaOsaka · Japan15 works · 517 citations🎓Ehime UniversityMatsuyama · Japan5 works · 3 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 works

Cite

Cite as Kanade Nonomura, Keita Fukushima, Risa Kondo and Tomoyuki Kajiwara. 2026. Disentangling Meaning and Language Components in Diverse Multilingual Sentence Embeddings. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-srw.102

@inproceedings{nonomura2026disentangling,
  title = {{Disentangling Meaning and Language Components in Diverse Multilingual Sentence Embeddings}},
  author = {Kanade Nonomura, Keita Fukushima, Risa Kondo and Tomoyuki Kajiwara},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-srw.102},
  url = {https://doi.org/10.18653/v1/2026.acl-srw.102}
}

References

28 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yauhen Babakhin, Radek Osmulski, Ronay Ak, Gabriel Moreira, Mengyao Xu, Benedikt Schifferer, Bo Liu, and Even Oldridge. 2025. Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks. arXiv:2511.07025. — [arXiv:2511.07025](https://arxiv.org/abs/2511.07025) find
  2. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation, pages 1–14.
  3. Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019. A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies… find
  4. Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, and James Glass. 2022. DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Associ…
  5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  7. Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Gabriel Sequeira, Diganta Misra, Shreeya Dhakal, Jonathan Rystrøm, Roman Soloma… — MMTEB: Massive Multilingual Text Embedding Benchmark
  8. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic BERT Sentence Embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 878–891.
  9. Keita Fukushima, Tomoyuki Kajiwara, and Takashi Ninomiya. 2025. Reversible Disentanglement of Meaning and Language Representations from Multilingual Sentence Encoders. In Proceedings of the 5th Workshop on Multilingual Representation Learning, pages 265–270. find
  10. Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. 2019. Representation Degeneration Problem in Training Natural Language Generation Models. In International Conference on Learning Representations.
  11. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple Contrastive Learning of Sentence Embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910.
  12. James Y. Huang, Kuan-Hao Huang, and Kai-Wei Chang. 2021. Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, page… find
  13. Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang. 2024. Scaling Sentence Embeddings with Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3182–3196.
  14. Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, and Qi Zhang. 2022. Prompt-BERT: Improving BERT Sentence Embeddings with Prompts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8826–… find
  15. Dayeon Ki, Cheonbok Park, and Hyunjoong Kim. 2024. Mitigating Semantic Leakage in Cross-lingual Embeddings via Orthogonality Constraint. In Proceedings of the 9th Workshop on Representation Learning for NLP, pages 256–273. find
  16. Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference for Learning Representations.
  17. Risa Kondo, Hiroki Yamauchi, Tomoyuki Kajiwara, Marie Katsurai, and Takashi Ninomiya. 2025. Domain Knowledge Distillation for Multilingual Sentence Encoders in Cross-lingual Sentence Similarity Estimation. In Proceedings of the 15th International Conference on Recent Advances in Natural Language Pro… find
  18. Yuto Kuroda, Tomoyuki Kajiwara, Yuki Arase, and Takashi Ninomiya. 2022. Adversarial Training on Disentangling Meaning and Language Representations for Unsupervised Quality Estimation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5240–5245. find
  19. Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the Sentence Embeddings from Pre-trained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9119–9130.
  20. Chong Li, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2024. Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language… find
  21. Jindřich Libovický, Rudolf Rosa, and Alexander Fraser. 2020. On the Language Neutrality of Pre-trained Multilingual Representations. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1663–1674.
  22. Llama Team. 2024. The Llama 3 Herd of Models. arXiv:2407.21783.
  23. Kanade Nonomura, Keita Fukushima, Risa Kondo, and Tomoyuki Kajiwara. 2026. Mitigating Language Bias in Multilingual Sentence Embeddings for Cross-Lingual Similarity Estimation. In Proceedings of the 15th Joint Conference on Lexical and Computational Semantics.
  24. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 48…
  25. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual E5 Text Embeddings: A Technical Report. arXiv:2402.05672.
  26. Tongzhou Wang and Phillip Isola. 2020. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. In Proceedings of the 37th International Conference on Machine Learning, pages 9929–9939. find
  27. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transf…
  28. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. arXiv:2506.05176.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nonomura-2026-disentangling-meaning-language-components-diverse/