Elamur Research⌕
☰
home / papers / 2026 / Mitigating Language Bias in Multilingual Sentence Embeddings for Cross-Lingual Similarity Estimation

Mitigating Language Bias in Multilingual Sentence Embeddings for Cross-Lingual Similarity Estimation

Summary

Multilingual sentence embeddings tend to group by language, which hurts cross-lingual tasks. Disentanglement methods address this by splitting an embedding into a language-agnostic "meaning" part and a language-specific part. This paper takes the SEED framework, where an MLP extracts the meaning vector and the language vector is the residual. It splits SEED's training objective into two families:

  • Intra-component constraints: pull meaning vectors of parallel sentences together and language vectors of same-language sentences together.
  • Inter-component constraints: swap components between related sentences and require the recombined embeddings to stay consistent.

The authors test each family alone and both together on encoder-based and LLM-based embedders, using WMT20 QE and SemEval-2017 STS.

  • Encoder models: work best with the combination of both, with intra-only second.
  • Decoder models: do best with inter-only.

A geometric analysis explains the split. Intra losses mainly spread embeddings more evenly (uniformity), while inter losses mainly bring translations closer (alignment). This gives practical guidance for choosing debiasing objectives per architecture.

Key points

  • Encoder-based models: LaBSE (Feng et al., 2022) and the Wang et al. (2024) model, both with 768–1024 dimensions.
  • Decoder/LLM-based models: Gemini Embedding (3072 dimensions) and Qwen3-Embedding-8B (4096 dimensions).
  • Benchmarks: WMT20 machine translation QE and SemEval-2017 cross-lingual STS. Disentanglers are trained on bilingual parallel corpora.
  • Geometry: Using the uniformity/alignment metrics of Wang and Isola (2020), intra (or combined) training clearly improves uniformity but weakens alignment. Inter-only training leaves uniformity roughly unchanged and greatly improves alignment.
  • Dimensionality: The LLM embeddings were truncated via Matryoshka Representation Learning (MRL) and tested on WMT20 QE. Inter-only remained best at every size, so the architectural difference is not explained by embedding dimensionality. Intra losses appear to over-regularize high-dimensional LLM spaces.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.starsem-conference.26

@inproceedings{nonomura2026mitigating,
  title = {{Mitigating Language Bias in Multilingual Sentence Embeddings for Cross-Lingual Similarity Estimation}},
  author = {Nonomura, Kanade and Fukushima, Keita and Kondo, Risa and Kajiwara, Tomoyuki},
  booktitle = {Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.starsem-conference.26},
  url = {https://doi.org/10.18653/v1/2026.starsem-conference.26}
}

References

20 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation, pages 1–14. find
  2. Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, and James Glass. 2022. DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Associ… find
  3. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  4. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple Contrastive Learning of Sentence Embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910. find our card
  5. Dayeon Ki, Cheonbok Park, and Hyunjoong Kim. 2024. Mitigating Semantic Leakage in Cross-lingual Embeddings via Orthogonality Constraint. In Proceedings of the 9th Workshop on Representation Learning for NLP, pages 256–273. find
  6. Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference for Learning Representations. find
  7. Risa Kondo, Hiroki Yamauchi, Tomoyuki Kajiwara, Marie Katsurai, and Takashi Ninomiya. 2025. Domain Knowledge Distillation for Multilingual Sentence Encoders in Cross-lingual Sentence Similarity Estimation. In Proceedings of the 15th International Conference on Recent Advances in Natural Language Pro… find
  8. Yuto Kuroda, Tomoyuki Kajiwara, Yuki Arase, and Takashi Ninomiya. 2022. Adversarial Training on Disentangling Meaning and Language Representations for Unsupervised Quality Estimation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5240–5245. find
  9. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual E5 Text Embeddings: A Technical Report. arXiv:2402.05672. find our card
  10. Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. 2022. Matryoshka Representation Learning. In Advances in Neural Information Processing Systems, pages 30233–30249. find
  11. Tongzhou Wang and Phillip Isola. 2020. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. In Proceedings of the 37th International Conference on Machine Learning, pages 9929–9939. find
  12. Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, Madhuri Shanbhogue, Iftekhar Naim, Gustavo Hernández Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter Vera, Xiaoqi Ren, Shanfeng Zhang, Daniel Salz, Michael Boratko, Jay Han, Blair Chen, Shuo Huang, Vikram Rao, Paul Suganthan, and 28 others. 2025. Ge… find
  13. Chong Li, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2024. Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language… find
  14. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transf… find our card
  15. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. arXiv:2506.05176. find our card
  16. Kanade Nonomura, Keita Fukushima, Risa Kondo, and Tomoyuki Kajiwara. 2026. Disentangling Meaning and Language Components in Diverse Multilingual Sentence Embeddings. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop. find
  17. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr… find our card
  18. Lucia Specia, Frédéric Blain, Marina Fomicheva, Erick Fonseca, Vishrav Chaudhary, Francisco Guzmán, and André F. T. Martins. 2020. Findings of the WMT 2020 Shared Task on Quality Estimation. In Proceedings of the Fifth Conference on Machine Translation, pages 743–764. find our card
  19. Nattapong Tiyajamorn, Tomoyuki Kajiwara, Yuki Arase, and Makoto Onizuka. 2021. Language-agnostic Representation from Multilingual Sentence Encoders for Cross-lingual Similarity Estimation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7764–7774. find
  20. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/nonomura-2026-mitigating-language-bias-multilingual-sentence/