Elamur Research⌕
☰
home / papers / 2026 / Lost in Translation: Repurposing semantic similarity benchmarks for evaluating lexical-semantic consistency in LLM-based machine translation

Lost in Translation: Repurposing semantic similarity benchmarks for evaluating lexical-semantic consistency in LLM-based machine translation

Summary

Human-rated word-similarity datasets such as SimLex-999 and Multi-SimLex were built to test word embeddings. Because their concept pairs are aligned across languages, the authors reuse them to check whether multilingual LLMs keep lexical meaning stable when moving between languages. They manually checked English–Mandarin pairs from Multi-SimLex and kept only those with direct, unambiguous translations. Using zero-shot prompts, they asked BLOOMZ-7b1 and GPT-4 to rate similarity in both languages and to translate and back-translate the pairs. The outputs were compared with human ratings, Levenshtein distance (a proxy for surface-form stability) and multilingual SentenceBERT embeddings (a proxy for semantic drift). Translation serves here as a stress test, not as a new MT metric. GPT-4's ratings agreed well with human judgments. BLOOMZ's ratings barely tracked humans, clustered in the middle of the scale, and became unstable under back-translation. Since SimLex-style resources exist for about 25 languages, the method could extend to many language pairs, including pairs without English, given some manual verification.

Key points

  • Data: Multi-SimLex English and Mandarin subsets (1,888 pairs per language, 0–6 scale). The abstract describes a manually verified subset of 500 pairs, of which 423 with accurate translations were used in the main analysis. A single coder did the verification.
  • Agreement with human ratings (Spearman ρ): GPT-4 reached about 0.77 for English and 0.75 for Mandarin. BLOOMZ reached only 0.12 for English and 0.20 for Mandarin. BLOOMZ's cross-lingual difference correlation was ρ = −0.05, so it does not mirror human perception of meaning shifts between the languages.
  • BLOOMZ translation stability:
    • Forward translation kept about 50% of tokens, but back-translation kept only about 30%.
    • Levenshtein differences were 1–2 characters on forward passes, with the spread rising sharply on back-translation (std ≈ 3.0–3.8).
    • Ratings before and after translation correlated weakly (r = 0.33 for Mandarin, 0.21 for English).
  • Score compression: BLOOMZ mostly produced mid-range integer ratings (2–5), which makes its poor alignment with humans hard to interpret.
  • Limitations: SimLex data may appear in LLM pretraining corpora, which could inflate results.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3n3847mvjzk2

@inproceedings{ye2026lost,
  title = {{Lost in Translation: Repurposing semantic similarity benchmarks for evaluating lexical-semantic consistency in LLM-based machine translation}},
  author = {Ye, Quin and Bloem, Jelke},
  booktitle = {Proceedings of the Fourth Workshop on the Role of Resources in the Age of Large Language Models (RESOURCEFUL 2026)},
  year = {2026},
  doi = {10.63317/3n3847mvjzk2},
  url = {https://doi.org/10.63317/3n3847mvjzk2}
}

References

30 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rachel Bawden and François Yvon. 2023. Investigating the translation performance of a large multilingual language model: The case of BLOOM. ↗
  2. Rishi Bommasani, Kelly Davis, and Claire Cardie. 2020. Interpreting pretrained contextualized representations via reductions to static embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4758–4781, Online. Association for Computational Linguistics. find
  3. Lizzy Brans and Jelke Bloem. 2024. SimLex-999 for Dutch. In Proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation (LREC-COLING 2024), pages 14832–14845, Torino, Italia. ELRA; ICCL. find
  4. Lizzy Brans and Jelke Bloem. in press. Multi-SimLex for Dutch: Benchmarking embedding-and prompt-based model performance on semantic similarity. In Proceedings of the 15th Language Resources and Evaluation Conference (LREC 2026)). European Language Resources Association. find
  5. Yijie Chen, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, and Jie Zhou. 2023. Improving translation faithfulness of large language models via augmenting instructions. ↗
  6. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500, Brussels, Belgium. Association for Computational Linguistics. ↗
  7. Anton Ehrmanntraut, Thora Hagen, Leonard Konle, and Fotis Jannidis. 2021. Type-and token-based word embeddings in the digital humanities. In Computational Humanities Research Conference, pages 16–38. find
  8. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Armand Joulin. 2021. Beyond English-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48. find
  9. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: The concept revisited. In Proceedings of the 10th international conference on World Wide Web, pages 406–414. find
  10. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695. ↗ our card
  11. Ggaliwango Marvin, Nakayiza Hellen, Daudi Jjingo, and Joyce Nakatumba-Nabende. 2024. Prompt engineering in large language models. In Data intelligence and cognitive informatics, pages 387– 402, Singapore. Springer Nature. find
  12. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26. ↗
  13. Leonidas Mylonadis and Jelke Bloem. in press. SimLex-999 for Modern Greek. In Proceedings of the 4th Annual Meeting of the Special Interest Group on Under-resourced Languages (SIGUL 2026) @ LREC 2026. find our card
  14. Aarathi Rajagopalan Nair, Deepa Gupta, and B. Premjith. 2024. Investigating translation for Indic languages with BLOOMZ-3b through prompting and LoRA fine-tuning. Scientific Reports, 14(1):24202. ↗
  15. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. ↗
  16. OpenAI. 2023. GPT-4 technical report. Accessed October 2025. ↗ our card
  17. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗ our card
  18. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  19. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics. ↗
  20. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512–4525, Online. Association for Computational Linguistics. find our card
  21. Nathaniel Robinson, Perez Ogayo, David R. Mortensen, and Graham Neubig. 2023. Chat-GPT MT: Competitive for high- (but not low-) resource languages. In Proceedings of the Eighth Conference on Machine Translation, pages 392– 418, Singapore. Association for Computational Linguistics. find
  22. Teven Le Scao et al. 2023. BLOOM: A 176Bparameter open-access multilingual language model. find
  23. Xander Snelder, Yunchong Huang, and Jelke Bloem. in press. Prompting instruction-tuned LLMs for semantic similarity values. In Proceedings of the 15th Language Resources and Evaluation Conference (LREC 2026). European Language Resources Association. find our card
  24. Miloš Stanojević and Khalil Sima’an. 2015. Evaluating MT systems with BEER. The Prague Bulletin of Mathematical Linguistics, 104(1):17–26. ↗
  25. Sean Trott. 2024. Can large language models help augment English psycholinguistic datasets? Behavior Research Methods, pages 1–19. ↗
  26. Ivan Vulić, Simon Baker, Edoardo Maria Ponti, Ulla Petti, Ira Leviant, Kelly Wing, Olga Majewska, Eden Bar, Matt Malone, Thierry Poibeau, Roi Reichart, and Anna Korhonen. 2021. Multi-SimLex: A large-scale evaluation of multilingual and crosslingual lexical semantic similarity. Computational Linguistics, 46(4):847–897. ↗
  27. Li Weigang and Pedro Carvalho Brom. 2025. LLM-BT-terms: Back-translation as a framework for terminology standardization and dynamic semantic embedding. ↗
  28. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. ↗
  29. Li Yujian and Liu Bo. 2007. A normalized Levenshtein distance metric. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(6):1091–1095. ↗
  30. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/ye-2026-lost-translation-repurposing-semantic-similarity/