Elamur Research⌕
☰
home / papers / 2026 / Embedding Similarity Is Not Quality Estimation: Lessons from Replacing a Dedicated QE Model

Embedding Similarity Is Not Quality Estimation: Lessons from Replacing a Dedicated QE Model

Summary

The paper asks whether a production machine translation quality estimation (QE) component can swap its dedicated COMET-family model (built on XLM-RoBERTa-large) for a cheaper scorer: cosine similarity between Gemini embeddings (gemini-embedding-001) of a source segment and its translation. The authors use 10,000 proprietary segments with binary human acceptability labels, drawn from six client accounts and covering 38 target languages. They run three experiments: removing outlier ("rogue") embedding dimensions, recalibrating scores, and training a LightGBM classifier. They then diagnose the underlying cause. The cosine scorer roughly matches COMET (AUC 0.627 vs. 0.631). Neither dimension removal nor monotone recalibration improves it, because similarity values cluster tightly between 0.94 and 0.99 regardless of quality: flawed translations still keep most of the source meaning. A LightGBM head that adds surface-level text features to normalized cosine reaches AUC 0.751, with most of the gain coming from the non-embedding signals. For localization pipelines, the takeaway is that raw embedding similarity cannot directly replace trained QE, but a light learned model is a workable alternative.

Key points

  • Data: 10,000 segments sampled from a pool of 2.1M across six anonymized accounts, mostly English source (some Swedish). 41.1% are labeled acceptable. The split is 70/30 stratified (7,000/3,000).
  • Rogue dimensions exist but do no harm. Dimension 115 has a mean of −0.242, and the top 5 dimensions contribute 24.9% of cosine similarity. Removing them still leaves r² = 0.9991 with the original scores, because they act as a near-constant offset.
  • Calibration cannot raise discrimination. Monotonic normalization and isotonic regression both stay at AUC 0.627. A logistic-regression feature combination drops to 0.561 (0/6 qualifying accounts), since Gemini vectors are L2-normalized and the distance features simply repeat cosine.
  • Saturation root cause: the mean cosine gap between classes is only 0.004 (0.951 acceptable vs. 0.947 needs review). Unlike COMET's [e_s; e_t; |e_s−e_t|; e_s⊙e_t] design, a single scalar discards element-wise differences.
  • The LightGBM calibration head (word-count-weighted training) achieves AUC 0.751, 2/6 qualifying accounts, and a 34.8% best per-account word acceptance rate, compared with 14.4% for the cosine baseline. Surface-level features outweigh the embedding-derived ones in feature importance.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zaikis2026embedding,
  title = {{Embedding Similarity Is Not Quality Estimation: Lessons from Replacing a Dedicated QE Model}},
  author = {Zaikis, Dimitrios and Biondo, Andrea and Dixon, Matthew and Karageorgos, Konstantinos and Schliem, Aaron},
  booktitle = {Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 2)},
  year = {2026},
  url = {https://aclanthology.org/2026.eamt-2.26/}
}

References

13 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cabeça, Mariana, Marianna Buchicchio, Madalena Gonçalves, Christine Maroti, João Godinho, Pedro Coelho, Helena Moniz, and Alon Lavie. 2023. Quality fit for purpose: Building business critical errors test suites. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation (EAMT), pages 451–460, Tampere, Finland, June. European Association for Machine Translation. find
  2. Dinh, Tu Anh, Tobias Palzer, and Jan Niehues. 2024. Quality estimation with k-nearest neighbors and automatic evaluation for model-specific quality estimation. In Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1), pages 133–146, Sheffield, UK, June. European Association for Machine Translation (EAMT). ↗
  3. Gladkoff, Serge, Lifeng Han, Gleb Erofeev, Irina Sorokina, and Goran Nenadic. 2024. MTUncer-tainty: Assessing the need for post-editing of machine translation outputs by fine-tuning OpenAI LLMs. In Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1), pages 337–346, Sheffield, UK, June. European Association for Machine Translation (EAMT). find
  4. Guerreiro, Nuno M., Ricardo Rei, Daan Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2024. xCOMET: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979–995. ↗ our card
  5. Ke, Guolin, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems 30 (NeurIPS), pages 3149–3157. find
  6. Lee, Jinhyuk, Feiyang Chen, Sahil Dua, Daniel Cer, and Madhuri Shanbhogue. 2025. Gemini embedding: Generalizable embeddings from gemini. arXiv preprint arXiv:2503.07891. ↗
  7. Niculescu-Mizil, Alexandru and Rich Caruana. 2005. Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference on Machine Learning (ICML), pages 625– 632. ACM. find
  8. Rei, Ricardo, Craig Stewart, Ana C. Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702. Association for Computational Linguistics. find our card
  9. Rei, Ricardo, Marcos Treviso, Nuno M. Yu, António C. Guerreiro, José G. C. de Souza, and Alon Lavie. 2022. CometKiwi: IST-Unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634–645. Association for Computational Linguistics. ↗ our card
  10. Steck, Harald, Chaitanya Ekanadham, and Nathan Kallus. 2024. Is cosine-similarity of embeddings really about similarity? In Companion Proceedings of the ACM Web Conference 2024. ACM. find
  11. Sun, Kun and Rong Wang. 2024. Textual similarity as a key metric in machine translation quality estimation. arXiv preprint arXiv:2406.07440. ↗
  12. Timkey, William and Marten van Schijndel. 2021. All bark and no bite: Rogue dimensions in transformer language models obscure representational quality. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4527–4546. Association for Computational Linguistics. find
  13. Zhao, Haofei, Yilun Liu, Shimin Tao, Weibin Meng, Yimeng Chen, Xiang Geng, Chang Su, Min Zhang, and Hao Yang. 2024. From handcrafted features to LLMs: A brief survey for machine translation quality estimation. arXiv preprint arXiv:2403.14118. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/zaikis-2026-embedding-similarity-is-not-quality/