Elamur Research⌕
☰
home / papers / 2026 / MTQE.en-he: Machine Translation Quality Estimation for English-Hebrew

MTQE.en-he: Machine Translation Quality Estimation for English-Hebrew

Summary

English-Hebrew lacked a public benchmark for estimating machine translation quality without reference translations. Hebrew is a mid-resource language whose rich morphology makes data scarcity worse. The authors built MTQE.en-he from 959 English segments drawn from WMT24++, covering literary, news, social and speech text. Each segment was translated into Hebrew with Google Translate and given a Direct Assessment (DA) score from 0 to 100 by three bilingual experts. The mean of the three scores serves as ground truth.

They evaluated ChatGPT prompting, TransQuest and CometKiwi against this ground truth. CometKiwi was the strongest single system, but averaging the outputs of all three models worked noticeably better. Fine-tuning on a 300-segment training slice showed two patterns:

  • Full fine-tuning either brought no gain or reduced quality.
  • Lightweight methods (LoRA, BitFit, head-only tuning) trained stably and improved results modestly.

The released data, seeded splits and baselines give future English-Hebrew QE research a common starting point. They also show which adaptation strategies hold up with very small training sets.

Key points

  • Dataset construction: WMT24++ originally had 998 segments. The authors dropped 38 flagged as bad sources and one duplicate. Mean inter-annotator agreement (Pearson) was 0.5337.
  • Score distribution: Scores skew high, with 73% of segments at 70 or above. Segments are mostly short, with 59% under 30 words.
  • Best single model vs. ensemble: CometKiwi reached 0.4828 Pearson and 0.5456 Spearman on the full set. Ensembling ChatGPT-freestyle, TransQuest-en-any and CometKiwi raised this to 0.5472 Pearson and 0.6014 Spearman.
  • Fine-tuning setup and results: The split was 300 train, 100 validation and 559 test, repeated over 5 stratified seeds. Full fine-tuning showed overfitting and distribution collapse. LoRA, BitFit and FTHead added about 2–3 points.
  • Limitations: The labels use DA rather than MQM, the translations come from a single MT system, and there are few low-quality examples.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.loreslm-1.48

@inproceedings{rosenbaum2026mtqe,
  title = {{MTQE.en-he: Machine Translation Quality Estimation for English-Hebrew}},
  author = {Rosenbaum, Andy and Siani, Assaf and Kernerman, Ilan},
  booktitle = {Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loreslm-1.48},
  url = {https://doi.org/10.18653/v1/2026.loreslm-1.48}
}

References

16 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1–9, Dublin, Ireland. Association for Computational Linguistics. ↗ our card
  2. Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. MetricX-24: The Google submission to the WMT 2024 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pages 492–504, Miami, Florida, USA. Association for Computational Linguistics. find our card
  3. C.M. Bishop. 2006. Pattern recognition and machine learning, volume 4. Springer New York. find
  4. OpenAI. 2024. Chatgpt. https://chat.openai.com/. Accessed Jan 2026. ↗
  5. Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3576–3588, Online. Association for Computational Linguistics. ↗ our card
  6. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, and 2 others. 2019. Pytorch: An imperative style, high-performance deep learning library. Preprint, arXiv:1912.01703. ↗ our card
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  8. Daniel Deutsch, Eleftheria Briakou, Isaac Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, Shruti Rijhwani, Parker Riley, Elizabeth Salesky, Firas Trabelsi, Stephanie Winkler, Biao Zhang, and Markus Freitag. 2025. Wmt24++: Expanding the language coverage of wmt24 to 55 languages & dialects. Preprint, arXiv:2502.12404. ↗
  9. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2017. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, 23(1):3–30. find
  10. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685. ↗
  11. Mohammed Hossein Jafari Harandi, Fatemeh Azadi, Mohammad Javad Dousti, and Heshaam Faili. 2024. EPOQUE: An English-Persian quality estimation dataset. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 6228–6235, Torino, Italia. ELRA and ICCL. find
  12. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2020. TransQuest: Translation quality estimation with cross-lingual transformers. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5070–5081, Barcelona, Spain (Online). International Committee on Computational Linguistics. find our card
  13. Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André F. T. Martins, and Alon Lavie. 2021. Are references really needed? unbabel-IST 2021 submission for the metrics shared task. In Proceedings of the Sixth Conference on Machine Translation, pages 1030–1040, Online. Association for Computational Linguistics. find our card
  14. Reut Tsarfaty, Shoval Sadde, Stav Klein, and Amit Seker. 2019. What’s wrong with Hebrew NLP? and how to make it right. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 259–264, Hong Kong, China. Association for Computational Linguistics. find our card
  15. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. find our card
  16. Haofei Zhao, Yilun Liu, Shimin Tao, Weibin Meng, Yimeng Chen, Xiang Geng, Chang Su, Min Zhang, and Hao Yang. 2024. From handcrafted features to llms: A brief survey for machine translation quality estimation. In 2024 International Joint Conference on Neural Networks (IJCNN), page 1–10. IEEE. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/rosenbaum-2026-mtqe-en-he-machine-translation/