Elamur Research⌕
☰
home / papers / 2026 / LLM-as-a-Jury for Machine Translation Publishability Assessment

LLM-as-a-Jury for Machine Translation Publishability Assessment

Summary

The paper asks whether raw machine translation can go live in enterprise workflows without human post-editing. The authors frame this as a yes/no decision. A segment is unpublishable if professional linguists found at least one major or critical MQM error, while minor and neutral issues are tolerated. Rather than relying on one LLM evaluator, they assemble a jury of models from different families. A logistic regression, fitted on a separately held-out split, learns how much weight each juror deserves. Three setups are compared:

  • an existing Edit Effort Estimation (EEE) prompt,
  • an MQM-style LQA prompt,
  • a dedicated Publishability pipeline combining glossary matching, BM25-retrieved examples, GPT-4o-induced rules, DSPy prompt optimization and per-domain fine-tuned jurors.

The data are Smartling's linguist-annotated segments covering three domains and nine en-US-source locales. Across almost all conditions, the jury equals or beats its strongest member, and it repairs individual juror mistakes much more often than it introduces new ones. The generic prompts turn out to be strong baselines on macro-F1, whereas the Publishability setup stands out for precision. Overall, the results support automated publish-or-review gating.

Key points

  • Data: The data covers Logistics (de-DE, el-GR, fr-FR), IT (es-ES, id-ID, it-IT, pt-BR) and Media (es-US, ja-JP). Each dataset is split four ways: train, validation, LR-train and test.
  • Prompt optimization: DSPy optimizers were tested on Gemini 2.5 Pro. SIMBA performed best (macro-F1 0.48, about $162) versus GEPA (0.46) and MIPROv2 (0.42), and its prompt was adopted for all jurors.
  • Jury composition: Both the EEE and LQA juries consist of three models: Claude Opus 4.5, GPT-5.2, and the preview release of Gemini 3.1 Pro. The Publishability jury has five members, adding GPT-5.4 plus fine-tuned GPT-4o and Gemini 2.5 Pro.
  • Error correction and weights: Fix-to-break ratios are roughly 4–7× for EEE (medium counted as unpublishable), 5–8× for Publishability and about 2× for LQA. The learned weights can sideline unreliable jurors, for example GPT-5.4 at −1.29 in IT.
  • Fine-tuning: Jury recall rose by 20.3 points in Logistics and 23.5 points in Media, with the authors citing gains of up to 43 points. IT saw no benefit, and several hundred labeled segments per locale appear necessary.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{yanishevsky2026llm,
  title = {{LLM-as-a-Jury for Machine Translation Publishability Assessment}},
  author = {Yanishevsky, Alex and Norris, Olivia},
  booktitle = {Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 1)},
  year = {2026},
  url = {https://aclanthology.org/2026.eamt-1.7/}
}

References

14 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anonymous. 2026. Using model disagreement to identify unstable regions in MT evaluation. Under review. find
  2. Fernandes, Patrick, Daniel Deutsch, Mara Finkelstein, Parker Riley, André F. T. Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. 2023. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. In Proceedings of the Eighth Conference on Machine Translation (WMT 2023), pages 1066–1083. ↗ our card
  3. Freitag, Markus, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. ↗ our card
  4. Khattab, Omar, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2023. DSPy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714. ↗ our card
  5. Kocmi, Tom and Christian Federmann. 2023a. GEMBA-MQM: Detecting translation quality error spans with GPT-4. In Proceedings of the Eighth Conference on Machine Translation (WMT 2023), pages 768–775. find our card
  6. Kocmi, Tom and Christian Federmann. 2023b. Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation (EAMT 2023), pages 193–203. ↗
  7. Lannelongue, Loı̈c, Jason Grealey, and Michael Inouye. 2021. Green algorithms: Quantifying the carbon footprint of computation. Advanced Science, 8(12):2100707. ↗
  8. Li, Xinpeng, Kuan Wang, Gautam Gouda, Shubham Choudhary, Yanfu Wang, Lifu Hu, Jennifer Vaughan, and Freddy Lecue. 2025. Who judges the judge? LLM jury-on-demand: Building trustworthy LLM evaluation systems. arXiv preprint arXiv:2512.01786. ↗
  9. Liu, Yang, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG evaluation using GPT-4 with better human alignment. arXiv preprint arXiv:2303.16634. ↗
  10. Lu, Qingyu, Baopu Qiu, Liang Ding, Kanjian Zhang, Tom Kocmi, and Dacheng Tao. 2024. Error analysis prompting enables human-like translation evaluation in large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 8801–8816. find our card
  11. Qian, Mengie, Guangzhi Sun, Mark Gales, and Kate Knill. 2026. Who can we trust? LLM-as-a-jury for comparative assessment. arXiv preprint arXiv:2602.16610. ↗
  12. Verga, Pat, Sebastian Hofstätter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. Replacing judges with juries: Evaluating LLM generations with a panel of diverse models. arXiv preprint arXiv:2404.18796. ↗ our card
  13. Zhang, Hongbin, Kehai Chen, Xuefen Bai, Youcheng Pan, Yang Xiang, Jinpeng Wang, and Min Zhang. 2025. Mitigating translationese bias in multilingual LLM-as-a-judge via disentangled information bottleneck. arXiv preprint arXiv:2603.10351. ↗
  14. Zheng, Lianmin, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. arXiv preprint arXiv:2306.05685. ↗

This page in Markdown → https://md.research.elamur.ai/free/papers/yanishevsky-2026-llm-as-jury-machine-translation/