Elamur Research⌕
☰
home / papers / 2024 / Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models

Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models

Summary

LLMs prompted to output a single quality score directly, as in GEMBA, rank MT systems well but judge individual sentences poorly and give no explanation. This paper introduces Error Analysis Prompting (EAPrompt), which merges chain-of-thought reasoning with the error-analysis idea of Lu et al. (2023) to imitate how MQM annotators work. The prompt has two turns. First, the model lists major and minor errors in a translation, guided by a one-shot example tailored to the language pair. Then it reports how many errors of each type it found. The counts become a weighted penalty score. The authors tested the method on the WMT22 metrics shared task test set (En-De, En-Ru, Zh-En; 106,758 segments from 54 systems) with GPT-3.5-Turbo, Llama2-70b-Chat and Mixtral-8x7b-Instruct, using MQM annotations as the human gold standard. EAPrompt improved results at both the system and segment level, and GPT-3.5-Turbo with EAPrompt set a new system-level state of the art. The method also held up without references. This makes LLM-based MT evaluation more interpretable and closer to human judgment.

Key points

  • Test data: WMT22 covering four domains (conversational, e-commerce, news, social).

    • En-De: 2037 segments, 17 systems.
    • En-Ru: 2037 segments, 17 systems.
    • Zh-En: 1875 segments, 20 systems.
    • WMT20/21 were left out on purpose to reduce the risk of training-data contamination.

  • Scoring formula: score = −w_major·n_major − w_minor·n_minor. The minor weight is fixed at 1, and the major weight is treated as a tunable latent variable, because different LLMs draw the major/minor line differently.

  • Segment-level comparison: EAPrompt outperformed GEMBA in 8 of 9 model × language-pair settings.

  • Best variant: two separate prompting steps (identify errors, then count them) combined with itemized error demonstrations.

  • Cheaper counting: replacing the counting query with regular-expression matching saves inference cost and loses little system-level accuracy:

    ModelRegexCounting query
    GPT-3.5-Turbo90.191.2
    Llama2-70b-Chat85.085.4
    Mixtral-8x7b-Instruct82.884.0

Interesting ideas

  • Detect first, aggregate later. Asking the model to find and classify errors before any number is produced makes the score traceable to concrete problems. This could carry over to judging summaries, data-to-text output, or other generated text.
  • Leave the severity weight open. Keeping the major-error weight as a parameter calibrated per model, rather than hard-coding it, accommodates evaluators with different strictness. This is handy when mixing or swapping judge models.
  • Itemized one-shot demos per language pair. Listing each error separately in a single example, chosen for the specific language pair, makes the model's output easier to parse and count. Such demos would be cheap to build for new pairs, including Ukrainian ones.
  • Parse instead of re-asking. Extracting counts from the first response with regex can replace a second LLM call at little accuracy cost. This is a general trick for cutting multi-turn evaluation costs.
  • Evaluate on post-cutoff test sets. Choosing test data that likely postdates the models' training data is a sensible guard against contamination when benchmarking LLM judges.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

104 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 89 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.findings-acl.520

@inproceedings{lu2024error,
  title = {{Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models}},
  author = {Lu, Qingyu and Qiu, Baopu and Ding, Liang and Zhang, Kanjian and Kocmi, Tom and Tao, Dacheng},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.520},
  url = {https://doi.org/10.18653/v1/2024.findings-acl.520}
}

References

56 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint. find
  2. Antonios Anastasopoulos, Ondřej Bojar, Jacob Bremerman, Roldano Cattoni, Maha Elbayad, Marcello Federico, et al. 2021. Findings of the IWSLT 2021 evaluation campaign. In IWSLT. find
  3. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint. find our card
  4. Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, et al. 2019. Findings of the 2019 conference on machine translation (WMT19). In WMT. find our card
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et al. 2020. Language models are few-shot learners. NeurIPS. find
  6. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. A survey on evaluation of large language models. ACM. ↗ our card
  7. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint. find
  8. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint. ↗ our card
  9. Daniel Deutsch, Rotem Dror, and Dan Roth. 2021. A statistical analysis of summarization evaluation metrics using resampling methods. TACL. ↗
  10. Daniel Deutsch, George Foster, and Markus Freitag. 2023. Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration. In EMNLP. find
  11. Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. 2023. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. In WMT. ↗ our card
  12. Markus Freitag, George Foster, David Grangier, et al. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. TACL. ↗ our card
  13. Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023. Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent. In WMT. find our card
  14. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, et al. 2022. Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust. In WMT. find our card
  15. Nuno M Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André FT Martins. 2023. xcomet: Transparent machine translation evaluation through fine-grained error detection. arXiv preprint. ↗ our card
  16. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, et al. 2023. How good are gpt models at machine translation? a comprehensive evaluation. arXiv preprint. ↗
  17. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint. find
  18. Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? a preliminary study. arXiv preprint. find
  19. Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. 2023. MetricX-23: The Google submission to the WMT 2023 metrics shared task. In WMT. find our card
  20. Tom Kocmi, Rachel Bawden, Ondřej Bojar, et al. 2022. Findings of the 2022 conference on machine translation (WMT22). In WMT. find
  21. Tom Kocmi and Christian Federmann. 2023a. GEMBA-MQM: Detecting translation quality error spans with GPT-4. In WMT. find our card
  22. Tom Kocmi and Christian Federmann. 2023b. Large language models are state-of-the-art evaluators of translation quality. In EAMT. ↗
  23. Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In WMT. ↗
  24. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In EMNLP. find
  25. Qingyu Lu, Liang Ding, Liping Xie, Kanjian Zhang, Derek F. Wong, and Dacheng Tao. 2023. Toward human-like evaluation for natural language generation with error analysis. In ACL. find
  26. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020a. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In ACL. find our card
  27. Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020b. Results of the WMT20 metrics shared task. In WMT. find our card
  28. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint. find
  29. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL. find our card
  30. Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023. Towards making the most of chatgpt for machine translation. arXiv preprint. ↗ our card
  31. Stefano Perrella, Lorenzo Proietti, Alessandro Scirè, Niccolò Campolungo, and Roberto Navigli. 2022. MaTESe: Machine translation evaluation as a sequence tagging problem. In WMT. find our card
  32. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language processing task solver? arXiv preprint. find our card
  33. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint. ↗ our card
  34. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint. ↗
  35. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint. find
  36. Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. 2023. Investigating the effectiveness of self-critiquing in LLMs solving planning tasks. In NeurIPS Foundation Models for Decision Making Workshop. find
  37. Blanca Vidal, Albert Llorens, and Juan Alonso. 2022. Automatic post-editing of MT output using large language models. In AMTA. find
  38. Yu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang, Boxing Chen, Derek Wong, and Lidia Chao. 2022. UniTE: Unified translation evaluation. In ACL. find our card
  39. Baopu Qiu, Liang Ding, Di Wu, Lin Shang, Yibing Zhan, and Dacheng Tao. 2022. Original or translated? on the use of parallel data for translation quality estimation. arXiv preprint. ↗
  40. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint. find our card
  41. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog. find
  42. Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. 2023. Chatgpt or grammarly? evaluating chatgpt on grammatical error correction benchmark. arXiv preprint. ↗
  43. Ricardo Rei, Nuno M. Guerreiro, José Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José G. C. de Souza, and André Martins. 2023. Scaling up CometKiwi: Unbabel-IST 2023 submission for the quality estimation shared task. In WMT. find our card
  44. Wenda Xu, Yi-Lin Tuan, Yujie Lu, Michael Saxon, Lei Li, and William Yang Wang. 2022. Not all errors are equal: Learning text generation metrics using stratified error synthesis. In EMNLP. find
  45. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In EMNLP. find our card
  46. Zhiyao Ren, Yibing Zhan, Baosheng Yu, Liang Ding, and Dacheng Tao. 2024. Healthcare copilot: Eliciting the power of general llms for medical consultation. arXiv preprint. ↗
  47. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In ACL. find
  48. Lucia Specia, Dhwaj Raj, and Marco Turchi. 2010. Machine translation evaluation versus quality estimation. Machine translation. ↗
  49. Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Wang, and Lei Li. 2023. INSTRUCTSCORE: Towards explainable text generation evaluation with automatic feedback. In EMNLP. find
  50. Changtong Zan, Keqin Peng, Liang Ding, Baopu Qiu, et al. 2022. Vega-MT: The JD explore academy machine translation system for WMT22. In WMT. find
  51. Chrysoula Zerva, Frédéric Blain, Ricardo Rei, et al. 2022. Findings of the WMT 2022 shared task on quality estimation. In WMT. find our card
  52. Yuqi Zhang, Liang Ding, Lefei Zhang, and Dacheng Tao. 2024. Intention analysis prompting makes large language models a good jailbreak defender. arXiv preprint. find
  53. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint. find our card
  54. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint. find
  55. Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert. arXiv preprint. ↗
  56. Qihuang Zhong, Kang Wang, Ziyang Xu, Juhua Liu, Liang Ding, Bo Du, and Dacheng Tao. 2024. Achieving> 97% on gsm8k: Deeply understanding the problems makes llms perfect reasoners. arXiv preprint. find

↑ back to top