Elamur Research⌕
☰
home / papers / 2026 / When the Gold Standard Isn't Necessarily Standard: Challenges of Evaluating the Translation of User-Generated Content

When the Gold Standard Isn't Necessarily Standard: Challenges of Evaluating the Translation of User-Generated Content

Summary

Machine translation of user-generated content (UGC) is hard to evaluate. Social media and forum posts are full of non-standard language, such as typos, slang, letter repetitions and emojis, and nobody agrees on how much of it should survive in the target text. The authors examine the human translation instructions behind four UGC parallel corpora: RoCS-MT, FooTweets, MMTC and PFSMB. These cover English–French, English–German and French–English. Drawing on these instructions, they classify twelve kinds of non-standard phenomena and define five possible ways of handling them in translation (NORMALISE, COPY, TRANSFER, OMIT, CENSOR). The corpora turn out to sit at different points on a standardness scale, from heavily normalising to preservation-oriented.

They then prompt several LLMs in three ways: with no guidelines, with each dataset's own guidelines, and with guidelines borrowed from another dataset. Metric scores react strongly to these instructions and improve when the prompt matches the guidelines behind the references. Reference-based metrics reward agreement with the reference style, while COMET-Kiwi prefers standardised output. Without documented guidelines that both systems and metrics take into account, UGC benchmark comparisons can be misleading.

Key points

  • Datasets and pairs: RoCS-MT is English–French and translates lexically normalised source text. FooTweets is English–German, built from 2014 FIFA World Cup tweets with sentiment analysis in mind. MMTC and PFSMB are French–English. MTNT and Foursquare were excluded because they document no translator guidelines.
  • Shared and divergent guidelines: All four corpora correct grammar and spelling and copy emojis. They differ on other phenomena. For example, RoCS-MT normalises word elongation and atypical punctuation, whereas the other three transfer elongation and copy unusual punctuation.
  • Models tested: The baseline was NLLB-3B, which cannot be prompted with guidelines. The instruction-tuned LLMs were Gemma-2-9B, Granite-4.1-8B, Mistral-0.3-7B, Qwen-2.5-7B, Tower-0.2-7B and LLaMA-3.1-8B.
    • Most LLMs adapted to guideline prompts.
    • Tower barely reacted to them.
    • LLaMA often refused "unsafe" inputs, which lowered its corpus-level scores.
  • Metrics: BLEU and COMET (reference-based) penalise style mismatches with the references, for instance when minimally standardising prompts are scored against RoCS-MT. COMET-Kiwi (reference-less) gave lower scores to highly non-standard but guideline-faithful outputs on PFSMB.
  • Recommendations:
    • If style does not matter, use reference-less metrics or multiple references at different standardness levels.
    • If a specific style is required, prompt the model with explicit guidelines and evaluate with a reference-based method, ideally an LLM-as-a-judge given the same guidelines.
    • Future human evaluation should use phenomenon-specific rubrics.
    • Code and guidelines are released on GitHub (lydianish-phd/ugc-mt-eval-challenges).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{nishimwe2026when,
  title = {{When the Gold Standard Isn't Necessarily Standard: Challenges of Evaluating the Translation of User-Generated Content}},
  author = {Nishimwe, Lydia and Sagot, Benoît and Bawden, Rachel},
  booktitle = {Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 1)},
  year = {2026},
  url = {https://aclanthology.org/2026.eamt-1.30/}
}

References

83 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aepli, Noëmi, Chantal Amrhein, Florian Schottmann, and Rico Sennrich. 2023. A Benchmark for Evaluating Machine Translation Metrics on Dialects without Standard Orthography. In Proceedings of the Eighth Conference on Machine Translation, pages 1045–1065, Singapore. Association for Computational Linguistics. find
  2. Agrawal, Sweta, António Farinhas, Ricardo Rei, and Andre Martins. 2024. Can Automatic Metrics Assess High-Quality Translations? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14491–14502, Miami, Florida, USA. Association for Computational Linguistics. find
  3. Alam, Md Mahfuz Ibn and Antonios Anastasopoulos. 2025. Large Language Models as a Normalizer for Transliteration and Dialectal Translation. In Proceedings of the 12th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 39– 67, Abu Dhabi, UAE. Association for Computational Linguistics. find
  4. Alves, Duarte Miguel, José Pombal, Nuno M Guerreiro, Pedro Henrique Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and Andre Martins. 2024. Tower: An Open Multilingual Large Language Model for Translation-Related Tasks. In First Conference on Language Modeling, Philadelphia, Pennsylvania, USA. ↗ our card
  5. Baldwin, Tyler and Yunyao Li. 2015. An In-depth Analysis of the Effect of Text Normalization in Social Media. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 420–429, Denver, Colorado. Association for Computational Linguistics. find
  6. Bawden, Rachel and Benoı̂t Sagot. 2023. RoCS-MT: Robustness Challenge Set for Machine Translation. In Proceedings of the Eighth Conference on Machine Translation, pages 198–216, Singapore. Association for Computational Linguistics. ↗
  7. Berard, Alexandre, Ioan Calapodescu, Marc Dymetman, Claude Roux, Jean-Luc Meunier, and Vassilina Nikoulina. 2019. Machine Translation of Restaurant Reviews: New Corpus for Domain Adaptation and Robustness. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 168–176, Hong Kong. Association for Computational Linguistics. find
  8. Bolding, Quinten, Baohao Liao, Brandon Denis, Jun Luo, and Christof Monz. 2023. Ask Language Model to Clean Your Noisy Translation Data. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3215–3236, Singapore. Association for Computational Linguistics. find
  9. Briakou, Eleftheria, Zhongtao Liu, Colin Cherry, and Markus Freitag. 2024. On the Implications of Verbose LLM Outputs: A Case Study in Translation Evaluation. CoRR, abs/2410.00863. ↗
  10. Callison-Burch, Chris, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (Meta-) Evaluation of Machine Translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 136–158, Prague, Czech Republic. Association for Computational Linguistics. find our card
  11. Callison-Burch, Chris, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further Meta-Evaluation of Machine Translation. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 70–106, Columbus, Ohio. Association for Computational Linguistics. find our card
  12. Ceballos-Arroyo, Alberto Mario, Monica Munnangi, Jiuding Sun, Karen Zhang, Jered McInerney, Byron C. Wallace, and Silvio Amir. 2024. Open (Clinical) LLMs are Sensitive to Instruction Phrasings. In Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, pages 50–71, Bangkok, Thailand. Association for Computational Linguistics. ↗
  13. Chatterjee, Anwoy, H S V N S Kowndinya Renduchintala, Sumit Bhatia, and Tanmoy Chakraborty. 2024. POSIX: A Prompt Sensitivity Index For Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14550–14565, Miami, Florida, USA. Association for Computational Linguistics. find
  14. Cohen, Jacob. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. ↗
  15. Coyne, Steven, Keisuke Sakaguchi, Diana Galvan-Sosa, Michael Zock, and Kentaro Inui. 2023. Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction. CoRR, abs/2303.14342. ↗
  16. Eisenstein, Jacob. 2013. What to do about bad language on the internet. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 359–369, Atlanta, Georgia. Association for Computational Linguistics. find
  17. Fang, Tao, Shu Yang, Kaixin Lan, Derek F. Wong, Jinpeng Hu, Lidia S. Chao, and Yue Zhang. 2023. Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluation. CoRR, abs/2304.01746. ↗
  18. Fleisig, Eve, Genevieve Smith, Madeline Bossi, Ishita Rustagi, Xavier Yin, and Dan Klein. 2024. Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discrimination. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13541–13564, Miami, Florida, USA. Association for Computational Linguistics. find
  19. Foster, Jennifer. 2010. “cba to check the spelling”: Investigating Parser Performance on Discussion Forum Posts. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 381–384, Los Angeles, California. Association for Computational Linguistics. find our card
  20. Freitag, Markus, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. ↗ our card
  21. Freitag, Markus, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, David Ifeoluwa Adelani, Marianna Buchicchio, Chrysoula Zerva, and Alon Lavie. 2024. Are LLMs Breaking MT Metrics? Results of the WMT24 Metrics Shared Task. In Proceedings of the Ninth Conference on Machine Translation, pages 47–81, Miami, Florida, USA. Association for Computational Linguistics. find our card
  22. Gao, Yuan, Ruili Wang, and Feng Hou. 2024. How to Design Translation Prompts for ChatGPT: An Empirical Study. In Proceedings of the 6th ACM International Conference on Multimedia in Asia Workshops, MMAsia ’24 Workshops, New York, NY, USA. Association for Computing Machinery. find
  23. Garcia, Xavier, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Melvin Johnson, and Orhan Firat. 2023. The unreasonable effectiveness of few-shot learning for machine translation. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. ↗
  24. Gemma Team. 2024. Gemma 2: Improving Open Language Models at a Practical Size. CoRR, abs/2408.00118. ↗
  25. Hendy, Amr, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation. CoRR, abs/2302.09210. ↗
  26. IBM Research. 2026. Granite 4.1 Language Models. https://huggingface.co/blog/ibm-granite/granite-4-1. ↗
  27. Kalamkar, Dhiraj D., Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. 2019. A Study of BFLOAT16 for Deep Learning Training. CoRR, abs/1905.12322. ↗
  28. Katinskaia, Anisia and Roman Yangarber. GPT-3.5 for Grammatical Error Correction. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 7831–7843, Torino, Italia. ELRA and ICCL. find
  29. Koehn, Philipp and Christof Monz. 2006. Manual and Automatic Evaluation of Machine Translation between European Languages. In Proceedings on the Workshop on Statistical Machine Translation, pages 102–121, New York City. Association for Computational Linguistics. find our card
  30. Koehn, Philipp. 2004. Statistical Significance Tests for Machine Translation Evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388– 395, Barcelona, Spain. Association for Computational Linguistics. find
  31. Krippendorff, Klaus. 1970. Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement, 30(1):61–70. ↗
  32. Kwon, Sang Yun, Gagan Bhatia, El Moatez Billah Nagoud, and Muhammad Abdul-Mageed. 2023a. ChatGPT for Arabic Grammatical Error Correction. CoRR, abs/2308.04492. Kwon, Woosuk, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. ↗
  33. Gonzalez, Hao Zhang, and Ion Stoica. 2023b. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles. find
  34. Lavie, Alon, Greg Hanneman, Sweta Agrawal, Diptesh Kanojia, Chi-Kiu Lo, Vilém Zouhar, Frederic Blain, Chrysoula Zerva, Eleftherios Avramidis, Sourabh Deoghare, Archchana Sindhujan, Jiayi Wang, David Ifeoluwa Adelani, Brian Thompson, Tom Kocmi, Markus Freitag, and Daniel Deutsch. 2025. Findings of the WMT25 Shared Task on Automated Translation Evaluation Systems: Linguistic Diversity is Challenging and References Still Help. In Proceedings of the Tenth Conference on Machine Translation, pages 436–483, Suzhou, China. Association for Computational Linguistics. find
  35. Li, Kunting, Yong Hu, Liang He, Fandong Meng, and Jie Zhou. 2024. C-LLM: Learn to Check Chinese Spelling Errors Character by Character. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5944– 5957, Miami, Florida, USA. Association for Computational Linguistics. find
  36. Liu, Pusheng, Lianwei Wu, Linyong Wang, Sensen Guo, and Yang Liu. 2024. Step-by-Step: Controlling Arbitrary Style in Text with Large Language Models. In Proceedings of the 2024 Joint International Conference on Computational Linguis-2024. tics, Language Resources and Evaluation (LREC-COLING 2024), pages 15285–15295, Torino, Italia. ELRA and ICCL. find
  37. Llama Team. 2024. The Llama 3 Herd of Models. CoRR, abs/2407.21783. ↗
  38. Lommel, Arle, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional Quality Metrics (MQM): a Framework for Declaring and Describing Translation Quality Metrics. Tradumàtica, 12:455– 463. ↗
  39. Lyu, Chenyang, Zefeng Du, Jitao Xu, Yitao Duan, Minghao Wu, Teresa Lynn, Alham Fikri Aji, Derek F. Wong, and Longyue Wang. 2024. A Paradigm Shift: The Future of Machine Translation Lies with Large Language Models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 1339– 1352, Torino, Italia. ELRA and ICCL. find
  40. Maeng, Junghwan, Jinghang Gu, and Sun-A Kim. 2023. Effectiveness of ChatGPT in Korean Grammatical Error Correction. In Proceedings of the 37th Pacific Asia Conference on Language, Information and Computation, pages 464–472, Hong Kong, China. Association for Computational Linguistics. find
  41. Mathur, Nitika, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020. Results of the WMT20 Metrics Shared Task. In Proceedings of the Fifth Conference on Machine Translation, pages 688–725, Online. Association for Computational Linguistics. find our card
  42. McNamee, Paul and Kevin Duh. 2022. The Multilingual Microblog Translation Corpus: Improving and Evaluating Translation of User-Generated Text. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 910–918, Marseille, France. European Language Resources Association. find
  43. Michel, Paul and Graham Neubig. 2018. MTNT: A Testbed for Machine Translation of Noisy Text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 543–553, Brussels, Belgium. Association for Computational Linguistics. find
  44. Mistral AI Team. 2024. Mistral-7B-Instruct-v0.3. https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3. ↗
  45. Moslem, Yasmin, Gianfranco Romani, Mahdi Molaei, John D. Kelleher, Rejwanul Haque, and Andy Way. 2023. Domain Terminology Integration into Machine Translation: Leveraging Large Language Models. In Proceedings of the Eighth Conference on Machine Translation, pages 902–911, Singapore. Association for Computational Linguistics. find
  46. Nida, Eugene A. (Eugene Albert). 1969. The theory and practice of translation. Leiden, E.J. Brill. find
  47. Niu, Xing and Marine Carpuat. 2020. Controlling Neural Machine Translation Formality with Synthetic Supervision. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8568–8575. find
  48. NLLB Team. 2022. No language left behind: Scaling human-centered machine translation. CoRR, abs/2207.04672. ↗
  49. OpenAI. 2023. GPT-4 Technical Report. abs/2303.08774. ↗ our card
  50. CoRR, Pan, Leiyu, Yongqi Leng, and Deyi Xiong. 2024. Can Large Language Models Learn Translation Robustness from Noisy-Source In-context Demonstrations? In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2798–2808, Torino, Italia. ELRA and ICCL. find
  51. Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311– 318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. find our card
  52. Penteado, Maria Carolina and Fábio Perez. 2023. Evaluating GPT-3.5 and GPT-4 on Grammatical Error Correction for Brazilian Portuguese. In LatinX in AI Workshop at ICML 2023 (Regular Deadline). ↗
  53. Peters, Ben and André F. T. Martins. 2024. Did Translation Models Get More Robust Without Anyone Even Noticing? CoRR, abs/2403.03923. ↗
  54. Popović, Maja, Ekaterina Lapshinova-Koltunski, and Maarit Koponen. 2024. Effects of different types of noise in user-generated reviews on human and machine translations including ChatGPT. In Proceedings of the Ninth Workshop on Noisy and User-generated Text (W-NUT 2024), pages 17–30, San Ġiljan, Malta. Association for Computational Linguistics. find
  55. Post, Matt. 2018. A Call for Clarity in Reporting BLEU Scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics. find our card
  56. Qian, Shenbin, Constantin Orasan, Diptesh Kanojia, and Félix Do Carmo. 2024. Are large language models state-of-the-art quality estimators for machine translation of user-generated content? In Proceedings of the Eleventh Workshop on Asian Translation (WAT 2024), pages 45–55, Miami, Florida, USA. Association for Computational Linguistics. find
  57. Qwen Team. 2024. Qwen2.5: A Party of Foundation Models. https://qwenlm.github.io/blog/qwen2.5. ↗
  58. Rei, Ricardo, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A Neural Framework for MT Evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  59. Rei, Ricardo, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022a. COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find our card
  60. Rei, Ricardo, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022b. CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. ↗ our card
  61. Riley, Parker, Timothy Dozat, Jan A. Botha, Xavier Garcia, Dan Garrette, Jason Riesa, Orhan Firat, and Noah Constant. 2023. FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation. Transactions of the Association for Computational Linguistics, 11:671–685. ↗ our card
  62. Rippeth, Elijah, Sweta Agrawal, and Marine Carpuat. 2022. Controlling Translation Formality Using Pre-trained Multilingual Language Models. In Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022), pages 327– 340, Dublin, Ireland (in-person and online). Association for Computational Linguistics. find
  63. Rosales Núñez, José Carlos, Djamé Seddah, and Guillaume Wisniewski. 2019. Comparison between NMT and PBSMT Performance for Translating Noisy User-Generated Content. In Proceedings of the 22nd Nordic Conference on Computational Linguistics, pages 2–14, Turku, Finland. Linköping University Electronic Press. find
  64. Rosales Núñez, José Carlos, Djamé Seddah, and Guillaume Wisniewski. 2021. Understanding the Impact of UGC Specificities on Translation Quality. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 189–198, Online. Association for Computational Linguistics. find
  65. Sánchez, Eduardo, Pierre Andrews, Pontus Stenetorp, Mikel Artetxe, and Marta R. Costa-jussà. 2024. Gender-specific Machine Translation with Large Language Models. In Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024), pages 148–158, Miami, Florida, USA. Association for Computational Linguistics. find
  66. Sanguinetti, Manuela, Cristina Bosco, Lauren Cassidy, Özlem Çetinoğlu, Alessandra Teresa Cignarella, Teresa Lynn, Ines Rehbein, Josef Ruppenhofer, Djamé Seddah, and Amir Zeldes. 2020. Treebanking User-Generated Content: A Proposal for a Unified Representation in Universal Dependencies. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 5240–5250, Marseille, France. European Language Resources Association. find
  67. Sclar, Melanie, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations. find
  68. Seddah, Djamé, Benoit Sagot, Marie Candito, Virginie Mouilleron, and Vanessa Combet. 2012. The French Social Media Bank: a Treebank of Noisy User Generated Content. In Proceedings of COLING 2012, pages 2441–2458, Mumbai, India. find
  69. The COLING 2012 Organizing Committee. find
  70. Sennrich, Rico, Barry Haddow, and Alexandra Birch. 2016. Controlling Politeness in Neural Machine Translation via Side Constraints. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 35–40, San Diego, California. Association for Computational Linguistics. find our card
  71. Sluyter-Gäthje, Henny, Pintu Lohar, Haithem Afli, and Andy Way. 2018. FooTweets: A Bilingual Parallel Corpus of World Cup Tweets. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  72. Specia, Lucia, Frédéric Blain, Marina Fomicheva, Erick Fonseca, Vishrav Chaudhary, Francisco Guzmán, and André F. T. Martins. 2020. Findings of the WMT 2020 Shared Task on Quality Estimation. In Proceedings of the Fifth Conference on Machine Translation, pages 743–764, Online. Association for Computational Linguistics. find our card
  73. Supryadi, Supryadi, Leiyu Pan, and Deyi Xiong. 2024. An Empirical Study on the Robustness of Massively Multilingual Neural Machine Translation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 1086–1097, Torino, Italia. ELRA and ICCL. find our card
  74. Touvron, Hugo, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. CoRR, abs/2307.09288. ↗
  75. Treviso, Marcos V, Nuno M Guerreiro, Sweta Agrawal, Ricardo Rei, José Pombal, Tania Vaz, Helena Wu, Beatriz Silva, Daan Van Stigt, and Andre Martins. 2024. xTower: A Multilingual LLM for Explaining and Correcting Translation Errors. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15222–15239, Miami, Florida, USA. Association for Computational Linguistics. ↗
  76. van der Goot, Rob, Rik van Noord, and Gertjan van Noord. 2018. A Taxonomy for In-depth Evaluation of Normalization for User Generated Content. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 684–688, Miyazaki, Japan. European Language Resources Association (ELRA). find
  77. Yamada, Masaru. 2023. Optimizing Machine Translation through Prompt Engineering: An Investigation into ChatGPT‘s Customizability. In Proceedings of Machine Translation Summit XIX, Vol. 2: Users Track, pages 195–204, Macau SAR, China. Asia-Pacific Association for Machine Translation. find
  78. Zebaze, Armel Randy, Benoı̂t Sagot, and Rachel Bawden. 2025a. Compositional Translation: A Novel LLM-based Approach for Low-resource Machine Translation. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 22328– 22357, Suzhou, China. Association for Computational Linguistics. find
  79. Zebaze, Armel Randy, Benoı̂t Sagot, and Rachel Bawden. 2025b. In-Context Example Selection via Similarity Search Improves Low-Resource Machine Translation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1222– 1252, Albuquerque, New Mexico. Association for Computational Linguistics. find
  80. Zerva, Chrysoula, Frederic Blain, José G. C. De Souza, Diptesh Kanojia, Sourabh Deoghare, Nuno M. Guerreiro, Giuseppe Attanasio, Ricardo Rei, Constantin Orasan, Matteo Negri, Marco Turchi, Rajen Chatterjee, Pushpak Bhattacharyya, Markus Freitag, and André Martins. 2024. Findings of the Quality Estimation Shared Task at WMT 2024: Are LLMs Closing the Gap in QE? In Proceedings of the Ninth Conference on Machine Translation, pages 82–109, Miami, Florida, USA. Association for Computational Linguistics. find
  81. Zhang, Xiaowu, Xiaotian Zhang, Cheng Yang, Hang Yan, and Xipeng Qiu. 2023. Does Correction Remain A Problem For Large Language Models? CoRR, abs/2308.01776. ↗
  82. Zheng, Lianmin, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and Chatbot Arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc. find
  83. Zouhar, Vilém, Pinzhen Chen, Tsz Kin Lam, Nikita Moghe, and Barry Haddow. 2024. Pitfalls and Outlooks in Using COMET. In Proceedings of the Ninth Conference on Machine Translation, pages 1272– 1288, Miami, Florida, USA. Association for Computational Linguistics. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/nishimwe-2026-when-gold-standard-isn-t/