Elamur Research⌕
☰
home / papers / 2026 / NLG Evaluation: Past, Present, Future

NLG Evaluation: Past, Present, Future

Paper · 2026 · Ehud Reiter

Summary

In this position paper, Ehud Reiter reviews how Natural Language Generation (NLG) systems have been evaluated over about 35 years and predicts how evaluation will look by 2036. He presents no new data or experiments. Instead, he looks at INLG conference proceedings at ten-year intervals (1990, 2000, 2010, 2020) and compares them with the LLM-dominated landscape of 2026. Over that period, papers first justified their contributions with qualitative linguistic or engineering arguments. They then tried many evaluation styles, settled on standardised shared tasks, and eventually treated evaluation as a research topic in its own right. Large language models created new problems. Their outputs can equal or surpass human references, fluency no longer tells evaluators much, test sets may be memorised, and even rare harmful outputs are unacceptable in fields like medicine. The community has responded with LLM-as-Judge, expert error annotation and methods borrowed from other disciplines. Reiter argues that future work must look beyond test-set scores to real-world impact, qualitative insight and worst-case safety analysis. This is essential once NLG is used by large numbers of people.

Key points

  • Conference snapshots:
    • INLG 1990 had 25 papers, and none contained a structured quantitative hypothesis test.
    • By 2000 (38 papers), human ratings, metrics and task-based studies coexisted without any standard approach.
    • In 2010 (37 papers), shared tasks such as GREC were prominent.
    • In 2020 (46 papers), both best papers dealt with evaluation (Belz et al.; Dušek and Kasner).
  • NLP vs. NLG around 2010: wider NLP converged on BLEU for machine translation and ROUGE for summarisation. The NLG community kept relying on human judgments, typically Likert scales or ranking.
  • Shifts by 2026:
    • LLM-as-Judge (often adopted without validation for the specific use case).
    • Expert annotation of specific errors instead of crowdworker ratings, since crowdworkers may secretly use LLMs.
    • Unpublished test sets to avoid contamination.
    • Safety practices drawn from cyber security, such as red teaming.
    • Randomised controlled trials from medicine.
  • Unresolved problems:
    • Weak experimental rigour and poor replicability.
    • Doubtful construct validity of benchmarks.
    • Reward hacking, and models that still score well on benchmarks when the input data has been removed (Asadi et al., 2026).
    • Commercial incentives and quickly saturating benchmarks.
    • A publish-fast research culture.
  • Impact example (Bean et al., 2026): LLMs handle health-query scenarios well when given them directly or when talking with simulated users, but perform much worse with real people. Proposed impact methods include RCTs, A/B tests, pre-post studies and observational studies.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AberdeenAberdeen · United Kingdom20 works · 1,385 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksText generation · 202 works

Cite

Cite as Ehud Reiter. 2026. NLG Evaluation: Past, Present, Future. Proceedings of the 1st Symposium on Natural Language Generation Evaluations.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{reiter2026nlg,
  title = {{NLG Evaluation: Past, Present, Future}},
  author = {Ehud Reiter},
  booktitle = {Proceedings of the 1st Symposium on Natural Language Generation Evaluations},
  year = {2026},
  url = {https://aclanthology.org/2026.retroeval-main.2/}
}

References

47 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mubashara Akhtar, Anka Reuel, Prajna Soni, Sanchit Ahuja, Pawan Sasanka Ammanamanchi, Ruchit Rawal, Vilém Zouhar, Srishti Yadav, Chenxi Whitehouse, Dayeon Ki, Jennifer Mickel, Leshem Choshen, Marek Šuppa, Jan Batzner, Jenny Chim, Jeba Sania, Yanan Long, Hossein A. Rahmani, Christina Knight, and 18 o… — [arXiv:2602.16763](https://arxiv.org/abs/2602.16763) find
  2. Sydney Von Arx, Lawrence Chan, and Elizabeth Barnes. 2025. Recent frontier models are reward hacking. https://metr.org/blog/ 2025-06-05-recent-reward-hacking/. ↗
  3. Mohammad Asadi, Jack W. O’Sullivan, Fang Cao, Tahoura Nedaee, Kamyar Rajabalifardi, Fei-Fei Li, Ehsan Adeli, and Euan Ashley. 2026. Mirage: The illusion of visual understanding. Preprint, arXiv:2603.21687. — [arXiv:2603.21687](https://arxiv.org/abs/2603.21687) find
  4. Michael W. Asher, Gillian Gold, Eason Chen, and Paulo F. Carvalho. 2026. Chatbots are undermining crowdsourced research in the behavioral sciences: Detecting artificial intelligence–assisted cheating with a keystroke-based tool. Advances in Methods and Practices in Psychological Science, 9(1):251524… find
  5. Nadeem Badshah. 2026. Teenager died after asking chatgpt for ‘most successful’ way to take his life, inquest told. The Guardian. find
  6. Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondřej Dušek. 2024. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source llms. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Pape…
  7. See https://www.thehumanlineproject.org/ stories, such as Badshah (2026) 2 https://www.gov.uk/government/publications/ generative-ai-product-safety-standards Srinivas Bangalore, Owen Rambow, and Steve Whittaker. 2000. Evaluation metrics for generation. In INLG’2000 Proceedings of the First Internati… ↗
  8. Wei-Hung Weng, Hannah M. Ferrera, David Barrett, Lindsey Crowley, Jihyeon Lee, Spencer E. Rittner, Ellery Wulczyn, Selena K. Zhang, and 29 others. 2026. A prospective clinical feasibility study of a conversational diagnostic ai in an ambulatory primary care clinic. Preprint, arXiv:2603.08448. — [arXiv:2603.08448](https://arxiv.org/abs/2603.08448) find
  9. Andrew M Bean, Ryan Othniel Kearns, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, Jan Batzner, Negar Foroutan, Chris Schmitz, Karolina Korgul, Hunar Batra, and 1 others. 2025. Measuring what matters: Construct validity in large language model benchmarks. arXiv preprint arXiv:2511.04703; pre… — [arXiv:2511.04703](https://arxiv.org/abs/2511.04703) find
  10. Peter F. Brown, John Cocke, Stephen A. Della Pietra, Vincent J. Della Pietra, Fredrick Jelinek, John D. Lafferty, Robert L. Mercer, and Paul S. Roossin. 1990. A statistical approach to machine translation. Computational Linguistics, 16(2):79–85.
  11. Giuseppe Carenini. 2000. A task-based framework to evaluate evaluative arguments. In INLG’2000 Proceedings of the First International Conference on Natural Language Generation, pages 9–16, Mitzpe Ramon, Israel. Association for Computational Linguistics. find
  12. Andrew M Bean, Rebecca Elizabeth Payne, Guy Parsons, Hannah Rose Kirk, Juan Ciro, Rafael Mosquera-Gómez, Sara Hincapié M, Aruna S Ekanayaka, Lionel Tarassenko, Luc Rocher, and 1 others. 2026. Reliability of llms as medical assistants for the general public: a randomized preregistered study. Nature M… find
  13. Hua Cheng and Chris Mellish. 2000. An empirical analysis of constructing non-restrictive NP modifiers to express semantic relations. In INLG’2000 Proceedings of the First International Conference on Natural Language Generation, pages 108–115, Mitzpe Ramon, Israel. Association for Computational Lingu… find
  14. Anja Belz and Eric Kow. 2010. The GREC challenges 2010: Overview and evaluation results. In Proceedings of the 6th International Natural Language Generation Conference. Association for Computational Linguistics. find
  15. Zerui Cheng, Stella Wohnig, Ruchika Gupta, Samiul Alam, Tassallah Abdullahi, João Alves Ribeiro, Christian Nielsen-Garcia, Saif Mir, Siran Li, Jason Orender, and 1 others. 2025. Benchmarking is broken–don’t let ai be its own judge. arXiv preprint arXiv:2510.07575, presented at Neurips2025. — [arXiv:2510.07575](https://arxiv.org/abs/2510.07575) find
  16. Anya Belz, Simon Mille, and David M. Howcroft. 2020. Disentangling the properties of human evaluation methods: A classification system to support comparability, meta-evaluation and reproducibility testing. In Proceedings of the 13th International Conference on Natural Language Generation, pages 183–… find
  17. Matthew J. Duggan, Julietta Gervase, Anna Schoenbaum, William Hanson, III Howell, John T., Michael Sheinberg, and Kevin B. Johnson. 2025. Clinician experiences with ambient scribe technology to assist with documentation burden and efficiency. JAMA Network Open, 8(2):e2460637–e2460637. find
  18. Anya Belz, Craig Thomson, Ehud Reiter, and Simon Mille. 2023. Non-repeatable experiments and non-reproducible results: The reproducibility crisis in human evaluation in NLP. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3676–3687, Toronto, Canada. Association for Comp… find
  19. Ondřej Dušek and Zdeněk Kasner. 2020. Evaluating semantic accuracy of data-to-text generation with natural language inference. In Proceedings of the 13th International Conference on Natural Language Generation, pages 131–137, Dublin, Ireland. Association for Computational Linguistics. find
  20. Yoshua Bengio, Stephen Clare, Carina Prunkl, Maksym Andriushchenko, Ben Bucknall, Malcolm Murray, Rishi Bommasani, Stephen Casper, Tom Davidson, Raymond Douglas, David Duvenaud, Philip Fox, Usman Gohar, Rose Hadshar, Anson Ho, Tiancheng Hu, Cameron Jones, Sayash Kapoor, Atoosa Kasirzadeh, and 73 oth… find
  21. Mingqi Gao, Xinyu Hu, Xunjian Yin, Jie Ruan, Xiao Pu, and Xiaojun Wan. 2025. LLM-based NLG evaluation: Current status and challenges. Computational Linguistics, 51:661–687.
  22. Daniel Gildea and Daniel Jurafsky. 2000. Automatic labeling of semantic roles. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pages 512–520, Hong Kong. Association for Computational Linguistics.
  23. Timothy W Bickmore, Ha Trinh, Stefan Olafsson, Teresa K O’Leary, Reza Asadi, Nathaniel M Rickles, and Ricardo Cruz. 2018. Patient and consumer safety risks when using conversational assistants for medical information: an observational study of siri, alexa, and google assistant. Journal of medical In… find
  24. T Greenhaigh and Rod Taylor. 1997. Papers that go beyond numbers (qualitative research)’. British Medical Journal, 315(7110):740–743. find
  25. Peter Brodeur, Jacob M. Koshy, Anil Palepu, Khaled Saab, Ava Homiar, Roma Ruparel, Charles Wu, Ryutaro Tanno, Joseph Xu, Amy Wang, David Stutz, Greg Guest, Kathleen M MacQueen, and Emily E Namey. 2011. Applied thematic analysis. sage publications. find
  26. David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In Proceedings… find
  27. Ehud Reiter, Roma Robertson, A. Scott Lennox, and Liesl Osman. 2001. Using a randomised controlled clinical trial to evaluate an NLG system. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics, pages 442–449, Toulouse, France. Association for Computational Ling… find
  28. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902– 4912, Online. Association for Computational Linguisti… find
  29. Inderjeet Mani, David House, Gary Klein, Lynette Hirschman, Therese Firmin, and Beth Sundheim. 1999. The TIPSTER SUMMAC text summarization evaluation. In Ninth Conference of the European Chapter of the Association for Computational Linguistics, pages 77–85, Bergen, Norway. Association for Computatio… find
  30. Rahul Sambaraju, Ehud Reiter, Robert Logie, Andy Mckinlay, Chris McVittie, Albert Gatt, and Cindy Sykes. 2011. What is in a text and what does it do: Qualitative evaluations of an NLG system – the BT-nurse – using content analysis and discourse analysis. In Proceedings of the 13th European Workshop… find
  31. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984–4997, Online. Association for Computation…
  32. Mengxuan Sun, Ehud Reiter, Peter Murchie, Anne E Kiltie, George Ramsay, Lisa Duncan, and Rosalind Adam. 2026. Can chatgpt give holistic and accurate patient-centred information to oncology patients? a mixed-methods evaluation with stakeholders. medRxiv. find
  33. Kathleen F. McCoy, K. Vijay-Shanker, and Gijoo Yang. 1990. Using Tree Adjoining Grammars systemic framework in the. In Proceedings of the Fifth International Workshop on Natural Language Generation, Linden Hall Conference Center, Dawson, Pennsylvania. Association for Computational Linguistics. find
  34. Craig Thomson and Ehud Reiter. 2020. A gold standard methodology for evaluating accuracy in data-to-text systems. In Proceedings of the 13th International Conference on Natural Language Generation, pages 158–168, Dublin, Ireland. Association for Computational Linguistics. find
  35. Guido Minnen, John Carroll, and Darren Pearce. 2000. Robust, applied morphological generation. In INLG’2000 Proceedings of the First International Conference on Natural Language Generation, pages 201–208, Mitzpe Ramon, Israel. Association for Computational Linguistics.
  36. Craig Thomson, Ehud Reiter, and Anya Belz. 2024. Common flaws in running human evaluation experiments in NLP. Computational Linguistics, 50(2):795–805. find
  37. Gabriel Murray, Giuseppe Carenini, and Raymond Ng. 2010. Generating and validating abstracts of meeting conversations: a user study. In Proceedings of the 6th International Natural Language Generation Conference. Association for Computational Linguistics. find
  38. Craig Thomson, Ehud Reiter, and Barkavi Sundararajan. 2023. Evaluating factual accuracy in complex data-to-text. Computer Speech and Language, 80:101482. find
  39. E.J. Tisdell, S.B. Merriam, and H.L. Stuckey-Peyrot. 2025. Qualitative Research: A Guide to Design and Implementation. Wiley. find
  40. Sandeep Reddy, Wendy Rogers, Ville-Petteri Makinen, Enrico Coiera, Pieta Brown, Markus Wenzel, Eva Weicken, Saba Ansari, Piyush Mathur, Aaron Casey, and 1 others. 2021. Evaluation framework to guide implementation of ai systems into healthcare settings. BMJ health & care informatics, 28(1):e100444. find
  41. Chris van der Lee, Albert Gatt, Emiel van Miltenburg, and Emiel Krahmer. 2021. Human evaluation of automatically generated text: Current trends and best practice guidelines. Computer Speech and Language, 67:101151. find
  42. Ehud Reiter. 2025a. Natural Language Generation. Springer. find
  43. Emiel van Miltenburg, Miruna Clinciu, Ondřej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Stephanie Schoch, Craig Thomson, and Luou Wen. 2023. Barriers and enabling factors for error analysis in NLG research. Northern European Journal of Language Technology, 9. find
  44. Ehud Reiter. 2025b. We should evaluate real-world impact. Computational Linguistics, 51(4):1419–1431. find
  45. Ehud Reiter and Anja Belz. 2009. An investigation into the validity of some metrics for automatically evaluating natural language generation systems. Computational Linguistics, 35(4):529–558. find
  46. Alexander Waibel and Kai-Fu Lee. 1990. Readings in speech recognition. Morgan Kaufmann. find
  47. Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, and Alexandra Olteanu. 2022. Deconstructing NLG evaluation: Evaluation practices, assumptions, and their implications. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computationa… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/reiter-2026-nlg-evaluation-past-present-future/