📄 Never Truly Out of Fashion: A Retrospective Look at Evaluation in NLG

Summary

Automatic metrics and LLM judges are cheap, which tempts researchers to drop human evaluation of generated text. The authors ask how NLG evaluation has actually changed over time. They compiled 85,792 ACL Anthology papers from ACL-OCL plus their own post-2022 crawl of nine core venues. Filtering left 24,291 generation-focused papers spanning 1952–2025. They then used precision-oriented regular expressions to detect evaluation types, criteria from the Howcroft et al. (2020) taxonomy, and annotation methods. An LLM-based extraction attempt had poor recall and was set aside.

The results show that papers with no evaluation dominated before 1990, while evaluation is now nearly universal. Human evaluation has stayed flat at roughly 35–45% of papers over the last decade. LLM-as-a-judge has spread quickly since 2023, mostly alongside human evaluation, but often without it. Faithfulness has returned as a criterion, driven by hallucination research, while grammaticality and adequacy are fading. The study gives the field a measurable baseline for future comparison. It also argues for calibrating LLM judges and for standardised faithfulness annotation.

Key points

• Human evaluation is stable: 34–42% of NLG papers per year since 2018, with no significant trend over 2015–2025 (logistic regression β = 0.0023, p = 0.69). INLG reaches 53.1% compared with 36.0% at core NLP venues.
• LLM judges are growing fast: 19 papers in 2023, 109 in 2024 and 224 in 2025. Of these, 61.4% also report human evaluation, while 38.6% rely on the LLM alone.
• Faithfulness is the fastest-rising criterion: its share of human-evaluation papers went from 1.7% in 2015 to 29.8% in 2025. Among the 8,894 human-evaluation papers, the most frequent criteria are fluency (22.5%), relevance (21.6%) and coherence (16.6%).
• Methods and modalities:
  • Automatic-only evaluation is the most common modality at 57.8%, and human-only evaluation has dropped to 2–3%.
  • Likert scales are the leading annotation method (19.0%), followed by post-editing (15.8%).
  • MT has the lowest human evaluation rate, at about 28%.
• Validation and data release: the regex approach achieved 82.4% recall on 51 manually annotated papers. Code and the extracted signals (no full texts) are available at github.com/patuchen/trends_in_nlg_eval.