Elamur Research⌕
☰
home / papers / 2026 / Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation

Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation

Paper · 2026 · Katelyn X. Mei, Yi-Li Hsu, Minjoon Choi et al.

Summary

Human judgment is still treated as the most trustworthy way to assess long-form or open-ended text produced by LLMs. Yet whether such studies can be interpreted or repeated depends on how thoroughly authors describe their protocols. The authors audit this documentation across *CL venues from 2023 to 2025. Drawing on reproducibility research, they build a codebook of 37 questions, 20 of which are binary core criteria. These are grouped into task documentation, annotation design, and analysis & interpretation.

From more than 9.1k papers, they identify over 1,800 that evaluate long-form generation with human raters. They code 284 of these fully by hand and process the rest with LLM-assisted annotation.

Most papers name the evaluated dimensions, the sample size, and some annotator details. Many other elements are frequently missing:

  • task guidelines
  • justification for the chosen dimensions
  • power analysis
  • statistical confidence

The share of papers relying on human evaluation is also shrinking, while LLM-judges are spreading. This leaves real uncertainty about what was measured and by whom. The paper ends with practical reporting recommendations for the community.

Key points

  • Corpus: 9.1k+ *CL papers (2023–2025). Of these, 1,800+ study long-form generation with human evaluation; 356 were sampled and 284 manually annotated. Roughly half of 2025 *CL papers involve long-form generation.
  • Criteria: 20 core criteria, split into task documentation (4), annotation design (9), and analysis & interpretation (7). The codebook was refined over four rounds, each applied to five papers. A typical study is expected to satisfy about 15–16 criteria.
  • Under-reporting: Only about half of papers provide annotator guidelines or a rationale for the evaluated dimensions. Power analysis and confidence reporting are very uncommon.
  • Trend: Use of human evaluation is falling, while use of LLM-judges is rising. Human judgments increasingly serve to meta-evaluate those judges.
  • Resources and validation: LLM annotations served only as supplementary evidence and had to reach a validation accuracy threshold of 0.75. Code and the annotated dataset are available at github.com/larchlab/Illusions-of-the-Gold-Standard.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States353 works · 200,043 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States275 works · 97,634 citations🎓Seoul National UniversitySeoul · South Korea41 works · 45,671 citations🔬Mila - Quebec Artificial Intelligence InstituteMontreal · Canada54 works · 13,565 citations🎓National Tsing Hua UniversityHsinchu · Taiwan8 works · 834 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksText generation · 202 works

Cite

Cite as Katelyn X. Mei, Yi-Li Hsu, Minjoon Choi, Zongwan Cao, Chenjun Xu, Bingbing Wen, Su Lin Blodgett and Lucy Lu Wang. 2026. Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.635

@inproceedings{mei2026illusions,
  title = {{Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation}},
  author = {Katelyn X. Mei, Yi-Li Hsu, Minjoon Choi, Zongwan Cao, Chenjun Xu, Bingbing Wen, Su Lin Blodgett and Lucy Lu Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.635},
  url = {https://doi.org/10.18653/v1/2026.acl-long.635}
}

References

21 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. evaluation framework: Rethinking human evaluation for generative large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1137–1160. find
  2. Zhiwei Fei, Songyang Zhang, Xiaoyu Shen, Dawei Zhu, Xiao Wang, Jidong Ge, and Vincent Ng. 2025. Internlm-law: An open-sourced chinese legal large language model. In Proceedings of the 31st International Conference on Computational Linguistics, pages 9376–9392. find
  3. Jennifer Fereday and Eimear Muir-Cochrane. 2006. Demonstrating rigor using thematic analysis: A hybrid approach of inductive and deductive coding and theme development. International journal of qualitative methods, 5(1):80–92. find
  4. Eve Fleisig, Su Lin Blodgett, Dan Klein, and Zeerak Talat. 2024. The perspectivist paradigm shift: Assumptions and challenges of capturing human labels. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (… find
  5. David Fraile Navarro, Enrico Coiera, Thomas W Hambly, Zoe Triplett, Nahyan Asif, Anindya Susanto, Anamika Chowdhury, Amaya Azcoaga Lorenzo, Mark Dras, and Shlomo Berkovsky. 2025. Expert evaluation of large language models for clinical dialogue summarization. Scientific reports, 15(1):1195. find
  6. David M Howcroft, Anja Belz, Miruna Clinciu, Dimitra Gkatzia, Sadid A Hasan, Saad Mahamood, Simon Mille, Emiel Van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: Nlg needs evaluation sheets and standardised definitions. In Proceedings of the 13… find
  7. Maximilian Idahl and Zahra Ahmadi. 2025. OpenReviewer: A specialized large language model for generating critical scientific paper reviews. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Syst… find
  8. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594. find
  9. Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. 2024. Fine-grained hallucination detection and editing for language models. In First Conference on Language Modeling.
  10. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pages 220–229. find
  11. Marcus R Munafò, Brian A Nosek, Dorothy VM Bishop, Katherine S Button, Christopher D Chambers, Nathalie Percie du Sert, Uri Simonsohn, Eric-Jan Wagenmakers, Jennifer J Ware, and John PA Ioannidis. 2017. A manifesto for reproducible science. Nature human behaviour, 1(1):0021. find
  12. Eric A Posner and Shivam Saran. 2025. Judge ai: Assessing large language models in judicial decision-making. University of Chicago Coase-Sandor Institute for Law & Economics Research Paper, (2503). find
  13. Juan-Pablo Rivera, Gabriel Mukobi, Anka Reuel, Max Lamparth, Chandler Smith, and Jacquelyn Schneider. 2024. Escalation risks from language models in military and diplomatic decision-making. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 836–898. find
  14. Anna Rogers, Timothy Baldwin, and Kobi Leins. 2021. ‘just what do you think you’re doing, dave?’ a checklist for responsible data use in NLP. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4821–4833, Punta Cana, Dominican Republic. Association for Computational Lingu… find
  15. Shaurya Rohatgi. 2022. Acl anthology corpus with full text. Github. find
  16. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 1668–1678.
  17. Anastasia Shimorina and Anja Belz. 2022. The human evaluation datasheet: A template for recording details of human evaluation experiments in nlp. In Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval), pages 54–75. find
  18. Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. 2025. Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM 2 ), pages 404–430. find
  19. Craig Thomson, Ehud Reiter, and Anya Belz. 2024. Common flaws in running human evaluation experiments in nlp. Computational Linguistics, 50(2):795– 805. find
  20. Lucy Lu Wang, Yulia Otmakhova, Jay DeYoung, Thinh Hung Truong, Bailey Kuehl, Erin Bransom, and Byron C. Wallace. 2023. Automated metrics for medical multi-document summarization disagree with human evaluations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistic… find
  21. Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. 2023. A critical evaluation of evaluations for long-form question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3225–3245, Toronto, Canada. Association for C…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mei-2026-illusions-gold-standard-large-scale/