Elamur Research⌕
☰
home / papers / 2026 / Process Standardisation for Human Evaluation of NLP System Outputs

Process Standardisation for Human Evaluation of NLP System Outputs

Paper · 2026 · Craig Thomson, Javier González Corbelle and Anya Belz

Summary

Running human evaluations of NLP output is demanding. Researchers often lack training in experimental design and must build interfaces, item lists and analysis code themselves for every study. This paper addresses the effort side of that problem by asking whether existing evaluations share enough structure to support standardisation and automation. The authors drew a stratified random sample of 100 ACL and EMNLP papers (2018–2024) that report human evaluations. They then encoded each experiment as a directed graph of data-transforming processes, such as sampling items, collecting responses, aggregating scores and running statistical tests. The annotation protocol was developed in stages: free-form annotation, a consensus vocabulary, a pilot round and a revision. The resulting graphs turned out to be remarkably uniform, with no loops or conditional branching. This allowed the authors to merge them into one canonical process model. They argue the model can guide the design of comparable, reproducible experiments and can serve as the blueprint for a modular toolkit that automates much of the work. The survey also confirms that statistical testing and agreement reporting are widely neglected.

Key points

  • An ACL Anthology search for "human evaluation" or "manual evaluation" returned 2,317 papers. Of these, 100 were sampled (48 ACL, 52 EMNLP), with 24 more reserved for development and piloting.
  • 22 initially sampled papers were excluded and replaced from the same conference × year stratum:
    • 13 had no human evaluation.
    • 3 reported no per-system scores.
    • 6 involved user-system interaction.
    • In total, 100 of the first 110 candidates (91%) were usable.
  • The protocol uses 10 standardised process names (e.g., Item Sampling, Interface Creation, Response Aggregation, Agreement, Statistical Test). Each graph runs from a Candidate Evaluation Items start node to a Results end node and was drawn in the Dia tool.
  • In the pilot, two annotators covered 14 papers containing 16 experiments. They agreed on 13 of the 16 graphs, and every disagreement came from a missed statistical test.
  • Roughly four in five experiments report no significance testing, and only about one third report inter-annotator agreement.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Dublin City UniversityDublin · Ireland70 works · 9,447 citations🎓Universidade de Santiago de CompostelaSantiago de Compostela · Spain13 works · 364 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Craig Thomson, Javier González Corbelle and Anya Belz. 2026. Process Standardisation for Human Evaluation of NLP System Outputs. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.gem-main.64

@inproceedings{thomson2026process,
  title = {{Process Standardisation for Human Evaluation of NLP System Outputs}},
  author = {Craig Thomson, Javier González Corbelle and Anya Belz},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.64},
  url = {https://doi.org/10.18653/v1/2026.gem-main.64}
}

References

32 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zeina Abu-Aisheh, Romain Raveaux, Jean-Yves Ramel, and Patrick Martineau. 2015. An Exact Graph Edit Distance Algorithm for Solving Pattern Recognition Problems. In 4th International Conference on Pattern Recognition Applications and Methods 2015, Lisbon, Portugal. find
  2. Anya Belz, Simon Mille, and David M. Howcroft. 2020. Disentangling the properties of human evaluation methods: A classification system to support comparability, meta-evaluation and reproducibility testing. In Proceedings of the 13th International Conference on Natural Language Generation, pages 183–… find
  3. Anya Belz, Simon Mille, and Craig Thomson. 2025a. The qcet taxonomy of standard quality criterion names and definitions for the evaluation of nlp systems. Preprint, arXiv:2509.22064. 39 pages, 7 figures. — [arXiv:2509.22064](https://arxiv.org/abs/2509.22064) find
  4. Anya Belz, Simon Mille, and Craig Thomson. 2025b. Standard quality criteria derived from current NLP evaluations for guiding evaluation design and grounding comparability and AI compliance assessments. In Findings of the Association for Computational Linguistics: ACL 2025, pages 26685–26715, Vienna,… find
  5. Anya Belz, João Sedoc, Craig Thomson, Simon Mille, and Rudali Huidrom. 2024. find
  6. The INLG 2024 tutorial on human evaluation of NLP system quality: Background, overall aims, and summaries of taught units. In Proceedings of the 17th International Natural Language Generation Conference: Tutorial Abstract, pages 1–12, Tokyo, Japan. Association for Computational Linguistics. find
  7. Anya Belz and Craig Thomson. 2025. HEDS 3.0: The human evaluation data sheet version 3.0. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²), pages 60–81, Vienna, Austria and virtual meeting. Association for Computational Linguistics. find
  8. Anya Belz, Craig Thomson, Ehud Reiter, and Simon Mille. 2023. Non-repeatable experiments and non-reproducible results: The reproducibility crisis in human evaluation in NLP. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3676–3687, Toronto, Canada. Association for Comp… find
  9. Pengshan Cai, Kaiqiang Song, Sangwoo Cho, Hongwei Wang, Xiaoyang Wang, Hong Yu, Fei Liu, and Dong Yu. 2023. Generating user-engaging news headlines. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3265–3280, Toronto, Canada. A… find
  10. Eduardo Calò, Lydia Penkert, and Saad Mahamood. 2025. Lessons from a user experience evaluation of NLP interfaces. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2915–2929, Albuquerque, New Mexico. Association for Computational Linguistics. find
  11. Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020. With little power comes great responsibility. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9263–9274, Online. Association for Computational… find
  12. Angela Fan, Mike Lewis, and Yann Dauphin. 2019. Strategies for structuring story generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2650– 2660, Florence, Italy. Association for Computational Linguistics. find
  13. Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. 2023. Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text. Journal of Artificial Intelligence Research, 77:103–166. find
  14. Mika Hämäläinen and Khalid Alnajjar. 2021. The great misalignment problem in human evaluation of NLP methods. In Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), pages 69–74, Online. Association for Computational Linguistics. find
  15. David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In Proceedings… find
  16. David M. Howcroft and Verena Rieser. 2021. What happens if you treat ordinal ratings as interval data? human evaluations in NLP are even more underpowered than you think. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8932–8939, Online and Punta Cana… find
  17. Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021. DExperts: Decoding-time controlled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th I…
  18. Kristýna Onderková, Mateusz Lango, Patrícia Schmidtová, and Ondrej Dusek. 2025. ReproHum \#0669-08: Reproducing sentiment transfer evaluation. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²), pages 601–608, Vienna, Austria and virtual meeting. Association for Comput… find
  19. Birgit Popp, Sarah Keck, Androniki Mertsiotaki, Emily Kratsch, and Alexander Daum. 2025. Which method(s) to pick when evaluating large language models with humans? - a comparison of 6 methods. Fraunhofer Publica Preprint. Preprint. find
  20. Abhilash Potluri, Fangyuan Xu, and Eunsol Choi. 2023. Concise answers to complex questions: Summarization of long-form answers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9709–9728, Toronto, Canada. Association for Comput… find
  21. Marzena Karpinska, Nader Akoury, and Mohit Iyyer. 2021. The perils of using Mechanical Turk to evaluate open-ended text generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1265–1285, Online and Punta Cana, Dominican Republic. Association for C… find
  22. Jie Ruan, Wenqing Wang, and Xiaojun Wan. 2024. Defining and detecting vulnerability in human evaluation guidelines: A preliminary study towards reliable NLG evaluation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag… find
  23. Zdeněk Kasner, Ondrej Platek, Patricia Schmidtova, Simone Balloccu, and Ondrej Dusek. 2024. factgenie: A framework for span-based evaluation of generated texts. In Proceedings of the 17th International Natural Language Generation Conference: System Demonstrations, pages 13–15, Tokyo, Japan. Associa… find
  24. Fahime Same, Guanyi Chen, and Kees Van Deemter. 2022. Non-neural models matter: a re-evaluation of neural referring expression generation systems. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5554–5567, Dublin, Ireland. Ass… find
  25. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388–395, Barcelona, Spain. Association for Computational Linguistics.
  26. Alberto Sanfeliu and King-Sun Fu. 1983. A distance measure between attributed relational graphs for pattern recognition. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13(3):353–362. find
  27. Elisa Kreiss, Fei Fang, Noah Goodman, and Christopher Potts. 2022. Concadia: Towards image-based text generation with a purpose. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4667–4684, Abu Dhabi, United Arab Emirates. Association for Computational… find
  28. Patricia Schmidtova, Eduardo Calò, Simone Balloccu, Dimitra Gkatzia, Rudali Huidrom, Mateusz Lango, Fahime Same, Vilém Zouhar, Saad Mahamood, and Ondrej Dusek. 2025. Do my eyes deceive me? a survey of human evaluations of hallucinations in NLG. In Proceedings of the 18th International Natural Langua… find
  29. Anastasia Shimorina and Anya Belz. 2022. The human evaluation datasheet: A template for recording details of human evaluation experiments in NLP. In Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval), pages 54–75, Dublin, Ireland. Association for Computational Linguistics. find
  30. Alexander Spangher, Nanyun Peng, Sebastian Gehrmann, and Mark Dredze. 2024. Do LLMs plan like human writers? comparing journalist coverage of press releases with LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21814–21828, Miami, Florida, USA. A… find
  31. Karen Sparck Jones and J. R. Galliers. 1995. Evaluating Natural Language Processing Systems: An Analysis and Review, volume 1083 of Lecture Notes in Computer Science; Lecture Notes in Artificial Intelligence. Springer, Berlin. Includes bibliographical references (p. 219–225) and index. find
  32. Craig Thomson and Anya Belz. 2024. (mostly) automatic experiment execution for human evaluations of NLP systems. In Proceedings of the 17th International Natural Language Generation Conference, pages 272–279, Tokyo, Japan. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/thomson-2026-process-standardisation-human-evaluation-nlp/