Elamur Research⌕
☰
home / papers / 2026 / ReproNLP 2026: A Third Replication of the Human Evaluation of a QAG System for Children's Storybooks

ReproNLP 2026: A Third Replication of the Human Evaluation of a QAG System for Children's Storybooks

Paper · 2026 · Marcel Mroczek, Chiara Albarello, Paul-Emmanuel Floch et al.

Summary

This paper is the third independent attempt, within the ReproNLP shared task, to redo the human-rated assessment originally reported by Yao et al. (2022). That study rated a Question-Answer Generation (QAG) system for children's storybooks, built on the FairytaleQA dataset, against the PAQ baseline and expert-written Ground Truth (GT). Five NLP-literate annotators scored QA pairs on 5-point scales for Readability, Question Relevance and Answer Relevance. Before the main study, the authors ran a pilot. Reproducibility was measured with the QRA3 framework.

The system rankings largely held: GT came out on top everywhere, and QAG beat PAQ on Readability and Question Relevance. The one exception was a small, non-significant QAG/PAQ swap on Answer Relevance. Agreement between annotators on Readability, however, fell to almost zero.

The authors also found flaws in the distributed data, most importantly a "val" placeholder in PAQ outputs. When the affected sections are removed, QAG's advantage disappears, both in the original data and in the new run. This suggests the result reproduces in terms of rankings, but those rankings may be driven by artifacts rather than real quality differences.

Key points

  • Under QRA3, 8 of 9 pairwise system comparisons were reproduced (P = 0.889). Readability rankings stayed fully consistent across four studies, including Florescu et al. (2024) and Braun (2025) (P = 1.000).
  • Krippendorff's α for Readability was 0.089, compared with 0.449 originally. The authors attribute this to two causes:
    • lenient and strict annotators were paired in a sparse randomized grid;
    • there was no calibration phase.
  • The pilot revealed several data problems:
    • 44% of PAQ outputs contained a "val" placeholder;
    • all text was lowercased;
    • GT answers had an extra space before punctuation;
    • some QA pairs were duplicated;
    • the number of pairs per section was uneven, reaching up to 13.
  • After excluding the "val"-affected sections (23 of 40, leaving 334 of 722 pairs), QAG no longer outperformed PAQ. In the new run, PAQ even scored significantly higher on Answer Relevance (3.95 vs 3.53, p = 0.008).
  • All data and code are released at github.com/SamsungLabs/ReproNLP2026.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢SamsungWarsaw · Poland2 works · 716 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Marcel Mroczek, Chiara Albarello, Paul-Emmanuel Floch and Maciej Gawinecki. 2026. ReproNLP 2026: A Third Replication of the Human Evaluation of a QAG System for Children's Storybooks. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.gem-main.85

@inproceedings{mroczek2026repronlp,
  title = {{ReproNLP 2026: A Third Replication of the Human Evaluation of a QAG System for Children's Storybooks}},
  author = {Marcel Mroczek, Chiara Albarello, Paul-Emmanuel Floch and Maciej Gawinecki},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.85},
  url = {https://doi.org/10.18653/v1/2026.gem-main.85}
}

References

21 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ron Artstein and Massimo Poesio. 2008. Survey article: Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596.
  2. Anya Belz, Shubham Agarwal, Anastasia Shimorina, and Ehud Reiter. 2021. A systematic review of reproducibility research in natural language processing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 381–393, Online.… find
  3. Anya Belz and Craig Thomson. 2023. find
  4. The 2023 ReproNLP shared task on reproducibility of evaluations in NLP: Overview and results. In Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems, pages 35–48, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria. find
  5. Anya Belz and Craig Thomson. 2024. find
  6. The 2024 ReproNLP shared task on reproducibility of evaluations in NLP: Overview and results. In Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024, pages 91–105, Torino, Italia. ELRA aRAnd ICCL. find
  7. Anya Belz and Craig Thomson. 2025. HEDS 3.0: The human evaluation data sheet version 3.0. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²), pages 60–81, Vienna, Austria and virtual meeting. Association for Computational Linguistics. find
  8. Anya Belz and Craig Thomson. 2026. Quantified reproducibility assessment for four common types of evaluation results in nlp/ml. Computational Linguistics, pages 1–10. find
  9. Anya Belz, Craig Thomson, and Javier Gonz’alez Corbelle. 2026. The shared task on reproducibility of evaluations in nlp (ReproNLP) 2026: Overview and results. In Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM²), San Diego, USA. Association for Computational Linguistics. find
  10. Anya Belz, Craig Thomson, Javier González Corbelle, and Malo Ruelle. 2025. find
  11. The 2025 ReproNLP shared task on reproducibility of evaluations in NLP: Overview and results. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²), pages 1002–1016, Vienna, Austria and virtual meeting. Association for Computational Linguistics. find
  12. Anya Belz, Craig Thomson, Ehud Reiter, Gavin Abercrombie, Jose M. Alonso-Moral, Mohammad Arvan, Anouck Braggaar, Mark Cieliebak, Elizabeth Clark, Kees van Deemter, Tanvi Dinkar, Ondřej Dušek, Steffen Eger, Qixiang Fang, Mingqi Gao, Albert Gatt, Dimitra Gkatzia, Javier González-Corbelle, Dirk Hovy,… find
  13. Daniel Braun. 2025. ReproHum \#0031-01: Reproducing the human evaluation of readability from “it is AI’s turn to ask humans a question”. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²), pages 576–582, Vienna, Austria and virtual meeting. Association for Computationa… find
  14. Andra-Maria Florescu, Marius Micluta-Campeanu, and Liviu P. Dinu. 2024. Once upon a replication: It is humans’ turn to evaluate AI’s understanding of children’s stories for QA generation. In Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024, pages 106… find
  15. David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In Proceedings… find
  16. Neslihan Iskender, Tim Polzehl, and Sebastian Möller. 2021. Reliability of human evaluation for text summarization: Lessons learned and challenges ahead. In Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), pages 86–96, Online. Association for Computational Linguistics. find
  17. Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021. PAQ: 65 million probably-asked questions and what you can do with them. Transactions of the Association for Computational Linguistics, 9:1098–1115.
  18. Siamak Shakeri, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng, Feng Nan, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang. 2020. End-to-end synthetic data generation for domain adaptation of question answering systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language… find
  19. Craig Thomson, Ehud Reiter, and Anya Belz. 2024. Common flaws in running human evaluation experiments in NLP. Computational Linguistics, 50(2):795–805. find
  20. Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Jia-Jun Li, Nora Bradford, Branda Sun, Tran Bao Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nanyun Peng, Zhou Yu, and Mark Warschauer. 2022. Fantastic questions and where to find them: FairytaleQA –…
  21. Bingsheng Yao, Dakuo Wang, Tongshuang Wu, Zheng Zhang, Toby Jia-Jun Li, Mo Yu, and Ying Xu. 2022. It is AI’s turn to ask humans a question: Question-answer pair generation for children’s story books. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mroczek-2026-repronlp-2026-third-replication-human/