Elamur Research⌕
☰
home / papers / 2023 / Automated evaluation of written discourse coherence using GPT-4

Automated evaluation of written discourse coherence using GPT-4

Summary

Automatically judging discourse coherence, meaning how well the ideas in a text connect and flow, has long been a weak spot of automated writing evaluation (AWE). This study tests whether GPT-4 can score coherence in learner writing the way trained human raters do, and explain its scores. The authors built DET-Coh, a set of 500 Writing Sample responses from the Duolingo English Test, balanced by gender, first language and proficiency level. Expert raters scored the responses on a new six-level coherence rubric aligned with CEFR. GPT-4 was queried through the OpenAI API with a prompt containing the task description, the rubric, style guidelines, six scored examples, the writing prompt and the essay. Three setups were tried: score then explanation, explanation then score, and score alone. A regression baseline built on Coh-Metrix-style overlap and similarity features served as the comparison. Every GPT-4 setup matched human scores far better than the baseline. The results suggest LLMs could give interpretable coherence feedback, but the authors warn that the generated rationales may not reflect how GPT-4 actually reached its ratings.

Key points

  • DET-Coh dataset: 500 responses from 2022, covering seven L1 groups (Chinese, Arabic, Spanish, Telugu, English, Bengali, Gujarati) and CEFR levels A1–C2. A score of 0 was reserved for blank or bad-faith answers.
  • Human reliability: Four expert raters took part, and 80 responses were scored by two raters independently. On these, agreement reached 0.72 exact and 0.93 QWK.
  • GPT-4 vs. humans:
    • QWK was 0.81–0.82 for the conditions that included rationales and 0.78 for rating-only.
    • Spearman ρ reached up to 0.82.
    • Adjacent agreement was 0.95–0.97.
    • Exact agreement was 0.51–0.56.
  • Baseline: The baseline used local and global argument, noun and stem overlap, coreference overlap and LSA similarity, with leave-one-out linear regression. It reached only 0.39 QWK, ρ 0.47, 0.36 exact and 0.82 adjacent agreement.
  • Qualitative findings:
    • GPT-4 appeared to give spelling errors too much weight.
    • GPT-4 could also rewrite a learner's essay into a personalized model answer, which both GPT-4 and a human rater scored 5 (CEFR C1).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

45 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 42 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.bea-1.32

@inproceedings{naismith2023automated,
  title = {{Automated evaluation of written discourse coherence using GPT-4}},
  author = {Naismith, Ben and Mulcaire, Phoebe and Burstein, Jill},
  booktitle = {Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)},
  year = {2023},
  doi = {10.18653/v1/2023.bea-1.32},
  url = {https://doi.org/10.18653/v1/2023.bea-1.32}
}

References

35 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater® v.2. The Journal of Technology, Learning and Assessment, 4(3). find
  2. Yigal Attali, Andrew Runge, Geoff T. LaFlair, Kevin Yancey, Sarah Goodwin, Yena Park, and Alina A. Von Davier. 2022. The interactive reading task: Transformer-based automatic item generation. Frontiers in Artificial Intelligence, 5. find
  3. Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Computational Linguistics, 34(1):1–34. find our card
  4. Daniel Blanchard, Joel R. Tetreault, Derrick Higgins, A. Cahill, and Martin Chodorow. 2013. TOEFL11: A corpus of non-native english. ETS Research Report Series, 2013:15. find
  5. Bren Bridgeman. 2013. Human ratings and automated essay evaluation, pages 243–254. Routledge. find
  6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, et al. 2020. Language models are few-shot learners. In Advances in neural information processing systems, volume 33, pages 1877–1901. find
  7. Jill Burstein. 2023. Duolingo English Test Responsible AI Standards. Technical report, Duolingo. find
  8. Jill Burstein, Joel Tetreault, and Martin Chodorow. 2013. Holistic discourse coherence annotation for noisy essay writing. Dialogue & Discourse, 4(2):34–52. find our card
  9. Marti A. Hearst. 1997. Text tiling: Segmenting text into multi-paragraph subtopic passages. In Computational Linguistics, volume 23, pages 33–64. find
  10. International Test Commission and Association of Test Publishers (ITC-ATP). 2022. Guidelines for technology-based assessment. Technical report, International Test Commission and Association of Test Publishers, Washington, D.C. find
  11. Ramsey Cardwell, Ben Naismith, Geoffrey T. LaFlair, and Steven Nydick. 2023. Duolingo English Test: Technical Manual. find
  12. Paraskevas Lagakis and Stavros Demetriadis. 2021. Automated essay scoring: A review of the field. In Proceedings of the 2021 International Conference on Computer, Information and Telecommunication Systems (CITS), pages 1–6. Institute of Electrical and Electronics Engineers. find
  13. Council of Europe. 2001. Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Council of Europe Publishing, Cambridge, UK. find
  14. Alice Lai and Joel Tetreault. 2018. Discourse coherence in the wild: a dataset, evaluation and methods. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 214–223. Association for Computational Linguistics. find our card
  15. Council of Europe. 2020. Common European Framework of Reference for Languages: Learning, teaching, assessment – Companion volume. Council of Europe Publishing, Cambridge, UK. find
  16. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of NAACL, pages 687– 692. Association for Computational Linguistics. find our card
  17. Department for Science, Technology & Innovation. 2023. A pro-innovation approach to AI regulation. White paper, Crown. find
  18. Tim F. McNamara, Ute Knoch, and Jason Fan. 2019. Fairness, Justice and Language Assessment. Oxford University Press, Oxford, UK. find
  19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL, pages 4171– 4186, Minneapolis, Minnesota. Association for Computational Linguistics. find
  20. Atsushi Mizumoto and Masaki Eguchi. 2023. Exploring the potential of using an AI language model for automated essay scoring. Educational Technology Research and Development. find
  21. Virginia Dignum. 2021. The role and challenges of education for responsible AI. London Review of Education, 19(1):1–11. find
  22. National Governors Association Center for Best Practices and Council of Chief State School Officers. 2010. Common Core State Standards. find
  23. OpenAI. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774. find our card
  24. Mark D. Shermis and Jill Burstein. 2003. Automated essay scoring: A cross-disciplinary perspective. Routledge, Mahwah, NJ. find
  25. Lance Eliot. 2022. Enraged worries that generative AI ChatGPT spurs students to vastly cheat when writing essays, spawns spellbound attention for AI ethics and AI law. Forbes. find
  26. Mark D. Shermis and Jill Burstein. 2013. Handbook of automated essay evaluation. Routledge, New York, NY. find
  27. Tyna Eloundo, Sam Manning, Pamela Mishkin, and Daniel Rock. 2023. GPTs are GPTs: An early look at the labor market impact potential of large language models. arXiv preprint axXiv:2303.10130. find
  28. Steve Ferrara and Saed Qunbar. 2022. Validity arguments for AI-based automated scores: Essay scoring as an illustration. Journal of Educational Measurement. find
  29. Arthur C. Graesser, Danielle S. McNamara, Max M. Louwerse, and Zhiquiang Cai. 2004. Coh-metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, & Computers, 36(2):193–202. find
  30. Michael A. K. Halliday and Ruqaiya Hasan. 1976. Cohesion in English. Longman, London, UK. find
  31. Swapna Somasundaran, Jill Burstein, and Martin Chodorow. 2014. Lexical chaining for measuring discourse coherence quality in test-taker essays. In The 25th International Conference on Computational Linguistics (COLING), pages 23–29. find our card
  32. The International Privacy Subcommittee of the ATP Security Committee (ATP). 2021. Artificial intelligence and the testing industry: A primer. Association of Test Publishers. find
  33. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. 2023. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. arXiv preprint arXiv:2305.04388. find our card
  34. Paul Van den Broek, Charles R. Fletcher, and Kirsten Risden. 2009. Investigations of inferential processes in reading: A theoretical and methodological integration. Discourse Processes, 16:169–180. find
  35. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS). find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/naismith-2023-automated-evaluation-written-discourse-coherence/