Elamur Research⌕
☰
home / papers / 2023 / Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction

Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction

Paper · 2023 · Steven Coyne, Keisuke Sakaguchi, Diana Galvan-Sosa et al.

Summary

OpenAI's technical reports give no benchmark results for grammatical error correction (GEC). This study fills that gap by testing a GPT-3.5 model (text-davinci-003) and a GPT-4 model (gpt-4-0314) on standard GEC benchmarks. The task is framed as sentence-level revision: the model gets one possibly faulty sentence and returns one corrected version.

The authors compared ten zero-shot prompts at three temperature settings, using GLEU on the JFLEG development set to choose the best one. They then tried few-shot variants. The chosen prompt was evaluated on the BEA-2019 and JFLEG test sets. Human raters scored GPT outputs alongside source sentences, human references and two baseline GEC systems.

Both models handled the task reliably once given a well-designed prompt. GPT-4 set a new best score on JFLEG. However, the models tend toward fluency rewrites and sometimes over-edit, which hurts them on BEA-2019, a benchmark that rewards minimal edits.

The work shows that prompt wording, task definition and benchmark choice strongly affect how LLM-based GEC is judged.

Key points

  • Best zero-shot prompt (#10): an explicit instruction to fix grammar and spelling and to copy the sentence unchanged if it is already correct. On the JFLEG dev set it reached GLEU 0.582 (GPT-3.5) and 0.601 (GPT-4). A plain "more fluent" prompt scored only 0.314 and 0.245.
  • Temperature: the authors tested τ of 0.1, 0.5 and 0.9. Lower temperature consistently gave better results.
  • Prompt sensitivity: small changes, such as adding quotes or field labels, noticeably changed both correction quality and output format. Few-shot prompts used up to six example pairs.
  • Human evaluation: about 200 candidate sets were rated. Baselines beat the GPT models in 4 (BEA-2019) and 7 (JFLEG) cases, while human references beat them in 13 and 10 cases.
  • Error patterns: the models sometimes changed a sentence's meaning or completed fragments with invented content. Occasionally they under-edited instead, leaving ungrammatical spans unchanged.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Aix-Marseille UniversitéMarseille · France25 works · 39,241 citations🎓Tohoku UniversitySendai · Japan23 works · 595 citations🔬RIKENWako · Japan30 works · 607 citations🔬Laboratoire d’Informatique et SystèmesMarseille · France10 works · 187 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksGrammatical error correction & learner language · 162 works

Cite

Cite as Steven Coyne, Keisuke Sakaguchi, Diana Galvan-Sosa, Michael Zock and Kentaro Inui. 2023. Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction. arXiv:2303.14342.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.14342 · arXiv 2303.14342

@misc{coyne2023analyzing,
  title = {{Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction}},
  author = {Steven Coyne, Keisuke Sakaguchi, Diana Galvan-Sosa, Michael Zock and Kentaro Inui},
  year = {2023},
  doi = {10.48550/arxiv.2303.14342},
  eprint = {2303.14342},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.14342}
}

References

29 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  2. Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019. The BEA-2019 shared task on grammatical error correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52–75, Florence, Italy. Association for Computatio…
  3. Shamil Chollampatt and Hwee Tou Ng. 2018. A multi-layer convolutional encoder-decoder neural network for grammatical error correction. In AAAI Conference on Artificial Intelligence. find
  4. Tao Fang, Shu Yang, Kaixin Lan, Derek F. Wong, Jinpeng Hu, Lidia S. Chao, and Yue Zhang. 2023. Is chatgpt a highly fluent grammatical error correction system? a comprehensive evaluation.
  5. Roman Grundkiewicz and Marcin Junczys-Dowmunt. 2018. Near human-level performance in grammatical error correction with hybrid machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume… find
  6. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
  7. Md Asadul Islam and Enrico Magnani. 2021. Is this the end of the gold standard? a straightforward reference-less grammatical error correction metric. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3009–3015, Online and Punta Cana, Dominican Republic.… find
  8. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438.
  9. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, and Kenneth Heafield. 2018. Approaching neural grammatical error correction as a low-resource machine translation task. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find
  10. Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2020. Encoder-decoder models can benefit from pre-trained masked language models in grammatical error correction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4248–4254, Onli… find
  11. Shun Kiyono, Jun Suzuki, Masato Mita, Tomoya Mizumoto, and Kentaro Inui. 2019. An empirical study of incorporating pseudo data into grammatical error correction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Na… find
  12. Zhenghao Liu, Xiaoyuan Yi, Maosong Sun, Liner Yang, and Tat-Seng Chua. 2021. Neural quality estimation with multiple hypotheses for grammatical error correction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn… find
  13. Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical error correction metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volum… find
  14. Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. JFLEG: A fluency corpus and benchmark for grammatical error correction. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 229–234, Valencia, Spai… find
  15. Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–14, Baltimore, Ma… find
  16. Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. GECToR – grammatical error correction: Tag, not rewrite. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 163–170, Seattle, WA, USA → Online. A… find
  17. OpenAI. 2023. Gpt-4 technical report.
  18. Robert Ostling and Murathan Kurfalı. 2022. Really good grammatical error correction, and how to evaluate it. Swedish Language Technology Conference (SLTC). find
  19. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train…
  20. Muhammad Qorib, Seung-Hoon Na, and Hwee Tou Ng. 2022. Frustratingly easy system combination for grammatical error correction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1964–1974, Seattle, U… find
  21. Keisuke Sakaguchi and Benjamin Van Durme. 2018. Efficient online scalar annotation with bounded support. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 208–218, Melbourne, Australia. Association for Computational Linguistics. find
  22. Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. 2022. PEER: A Collaborative Language Model.
  23. Timo Schick and Hinrich Schütze. 2021. It’s not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339–2352, Online. Associ…
  24. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235, Onlin…
  25. Felix Stahlberg and Shankar Kumar. 2021. Synthetic data generation for grammatical error correction with tagged corruption models. In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pages 37–47, Online. Association for Computational Linguistics. find
  26. Maksym Tarnavskyi, Artem Chernodub, and Kostiantyn Omelianchuk. 2022. Ensembling and knowledge distilling of large sequence taggers for grammatical error correction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3842–3852, D… find
  27. Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. 2023. Chatgpt or grammarly? evaluating chatgpt on grammatical error correction benchmark.
  28. Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2021. LM-critic: Language models for unsupervised grammatical error correction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7752–7763, Online and Punta Cana, Dominican Republic. Association for C… find
  29. Michael Zock and Debela Tesfaye Gemechu. 2017. Use your mind and learn to write: The problem of producing coherent text. In Cognitive Approach to Natural Language Processing, pages 129–158. Elsevier. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/coyne-2023-analyzing-performance-gpt-3-5/