Elamur Research⌕
☰
home / papers / 2023 / ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark

ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark

Paper · 2023 · Haoran Wu, Wenxuan Wang, Yuxuan Wan et al.

Summary

This early technical report asks whether ChatGPT can serve as a grammatical error correction (GEC) system. The authors compare it with the commercial assistant Grammarly and with GECToR, a strong research model built on RoBERTa. They use 100 sentences from the CoNLL-2014 test set, which contains essays by non-native English writers. ChatGPT was queried by hand with a single instruction prompt and scored with the official CoNLL-2014 scorer (Precision, Recall, F0.5). By these automatic measures, ChatGPT ranks last, and the gap widens on longer sentences. Inspecting the outputs explains much of this. ChatGPT tends to rephrase and restructure sentences rather than make minimal fixes, so its fluent and grammatical rewrites still diverge from the reference edits. A small manual annotation shows that ChatGPT misses fewer errors but adds many unrequested changes. The authors argue that reference-based metrics undervalue fluency-oriented correctors and that ChatGPT is a promising GEC tool. Caveats include the tiny sample, a single prompt, and no in-context examples.

Key points

  • Overall scores (Precision / Recall / F0.5):

    • GECToR: 71.2 / 38.4 / 60.8
    • Grammarly: 67.3 / 51.1 / 63.3
    • ChatGPT: 51.2 / 62.8 / 53.1

    ChatGPT has the highest recall, GECToR the highest precision, and Grammarly the best F0.5.

  • Sentence length: The test set was split into short, medium and long thirds. ChatGPT comes close to Grammarly on short sentences but drops sharply on long ones.

  • Grammarly as a second pass: Running Grammarly over the systems' outputs barely changes ChatGPT's results (+0.5 F0.5), mostly through punctuation fixes. For GECToR, it adds +2.1 F0.5 and +16.5 recall, meaning GECToR leaves many errors in place.

  • Human evaluation: Two annotators labelled 20 sentences, counting under-, mis- and over-corrections:

    • GECToR: 13 / 4 / 0
    • Grammarly: 14 / 0 / 1
    • ChatGPT: 3 / 3 / 30

  • Setup limitations: The study uses only the CoNLL-2014 data and one zero-shot prompt. It does not test few-shot or chain-of-thought prompting, and it relies only on M²-style metrics, with no pretraining-based metrics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢TencentShenzhen · China163 works · 21,032 citations🎓Chinese University of Hong KongHong Kong · China159 works · 16,154 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 worksGrammatical error correction & learner language · 162 works

Cite

Cite as Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao and Michael Lyu. 2023. ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark. arXiv:2303.13648.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.13648 · arXiv 2303.13648

@misc{wu2023chatgpt,
  title = {{ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark}},
  author = {Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao and Michael Lyu},
  year = {2023},
  doi = {10.48550/arxiv.2303.13648},
  eprint = {2303.13648},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.13648}
}

References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. ArXiv.
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. NeurIPS.
  3. Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019. The bea-2019 shared task on grammatical error correction. In BEA@ACL.
  4. Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng, and Ted Briscoe. 2022. Grammatical error correction: A survey of the state of the art. ArXiv. find
  5. Paul Francis Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. NeruIPS. find
  6. Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and J J Berner. 2023. Mathematical capabilities of chatgpt. ArXiv.
  7. Peiyuan Gong, Xuebo Liu, Heyan Huang, and Min Zhang. 2022. Revisiting grammatical error correction evaluation and beyond. EMNLP. find
  8. Grammarly. 2023. Grammarly website about us page. find
  9. Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is ChatGPT a good translator? a preliminary study. In ArXiv. find
  10. Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The conll-2014 shared task on grammatical error correction. In CoNLL. find
  11. Reham Omar, Omij Mangukiya, Panos Kalnis, and Essam Mansour. 2023. Chatgpt versus traditional question answering for knowledge graphs: Current status and future directions towards knowledge graph chatbots. ArXiv. find
  12. Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem N. Chernodub, and Oleksandr Skurzhanskyi. 2020. Gector – grammatical error correction: Tag, not rewrite. In Workshop on Innovative Use of NLP for Building Educational Applications. find
  13. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv.
  14. Sebastian Ruder. 2022. NLP-progress. find
  15. Joel R. Tetreault, Keisuke Sakaguchi, and Courtney Napoles. 2017. Jfleg: A fluency corpus and benchmark for grammatical error correction. In EACL. find
  16. Wenxuan Wang, Wenxiang Jiao, Yongchang Hao, Xing Wang, Shuming Shi, Zhaopeng Tu, and Michael Lyu. 2022. Understanding and improving sequence-to-sequence pretraining for neural machine translation. In ACL. find
  17. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022a. Finetuned language models are zero-shot learners. ICLR.
  18. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, Quoc Le, and Denny Zhou. 2022b. Chain of thought prompting elicits reasoning in large language models. NeurIPS.
  19. Wikipedia contributors. 2023a. F-score — Wikipedia, the free encyclopedia. [Online; accessed 5-March-2023]. find
  20. Wikipedia contributors. 2023b. Grammarly — Wikipedia, the free encyclopedia. [Online; accessed 2-March-2023]. find
  21. Chun Xia and Lingming Zhang. 2023. Conversational automated program repair. ArXiv. find
  22. Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen, and Wei Cheng. 2023. Exploring the limits of chatgpt for query or aspect-based text summarization. ArXiv.
  23. Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert. ArXiv. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2023-chatgpt-grammarly-evaluating-chatgpt-grammatical/