Elamur Research⌕
☰
home / papers / 2024 / EXCGEC: A Benchmark for Edit-Wise Explainable Chinese Grammatical Error Correction

EXCGEC: A Benchmark for Edit-Wise Explainable Chinese Grammatical Error Correction

Paper · 2024 · Jingheng Ye, Shang Qin, Yinghui Li et al.

Summary

Most grammatical error correction (GEC) systems return only a corrected sentence. This leaves second-language learners unsure what was wrong. Earlier explanation research also handled explaining apart from correcting. The authors define EXGEC, a task in which one model both fixes a text and justifies each edit. They release EXCGEC, a Chinese benchmark of 8,216 samples. Each edit has a hybrid explanation with three parts:

  • an error type from a new two-level taxonomy,
  • a severity score from 1 to 5,
  • a free-text description that names evidence words, the relevant rule, the cause and the fix.

GPT-4 drafted the data, and experienced annotators then removed invalid items. The authors tested open-source LLMs in two multi-task orders: post-explaining (correct, then explain) and pre-explaining (explain, then correct). They also added a decoding algorithm called COTE. Correcting first works better, but both multi-task variants fall behind a separate GEC-then-GEE pipeline. Human ratings show that n-gram metrics such as ROUGE and METEOR track description quality well. The field therefore gains a cheap evaluation suite that has been checked against human judgement.

Key points

  • EXCGEC dataset: 8,216 explanation-augmented Chinese samples, built semi-automatically from GPT-4 generations followed by human filtering. Code and data are at github.com/THUKElab/EXCGEC.
  • Explanation design: explanations are edit-wise and combine an error type (two-tier taxonomy aimed at L2 errors as well as native ones), a 1–5 severity level and a natural-language error description.
  • COTE decoding: COTE (Correct-Then-Explain) supplies gold alignment to post-explaining models. In an ablation on Qwen1.5-7B-chat, it raised the edit hit rate from 49.64 to 99.93.
  • Pipeline comparison: with Qwen1.5-7B-chat, post-explaining beats pre-explaining, yet a GEC→GEE pipeline still outperforms both on correction and explanation. The authors suggest 7B models struggle to make the two tasks reinforce each other.
  • Human evaluation: ROUGE-1 showed the strongest agreement with human scores (Pearson 0.9464, Spearman 0.8286), followed by METEOR (0.9280 / 0.7714). Inter-annotator Pearson correlation was 0.9874.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓Peking UniversityBeijing · China244 works · 39,359 citations🎓Sun Yat-sen UniversityGuangzhou · China53 works · 9,185 citations🎓Central South UniversityChangsha · China15 works · 582 citations🔬Guangdong Laboratory of Artificial Intelligence and Digital Economy (Shenzhen)Guangming District · China2 works · 18 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Jingheng Ye, Shang Qin, Yinghui Li, Xuxin Cheng, Libo Qin, Hai-Tao Zheng, Ying Shen, Peng Xing, Zishan Xu, Guo Cheng and Wenhao Jiang. 2024. EXCGEC: A Benchmark for Edit-Wise Explainable Chinese Grammatical Error Correction. arXiv:2407.00924.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2407.00924 · arXiv 2407.00924

@misc{ye2024excgec,
  title = {{EXCGEC: A Benchmark for Edit-Wise Explainable Chinese Grammatical Error Correction}},
  author = {Jingheng Ye, Shang Qin, Yinghui Li, Xuxin Cheng, Libo Qin, Hai-Tao Zheng, Ying Shen, Peng Xing, Zishan Xu, Guo Cheng and Wenhao Jiang},
  year = {2024},
  doi = {10.48550/arxiv.2407.00924},
  eprint = {2407.00924},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.00924}
}

References

52 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Bai, J.; Bai, S.; Chu, Y.; Cui, Z.; Dang, K.; Deng, X.; Fan, Y.; Ge, W.; Han, Y.; Huang, F.; et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.
  3. Banerjee, S.; and Lavie, A. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Goldstein, J.; Lavie, A.; Lin, C.-Y.; and Voss, C., eds., Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or S…
  4. Bi, X.; Chen, D.; Chen, G.; Chen, S.; Dai, D.; Deng, C.; Ding, H.; Dong, K.; Du, Q.; Fu, Z.; et al. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. find
  5. Bird, S.; and Loper, E. 2004. NLTK: The Natural Language Toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, 214–217. Barcelona, Spain: Association for Computational Linguistics.
  6. Bryant, C.; Felice, M.; Andersen, Ø. E.; and Briscoe, T. 2019. The BEA-2019 Shared Task on Grammatical Error Correction. In Yannakoudakis, H.; Kochmar, E.; Leacock, C.; Madnani, N.; Pilán, I.; and Zesch, T., eds., Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educatio…
  7. Bryant, C.; Felice, M.; and Briscoe, T. 2017. Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction. In Barzilay, R.; and Kan, M.-Y., eds., Proceedings of ACL, 793–805. Vancouver, Canada: Association for Computational Linguistics.
  8. Bryant, C.; Yuan, Z.; Qorib, M. R.; Cao, H.; Ng, H. T.; and Briscoe, T. 2023. Grammatical error correction: A survey of the state of the art. Computational Linguistics, 49(3): 643– 701. Camburu, O.-M.; Rocktäschel, T.; Lukasiewicz, T.; and Blunsom, P. 2018. e-snli: Natural language inference with n… find
  9. Cao, Q.; Kojima, T.; Matsuo, Y.; and Iwasawa, Y. 2023. Unnatural error correction: Gpt-4 can almost perfectly handle unnatural scrambled text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 8898–8913. find
  10. Clinciu, M.-A.; Eshghi, A.; and Hastie, H. 2021. A Study of Automatic Metrics for the Evaluation of Natural Language Explanations. In Merlo, P.; Tiedemann, J.; and Tsarfaty, R., eds., Proceedings of EACL, 2376–2387. Online: Association for Computational Linguistics. find
  11. Ding, B.; Qin, C.; Zhao, R.; Luo, T.; Li, X.; Chen, G.; Xia, W.; Hu, J.; Luu, A. T.; and Joty, S. 2024. Data augmentation using llms: Data perspectives, learning paradigms and challenges. arXiv preprint arXiv:2403.02990. find
  12. Dwivedi, R.; Dave, D.; Naik, H.; Singhal, S.; Omer, R.; Patel, P.; Qian, B.; Wen, Z.; Shah, T.; Morgan, G.; et al. 2023. Explainable AI (XAI): Core ideas, techniques, and solutions. ACM Computing Surveys, 55(9): 1–33. find
  13. Fei, Y.; Cui, L.; Yang, S.; Lam, W.; Lan, Z.; and Shi, S. 2023. Enhancing Grammatical Error Correction Systems with Explanations. In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Proceedings of ACL, 7489–7501. Toronto, Canada: Association for Computational Linguistics. find
  14. Gao, C.; Zhang, W.; Chen, G.; and Lam, W. 2023. Json-Tuning: Towards Generalizable, Robust, and Controllable Instruction Tuning. arXiv preprint arXiv:2310.02953. — [arXiv:2310.02953](https://arxiv.org/abs/2310.02953) find
  15. Hartmann, M.; and Sonntag, D. 2022. A survey on improving NLP models with human explanations. In Andreas, J.; Narasimhan, K.; and Nematzadeh, A., eds., Proceedings of the First Workshop on Learning with Natural Language Supervision, 40–47. Dublin, Ireland: Association for Computational Linguistics. find
  16. He, X.; Wu, Y.; Camburu, O.-M.; Minervini, P.; and Stenetorp, P. 2023. Using Natural Language Explanations to Improve Robustness of In-context Learning for Natural Language Inference. arXiv preprint arXiv:2311.07556. find
  17. Huang, B.; and Liao, X. 2011. Modern Chinese (Updated Fifth Edition). find
  18. Higher Education Press, Beijing, China. Huang, H.; Ye, J.; Zhou, Q.; Li, Y.; Li, Y.; Zhou, F.; and Zheng, H.-T. 2023. A Frustratingly Easy Plug-and-Play Detection-and-Reasoning Module for Chinese Spelling Check. In Bouamor, H.; Pino, J.; and Bali, K., eds., Findings of the Association for Computatio… find
  19. Kaneko, M.; Takase, S.; Niwa, A.; and Okazaki, N. 2022. Interpretability for Language Learners Using Example-Based Grammatical Error Correction. In Muresan, S.; Nakov, P.; and Villavicencio, A., eds., Proceedings of ACL, 7176–7187. Dublin, Ireland: Association for Computational Linguistics.
  20. Li, D.; Hu, B.; Chen, Q.; Xu, T.; Tao, J.; and Zhang, Y. 2022. Unifying model explainability and robustness for joint text classification and rationale extraction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 10947–10955. find
  21. Li, Q.; Peng, H.; Li, J.; Xia, C.; Yang, R.; Sun, L.; Yu, P. S.; and He, L. 2020. A survey on text classification: From shallow to deep learning. arXiv preprint arXiv:2008.00364. Li, S.; Chen, J.; yelong shen; Chen, Z.; Zhang, X.; Li, Z.; Wang, H.; Qian, J.; Peng, B.; Mao, Y.; Chen, W.; and Yan, X.… find
  22. Li, Y.; Ma, S.; Chen, S.; Huang, H.; Huang, S.; Li, Y.; Zheng, H.-T.; and Shen, Y. 2025. Correct like humans: Progressive learning framework for Chinese text error correction. Expert Systems with Applications, 265: 126039. find
  23. Lin, C.-Y. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, 74–81. Barcelona, Spain: Association for Computational Linguistics.
  24. Liu, R.; Wei, J.; Liu, F.; Si, C.; Zhang, Y.; Rao, J.; Zheng, S.; Peng, D.; Yang, D.; Zhou, D.; et al. 2024. Best practices and lessons learned on synthetic data for language models. arXiv preprint arXiv:2404.07503.
  25. Ma, S.; Li, Y.; Sun, R.; Zhou, Q.; Huang, S.; Zhang, D.; Yangning, L.; Liu, R.; Li, Z.; Cao, Y.; Zheng, H.; and Shen, Y. 2022. Linguistic Rules-Based Corpus Generation for Native Chinese Grammatical Error Correction. In Goldberg, Y.; Kozareva, Z.; and Zhang, Y., eds., Findings of the Association for…
  26. Nagata, R. 2019. Toward a task of feedback comment generation for writing learning. In Proceedings of EMNLP-IJCNLP, 3206–3215. find
  27. Nagata, R.; Hagiwara, M.; Hanawa, K.; Mita, M.; Chernodub, A.; and Nahorna, O. 2021. Shared task on feedback comment generation for language learners. In Proceedings of the 14th International Conference on Natural Language Generation, 320–324. find
  28. Nagata, R.; Inui, K.; and Ishikawa, S. 2020. Creating corpora for research in feedback comment generation. In Proceedings of the Twelfth Language Resources and Evaluation Conference, 340–345. find
  29. Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Isabelle, P.; Charniak, E.; and Lin, D., eds., Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 311–318. Philadelphia, Pennsylvania, USA…
  30. Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; Vanderplas, J.; Passos, A.; Cournapeau, D.; Brucher, M.; Perrot, M.; and Duchesnay, E. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning R… find
  31. Schneider, D.; and McCoy, K. F. 1998. Recognizing Syntactic Errors in the Writing of Second Language Learners. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 2, 1198–1204. Montreal, Quebec, Canada: Associ… find
  32. Shao, J. 2016. General Theory of Modern Chinese. find
  33. Shanghai Educational Publishing House, Shanghai, China. Shum, K.; Diao, S.; and Zhang, T. 2023. Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data. In Bouamor, H.; Pino, J.; and Bali, K., eds., Findings of the Association for Computational Linguistics: EMNLP 2023, 12…
  34. Song, Y.; Krishna, K.; Bhatt, R.; Gimpel, K.; and Iyyer, M. 2023. Gee! grammar error explanation with large language models. arXiv preprint arXiv:2311.09517. find
  35. Stahl, M.; Biermann, L.; Nehring, A.; and Wachsmuth, H. 2024. Exploring LLM Prompting Strategies for Joint Essay Scoring and Feedback Generation. arXiv preprint arXiv:2404.15845. find
  36. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  37. Wang, Y.; Kong, C.; Yang, L.; Wang, Y.; Lu, X.; Hu, R.; He, S.; Liu, Z.; Chen, Y.; Yang, E.; et al. 2021. YACLC: a Chinese learner corpus with multidimensional annotation. arXiv preprint arXiv:2112.15043. — [arXiv:2112.15043](https://arxiv.org/abs/2112.15043) find
  38. Wang, Y.; Zhong, W.; Li, L.; Mi, F.; Zeng, X.; Huang, W.; Shang, L.; Jiang, X.; and Liu, Q. 2023. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966.
  39. Whitehouse, C.; Choudhury, M.; and Aji, A. F. 2023. LLM-powered Data Augmentation for Enhanced Cross-lingual Performance. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of EMNLP, 671–686. Singapore: Association for Computational Linguistics. find
  40. Wiegreffe, S.; and Marasovic, A. 2021. Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing. In Vanschoren, J.; and Yeung, S., eds., Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. find
  41. Xu, L.; Wu, J.; Peng, J.; Fu, J.; and Cai, M. 2022. FCGEC: Fine-Grained Corpus for Chinese Grammatical Error Correction. In Goldberg, Y.; Kozareva, Z.; and Zhang, Y., eds., Findings of the Association for Computational Linguistics: EMNLP 2022, 1900–1918. Abu Dhabi, United Arab Emirates: Association… find
  42. Ye, J.; Li, Y.; Li, Y.; and Zheng, H. 2023a. MixEdit: Revisiting Data Augmentation and Beyond for Grammatical Error Correction. In Bouamor, H.; Pino, J.; and Bali, K., eds., Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, 10161–10175. Associatio… find
  43. Ye, J.; Li, Y.; Ma, S.; Xie, R.; Wu, W.; and Zheng, H.-T. 2022. Focus is what you need for chinese grammatical error correction. arXiv preprint arXiv:2210.12692. — [arXiv:2210.12692](https://arxiv.org/abs/2210.12692) find
  44. Ye, J.; Li, Y.; and Zheng, H. 2023. System Report for CCL23-Eval Task 7: THU KELab (sz) - Exploring Data Augmentation and Denoising for Chinese Grammatical Error Correction. In Sun, M.; Qin, B.; Qiu, X.; Jiang, J.; and Han, X., eds., Proceedings of the 22nd Chinese National Conference on Computation… find
  45. Ye, J.; Li, Y.; Zhou, Q.; Li, Y.; Ma, S.; Zheng, H.-T.; and Shen, Y. 2023b. CLEME: Debiasing Multi-reference Evaluation for Grammatical Error Correction. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of EMNLP, 6174– 6189. Association for Computational Linguistics. find
  46. Ye, J.; Wang, S.; Zou, D.; Yan, Y.; Wang, K.; Zheng, H.-T.; Xu, Z.; King, I.; Yu, P. S.; and Wen, Q. 2025. Position: LLMs Can be Good Tutors in Foreign Language Education. arXiv preprint arXiv:2502.05467. — [arXiv:2502.05467](https://arxiv.org/abs/2502.05467) find
  47. Ye, J.; Xu, Z.; Li, Y.; Cheng, X.; Song, L.; Zhou, Q.; Zheng, H.-T.; Shen, Y.; and Su, X. 2024. CLEME2. 0: Towards More Interpretable Evaluation by Disentangling Edits for Grammatical Error Correction. arXiv preprint arXiv:2407.00934. find
  48. Zhang, B. 2009. Features and functions of the HSK dynamic composition corpus. International Chinese Language Education, 4: 71–79. find
  49. Zhang, Y.; Li, Z.; Bao, Z.; Li, J.; Zhang, B.; Li, C.; Huang, F.; and Zhang, M. 2022. MuCGEC: a multi-reference multi-source evaluation dataset for Chinese grammatical error correction. arXiv preprint arXiv:2204.10994. find
  50. Zhao, H.; Chen, H.; Yang, F.; Liu, N.; Deng, H.; Cai, H.; Wang, S.; Yin, D.; and Du, M. 2024. Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology, 15(2): 1–38.
  51. Zhao, Y.; Jiang, N.; Sun, W.; and Wan, X. 2018. Overview of the nlpcc 2018 shared task: Grammatical error correction. In CCF International Conference on Natural Language Processing and Chinese Computing, 439–445. find
  52. The Details of EXCGEC Benchmark Description of Grammatical Error Types In the taxonomy of Chinese grammatical errors, we classify errors based on their level of granularity into five major types: punctuation-level, spelling-level, word-level, sentence-level, and other special errors. This section pr… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ye-2024-excgec-benchmark-edit-wise-explainable/