Elamur Research⌕
☰
home / papers / 2023 / GrammarGPT: Exploring Open-Source LLMs for Native Chinese Grammatical Error Correction with Supervised Fine-Tuning

GrammarGPT: Exploring Open-Source LLMs for Native Chinese Grammatical Error Correction with Supervised Fine-Tuning

Paper · 2023 · Yaxin Fan; Feng Jiang; Peifeng Li; and Haizhou Li

Summary

Native speakers of Chinese make grammatical errors that sound fluent but still break grammar rules. These errors are harder to catch than learner mistakes, and corpora of them are scarce. The authors ask whether an open-source LLM can be adapted to correct them with very little training data. They built a hybrid parallel set of roughly 1k pairs from two sources:

  • For error types signalled by surface cues, such as redundant pairs like "more than … about", they prompted ChatGPT with those cues so it would generate faulty sentences.
  • For cue-less errors, they collected real faulty sentences from public websites and corrected them by hand.

An error-invariant augmentation step then swaps named entities for similar ones while leaving the error itself unchanged. They used this data to instruction-tune Phoenix (phoenix-inst-chat-7b) in a Seq2seq setup. The resulting model, GrammarGPT, roughly doubles the F0.5 of the S2S BART baseline. That baseline was trained on about 1.2M learner-error pairs, around 1200× more data. GrammarGPT placed 3rd in NLPCC2023 SharedTask1. The result suggests that small amounts of targeted native-error data can be enough to adapt open LLMs for CGEC.

Key points

  • Dataset: 1,061 parallel pairs covering six native error types: redundant component, structural confusion, improper collocation, improper word order, improper logicality and missing component. Code and data are released at github.com/FreedomIntelligence/GrammarGPT.
  • Main result: On the validation data, GrammarGPT reaches word-level F0.5 32.56 and char-level F0.5 35.84, measured with the MaxMatch (M2) / ChERRANT scorer. S2S BART trained on Lang8 + HSK scores 17.99 and 17.59.
  • Data matters more than size for the baseline: S2S BART fine-tuned on only the authors' ~1k native-error set scores about the same as on 1.2M learner pairs (17.57 / 18.16). This indicates learner data transfers poorly to native errors.
  • Ablation: Without augmentation, the hybrid set scores 27.30 / 28.22 F0.5, and augmentation raises this to 32.56 / 35.84. Most of the gain comes from Recall. ChatGPT-generated data alone beats human-annotated data alone.
  • Training setup: phoenix-inst-chat-7b (about 20× more parameters than BART), AdamW, learning rate 2e-5 with a linear schedule, batch size 64, 3 epochs, max sequence length 256.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Science and Technology of ChinaHefei · China145 works · 114,409 citations🎓Soochow UniversitySuzhou · China69 works · 4,815 citations🎓Chinese University of Hong Kong, ShenzhenShenzhen · China56 works · 3,212 citations🎓Shenzhen Research Institute of Big DataShenzhen · China20 works · 2,016 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksGrammatical error correction & learner language · 162 works

Cite

Cite as Yaxin Fan; Feng Jiang; Peifeng Li; and Haizhou Li. 2023. GrammarGPT: Exploring Open-Source LLMs for Native Chinese Grammatical Error Correction with Supervised Fine-Tuning. arXiv:2307.13923.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.13923 · arXiv 2307.13923

@misc{li2023grammargpt,
  title = {{GrammarGPT: Exploring Open-Source LLMs for Native Chinese Grammatical Error Correction with Supervised Fine-Tuning}},
  author = {Yaxin Fan; Feng Jiang; Peifeng Li; and Haizhou Li},
  year = {2023},
  doi = {10.48550/arxiv.2307.13923},
  eprint = {2307.13923},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.13923}
}

References

3 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 1. Chen, F., Feng, Y.: Chain-of-Thought Prompt Distillation for Multimodal Named Entity and Multimodal Relation Extraction. ArXiv preprint arXiv:2306.14122 (2023) 2. Chen, Z., Jiang, F., Chen, J., Wang, T., Yu, F., Chen, G., Zhang, H., Liang, J., Zhang, C., Zhang, Z., et al.: Phoenix: Democratizing…
  2. Fan et al. 15. Rothe, S., Mallinson, J., Malmi, E., Krause, S., Severyn, A.: A Simple Recipe for Multilingual Grammatical Error Correction. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Proc… — Multitask Prompted Training Enables Zero-Shot Task Generalization
  3. Springer (2018) 29. Zhao, Z., Wang, H.: MaskGEC: Improving Neural Grammatical Error Correction via Dynamic Masking. Proceedings of the AAAI Conference on Artificial Intelligence 34(01), 1226–1233 (2020) find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fan-2023-grammargpt-exploring-open-source-llms/