面向语言学习者的跨语言反馈评语生成方法(Cross-Lingual Feedback Comment Generation for Language Learners)

Summary

The authors propose a new task: cross-lingual feedback comment generation (CLFCG). The system has to explain an error in a learner's English sentence in the learner's native language, here Chinese, rather than in English. This way, unfamiliar grammatical terms get in the way of understanding the explanation less. To support the task, the first English–Chinese dataset was built. It is based on the GenChal (FCG, INLG 2022) and ICNALE data. The data were cleaned, grammatical terms were unified with a custom mapping table, and the comments were translated with GPT-4-Turbo. Annotators then manually proofread the translations. The study focuses on preposition errors. The method is built as a pipeline. First, separate models predict corrections, evidence words and grammatical terms. This information is added to the input of mBART or mT5, which then generates the comment. The best variant (mBART with corrections and evidence words) reached a BLEU of 50.32 and a Precision of 62.84 in human evaluation. The 0.68B-parameter model clearly outperformed GPT-3.5 and fell only slightly behind GPT-4 in the few-shot setting. The work shows that a small fine-tuned model can provide accessible error explanations in educational systems.

Key points

• Data: the source ICNALE set contains 1,194 essays, 17,938 sentences and 10,463 comments. The shared-task portion is split into train/dev/test with 4,868 / 170 / 215 comments. The mapping table reduces 2,202 Japanese and 348 English grammatical terms to 1,745 Chinese ones.
• Pipeline: GECToR corrects the sentence, ERRANT extracts the edits within the marked span, an Explainable-GEC model (based on EXPECT) finds the evidence words, and RoBERTa as a multi-label classifier predicts the 10 most frequent terms.
• Human evaluation (0–2 scale, three raters, Kappa 0.8172). In the strict setting, the results are: mBART (Edit+EV) 62.84, GPT-3.5 37.21, GPT-4 69.77. In the lenient setting: 79.98, 56.98 and 89.53, respectively.
• Comment length: on average, in characters, mBART produces 56.26 and humans 59.72. Comments from GPT-3.5 (115.09) and GPT-4 (119.45) are almost twice as long.
• LLMs as evaluators: GPT-3.5 and GPT-4 tend to give inflated scores to their own outputs. When evaluation criteria and reference comments are added, LLM scores agree more closely with human ones.