📄 FEANEL: A Benchmark for Fine-Grained Error Analysis in K-12 English Writing

Summary

Feedback tools for school-age English learners usually stop at fixing errors (GEC) or giving holistic essay scores. Neither tells a student why a mistake happened or how serious it is. The authors define a task called Fine-grained Error Analysis for English Learners. A system receives an essay prompt, the student text, a corrected version and the list of edits between them. For each edit it must produce three things: an error category, a severity score from 1 to 5, and a written explanation of the problem and how to fix it. To support the task they built FEANEL, a set of 1,000 K-12 essays. Teaching experts corrected the essays and analysed every error using a new taxonomy organised around parts of speech. They then tested a broad set of current LLMs, both reasoning and standard models. The models struggle to categorise complex errors, and their explanations lack the precision and pedagogical care of human teachers. Results also change noticeably with prompt detail, the presence of examples, and the order of sub-tasks. The benchmark therefore measures something that existing GEC and essay-scoring evaluations miss.

Key points

• The dataset has 500 elementary-school essays (students aged 9–11, 3,003 errors) and 500 secondary-level essays (5,671 errors). The secondary essays come from the TECCL Corpus of Chinese EFL writing. The data is published on Hugging Face as Feanel/FEANEL.
• Average essay length is 48.2 tokens for elementary and 127.1 for secondary. Average edits per essay are 6.01 and 11.34 respectively.
• The taxonomy has 29 error types, mostly defined by part of speech. It avoids a catch-all "Other" category and has a priority scheme for choosing one main label when an edit contains several errors.
• Experts made minimal corrections following GEC conventions, and edits were extracted with CLEME. Annotators had to reach at least 90% accuracy on a trial round before formal annotation.
• Evaluated models include GPT-4o, o1, o3, o4-mini, Gemini-2.5-pro, DeepSeek-R1, Claude-3.7-Sonnet (with and without Thinking), Grok-3-Beta, Qwen-3, Llama-3 and Mistral-Small-3.1. By default, models output severity first, then type, then explanation.

Interesting ideas

• I1. The error taxonomy is built on part-of-speech categories, has no "Other" bucket, and uses an explicit priority rule for edits with several errors, so every case gets one dominant label while the explanation still covers the rest.
• I2. The task gives the system the gold correction and the extracted edits, so detecting an error and explaining it are evaluated separately.
• I3. Severity is a 1–5 score for how much an error damages sentence structure and meaning, not just whether it exists.
• I4. The paper treats the order of sub-tasks (severity → type → explanation) as a variable and shows that results depend on it.
• I5. Annotators had to pass a qualification trial at a fixed accuracy threshold, and those who failed were retrained or replaced.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Disagreements between witness parsers could be sorted with a similar exclusive, prioritised scheme (e.g. attachment vs. label vs. POS cause), which would make self-induced repair rules easier to target.
• I2 (back): Diffing input text against the text → graph → text reconstruction at the edit level would show exactly which spans the compressor loses, and each loss could then be explained and categorised.
• I3 (world): Parse or frame mistakes could be graded by how much they change the derived world state (entities, holdings, event order), so fixes that alter answers are prioritised.
• I4 (answer): The readers that solve school tests and word problems could be checked for order sensitivity in their steps, which would expose hidden dependencies between steps.
• I5 (learn): LLM teacher tiers could be admitted the same way, by first scoring them on a small gold set and only then trusting their silver labels.