📄 A Human-Computer Collaboration Approach to Improve Accuracy of an Automated English Scoring System

Summary

The paper tackles duplicated feedback in an automated scorer for English sentences written by Korean learners. The scorer analyses each answer in three independent phases: word, syntax, and mapping, where mapping compares the answer with reference answers. Each phase gives 0–2 points. Because the phases do not share information, one mistake is often reported twice under different error IDs, which distorts the score.

The authors define a redundant pair as two errors that overlap in position, come from different phases, and describe the same linguistic phenomenon. The first two conditions are checked automatically. Frequent co-occurring error-type pairs are then kept using a combined PMI and RFC score. English teachers sort the surviving types into three groups: always redundant, never redundant, and context-dependent. For the context-dependent group, a CART decision tree learns rules from features of the two errors, such as overlap, positions, and correction strings. When a pair is judged redundant, one error is dropped according to phase priority.

Redundancy detection reached 90.2% accuracy, and exact agreement with human raters rose by 2.6%. This shows that a small amount of targeted expert judgement can clean up the output of loosely coupled modules.

Key points

• The scorer has 70 error types across the word, syntax and mapping phases. Each error report has three fields: Error_ID, Error_Position, and Error_Correction_Info. Missing-word errors are positioned on the reference answer instead of the student sentence.
• Scoring 14,892 student answers produced 150,419 candidate pairs, grouped into 657 type pairs. Filtering with PMI·RFC ≥ k (k = 0.4) kept 111 types, covering 29,588 pairs.
• Expert teachers, each with 10+ years of experience, sorted the 111 types into three classes:
  • Class A: 20 types, confirmed redundant.
  • Class B: 47 types, not redundant.
  • Class C: 44 types, undecidable without more context.
• 12,178 Class C pairs were used to train CART. Its features were:
  • shared and non-shared length, normalised by the shorter sentence;
  • the correction info of each error;
  • the edit distance between the two correction strings;
  • each error's position and the difference between positions.
• On 200 held-out sentences, accuracy was 94.1% for A, 98.0% for B and 82.3% for C, or 90.2% overall. Exact agreement with human raters improved by 2.6%. When removing an error, syntax-phase errors are kept first and mapping-phase errors are discarded first.

Interesting ideas

• I1. Score co-occurring pairs of diagnostic types with PMI combined with relative frequency, so that reviewers see only the few type pairs that matter instead of every instance.
• I2. Have humans triage at the level of types into three outcomes: always yes, always no, and depends on context. Then train an interpretable learner only on the "depends" bucket.
• I3. When several independent modules report the same fact, merge the duplicates using an explicit priority that favours the module with the widest coverage (phrase-level over word-level).
• I4. Diagnose a learner answer by comparing it with a reference answer. Missing items are located on the reference side, and every other overlap test treats them as overlapping everything.
• I5. Report accuracy separately for classes decided by humans and by the model, and note that even human-labelled classes fall below 100% because upstream detection errors leak through.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Disagreements between witness parsers or repair-rule firings could be ranked this way before sending them to an LLM teacher tier, which cuts review cost.
• I2 (learn): Candidate self-induced repair rules could be split the same way, so that only context-dependent ones get a learned, explainable decision-tree condition.
• I3 (frames): When roles from VerbNet frames and from other cues overlap on one span, a declared priority order would resolve them while keeping the result explainable.
• I4 (answer): School-course test checkers could align student and reference trees and report omissions anchored to reference positions.
• I5 (learn): Evaluations should separate errors inherited from earlier stages from errors made by the component under test, much like a negative control.