📄 Training data generation for context-dependent rubric-based short answer grading

Summary

Automatically grading short open-ended answers needs a lot of domain-specific training data with rubrics and both correct and incorrect answers, but suitable data such as past OECD PISA items are confidential. The authors had access to the English portion of a PISA dataset under an NDA and wanted an open substitute for a subtask of the Sensemaking 2026 task at the ELOQUENT lab. They built a large synthetic candidate pool with gpt-oss-120b, using contexts drawn from openly available websites. The model generated questions, three-part grading rubrics and graded answers, and LLM judges filtered or iteratively refined each step. The confidential data entered the process only as simple derived text formats: relational features such as BAAI/bge-m3 similarities and n-gram overlaps between fields. These features were used to pick the synthetic samples closest to the reference. Three surrogate datasets resulted, and all of them resemble PISA more closely than plain prompted generation does. Only the most relaxed (third) selection method gave a statistically significant improvement when training graders. This matters because closed educational data can inform openly shareable resources without being disclosed.

Key points

• Each datapoint contains a context, a question, a rubric with full, partial and no credit sections, an answer, and a ternary label (FC=2, PC=1, NC=0).
• Candidate generation:
  • Web texts were split into chunks of 150–800 words and cleaned by an LLM.
  • The LLM produced 1–5 questions per context.
  • Rubrics went through a judge–critique loop, capped at 10 iterations, a cap that was never reached.
  • Five candidate answers were generated for each credit level, imitating hurried high-school writing with occasional typos.
• Selection signal: the features capture relations between fields, such as context–question cosine similarity via BAAI/bge-m3. Without parts of the original items, they reveal almost nothing about them.
• Third selection method: in a Wilcoxon signed-rank test, models trained on the selected subset beat random subsets (p = 0.00091), with an average accuracy gain of 0.0206.
• Second selection method: accuracy was slightly lower (0.58 vs 0.61), and its significance estimate was p = 0.715.
• Release: the code and the generated dataset are published on GitHub (sindelarp/training-data-generation-for-context-dependent-rubric-based-short-answer-grading).
• Limitation: most evaluations rely on a single training run.