łlinstruct: An Instruction-tuned model for English Language Proficiency Assessments

Summary

Writing items for English Language Proficiency Assessments (ELPA) by hand is slow, and general instruction-tuned LLMs often do poorly on narrow educational tasks. The authors built LL-INSTRUCT, a domain-specific instruction dataset and model for ELPA content generation. They started from 130 hand-written seed tuples, each with an instruction, input, output and explanation, derived from public sample test items. They then used GPT-4 in a self-instruct-style bootstrapping loop to expand these to about 70K tuples. Off-topic tuples were removed by a second LLM acting as a judge, along with keyword filters and ROUGE-L deduplication. Llama-3 8B was fine-tuned on 17K, 50K and 70K subsets and compared by human raters against Dolly-2, Mistral, base Llama-3 and GPT-3.5 on 200 unseen instructions. The 70K model gave the most test-ready outputs, on par with GPT-3.5, and much more usable explanations. This suggests a small, specialized 8B model can compete with a large general one. Expert revision is still needed for a sizeable share of outputs.

Key points

• The seed set has 130 tuples built by the authors from publicly available ELPA sample items covering reading, writing, speaking and listening. A single item is often split into several tuples, for example one for the correct indirect reply and one for three distractors.
• Each generation prompt holds four example tuples: three seeds plus one previously generated tuple. Seeds are mixed as two short and one long task, or the reverse, and GPT-4 returns ten new tuples per call.
• Filtering has three parts:
  • an LLM discriminator rejects non-ELPA items, such as trivia questions;
  • keyword filters remove tasks that mention images, videos, graphs or similar media;
  • a ROUGE-L similarity cap of 0.75 removes near-duplicate instructions.
• Llama-3 8B was fine-tuned into SFT-17K, SFT-50K and SFT-70K. SFT-70K and GPT-3.5 each exceeded 60% valid, ready outputs. Usable explanations reached 80.5% for SFT-70K versus 42% for GPT-3.5.
• About 20–30% of outputs still needed edits by subject matter experts. Figurative language, such as idioms, was the hardest task type, while grammar and vocabulary tasks were the easiest.

Interesting ideas

• I1. Each training tuple carries an explanation field that justifies the output, which makes generated data auditable and directly useful to people, not just to the trainer.
• I2. One exam item is decomposed into separate tasks: produce the correct answer, then produce wrong-but-plausible distractors, each with a reason it fails.
• I3. A second model acts as a gatekeeper that discards off-domain generations, layered with cheap keyword filters and a ROUGE-L near-duplicate threshold.
• I4. Prompts deliberately mix short and long task types, and seed instructions use varied phrasings for the same task, to fight the diversity collapse seen when training on synthetic text.
• I5. Error analysis found figurative language to be the consistent weak spot even after fine-tuning, while grammar-level tasks were easy.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Silver examples from the LLM teacher tiers could require a short rationale, so disagreements between witness parsers can be traced and turned into repair rules.
• I2 (answer): School-test readers could generate distractors along with a reason for each, giving negative controls that check whether the reader rejects options for the right reason.
• I3 (learn): The same three-layer filter fits LLM-generated training sentences: topic judge, banned-content keywords, then surface-similarity dedup to keep the silver corpus varied.
• I4 (tree): When asking a teacher LLM for synthetic sentences for gparse, mixing short and long constructions in each prompt could widen syntactic coverage beyond the teacher's favourite patterns.
• I5 (frames): Idioms and non-literal replies, such as the indirect-answer items, would make a natural dedicated test set for the implicature and irony layer in prag.