📄 From Arabic Text to Puzzles: LLM-Driven Development of Arabic Educational Crosswords

Summary

The paper addresses the lack of Arabic educational tools. It presents a system that turns any Arabic educational text into crossword material: answer keywords plus matching clues. The authors built Arabic-Clue-Instruct from Arabic Wikipedia. They took each article's opening section, used its bolded terms as candidate answers, and filtered articles and keywords by popularity, relevance and length. GPT-4-Turbo then wrote several context-grounded clues per entry, guided by a carefully engineered prompt in the spirit of SELF-INSTRUCT. This data was used to fine-tune GPT-3.5-Turbo and Llama3-8B-Instruct.

ROUGE overlap with GPT-4-Turbo clues rose after fine-tuning. Human judges, using a five-grade scale, saw a much larger effect: the fine-tuned 8B Llama produced top-grade clues far more often than its base version. Teachers can then pick from the generated clues to assemble puzzles. The dataset and the fine-tuned Llama model are openly released. This gives a low-resource language a reusable resource and a working recipe for educational gamification.

Key points

• Roughly 211,000 Arabic Wikipedia articles were filtered down to 14,497 entries across 20 thematic categories; of these, 14,000 went to the training set and the remaining 497 were held out for testing.
• GPT-4-Turbo generated at least three clues per entry, for 54,196 unique clues in Arabic-Clue-Instruct.
• Filtering rules:
  • articles under 50 words were dropped;
  • keywords longer than two words were removed;
  • keywords had to fall within a 3–20 character range.
• Mean ROUGE-1 against GPT-4-Turbo clues:
  • GPT-3.5-Turbo: 0.0152 base, 0.0405 fine-tuned;
  • Llama3-8B-Instruct: 0.0063 base, 0.0354 fine-tuned.
• Human evaluation covered 200 contexts with 3 clues each. The share of grade "A" clues changed as follows:
  • Llama3-8B-Instruct: 36.02% base, 78.86% fine-tuned;
  • GPT-3.5-Turbo: 28.47% base, 54.33% fine-tuned.

Interesting ideas

• I1. Bolded terms in a Wikipedia lead section serve as free, structure-derived labels for an article's central concepts, so answer keywords need no manual annotation.
• I2. A strong proprietary LLM produced the training data, and a small open 8B model fine-tuned on it more than doubled its top-grade output. A modest local model can therefore inherit a narrow skill from a large teacher.
• I3. The authors reported ROUGE but explicitly distrusted it, since even good clues scored near zero. They relied on a graded human judgment instead, which shows surface overlap is a poor proxy for whether a generated text is valid.
• I4. Clues were built by weaving facts from the surrounding passage around a target keyword, so a clue amounts to a compact, context-grounded definition whose answer is a single term.
• I5. The pipeline over-generates candidate clues and leaves the final choice to educators. This human-in-the-loop design keeps automation useful even when individual outputs are imperfect.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): the bolded lead term could be taken as the canonical name of an article's main entity when linking Wikidata items into the knowledge graph, giving cheap anchor labels.
• I2 (learn): this supports an LLM teacher tier that writes task-specific examples once, after which a small local component is trained and checked on held-out items.
• I3 (learn): evaluations of generated or rephrased output should use meaning-level checks or graded human/LLM judgments, with surface metrics only as a secondary signal.
• I4 (back): the world graph could drive clue-style generation, describing an entity through its stored states and relations without naming it, which tests whether the graph holds enough meaning.
• I5 (answer): school-course test items generated from a text could be produced in surplus, each with its explainable source path, for a teacher or checker to select from.