📄 Using k-Shot Prompting with Large k for the Automated Scoring of a German Written Elicited Imitation Test

Summary

The authors examine whether a large language model can automatically score a German Written Elicited Imitation Test (WEIT). In this test, learners of German as a foreign language briefly see a sentence and then type it from memory. Each response receives a score from 0 to 4 based on a rubric that separates typos, meaning-preserving changes, grammatical errors and loss of content. Using the WEIT dataset from Chifligarov et al. (2025), they prompted GPT-4o with up to 700 scored stimulus-response pairs but did not include the rubric. They studied three sources of variability: the number of shots k, which training examples were sampled, and repeated runs of the same prompt. Larger k raised accuracy significantly. Sample choice also mattered, though its effect faded at high k. Accuracy varied between identical runs, while QWK stayed more stable. The k-shot setup beat a rubric-only zero-shot GPT baseline but did not surpass earlier rule-based or BERT-based scorers. This suggests that LLM variability currently limits practical use, and that combining different model types is a promising direction.

Key points

• Data: The dataset has 3,900 stimulus-response pairs (195 participants × 20 sentences), split as in Chifligarov et al. (2025). The training pool contains 3,260 pairs, and the development set has 125 pairs balanced across scores.
• Experiment 1 design: k ranged from 10 to 700 across 18 values. Each k was tested with 2 samples and 4 runs, giving 144 GPT-4o runs at temperature 0. Experiment 2 focused on small k (≤ 25). The evaluation metrics were accuracy and quadratically weighted kappa (QWK).
• Comparison with earlier models: Differences in correctly scored items between GPT-k-shot and the rule-based or BERT-based (DistilBERT) models were not significant. Only the BERT-based model dropped significantly on unseen stimuli (χ²(1) = 11.31, p < .001).
• Error patterns by score: The rule-based model did best at the scale edges (scores 0, 1, 4), and BERT did best on score 2. GPT-k-shot had the highest F1 on score 3. GPT models tended to over-score grammatical errors (2 predicted as 3), whereas BERT and rule-based models tended to under-score (3 predicted as 2).
• Availability: Code and data are released at gitlab.ruhr-uni-bochum.de/vamos-cl/german-weit-k-shot-prompting.