Elamur Research⌕
☰
home / papers / 2026 / Controlling Implicit Shortcut Reliance in L2 Spoken English Auto-markers

Controlling Implicit Shortcut Reliance in L2 Spoken English Auto-markers

Paper · 2026 · Shilin Gao, Mark J. F. Gales and Kate M. Knill

Summary

End-to-end graders for spoken English proficiency read raw audio or ASR transcripts directly. They can quietly come to depend on easy surface cues, such as how long a candidate keeps talking. This makes them gameable: a learner could pad an answer to raise the mark without speaking any better. Earlier fixes assume the suspect feature is an explicit, differentiable input to the model, which is not the case for fine-tuned encoders.

The authors add a training penalty on the Spearman rank correlation between predicted scores and an externally computed proxy feature. It sits next to a rank-based main loss and works only at the output, so the encoder needs no changes. They test it on the Speak & Improve 2025 Corpus across four task types. The graders are a ModernBERT transcript grader, with word count as the proxy, and a wav2vec 2.0 audio grader, with VAD time as the proxy. Zero-shot Qwen2.5-72B serves as a reference.

Both graders track their proxy more strongly than human scores do. The penalty brings this down to the human level or below while keeping scoring competitive, and unrelated features stay largely unchanged. The result is a tunable defence against score inflation.

Key points

  • A shortcut is defined as model predictions correlating with a proxy feature much more strongly than human labels do, measured with Spearman rank correlation. The test works for any model type and for both positive proxies (length) and negative ones (disfluency count).
  • Experiments use the Speak & Improve 2025 Corpus (four task types), a ModernBERT text grader on ASR output, and a wav2vec 2.0 audio grader. The shortcut proxies are word count and VAD time respectively.
  • Zero-shot Qwen2.5-72B is a non-fine-tuned baseline only. The paper notes zero-shot LLMs still trail fine-tuned graders and are costly to run at scale.
  • The paper offers two operating modes:
    • human-alignment: match the human raters' own reliance on the proxy;
    • malpractice-suppression: push reliance lower, accepting some loss in accuracy.
  • The motivating example comes from essay scoring: an essay with the same paragraphs repeated 37 times received the top machine score and the bottom human score.

Interesting ideas

  • I1. Penalise the rank correlation between a model's outputs and a cheap, externally computed proxy, rather than requiring that proxy to be a model input; this curbs hidden reliance without touching the architecture.
  • I2. Set the target for acceptable reliance by measuring how strongly the gold labels themselves correlate with the proxy, so the goal is "no more than humans," not zero.
  • I3. After suppressing one shortcut, check that correlations with unrelated features are unchanged, which shows the fix was targeted rather than a general degradation.
  • I4. Express both the task loss and the penalty in rank terms, so features with very different numeric scales can be penalised on the same footing.
  • I5. Treat a shortcut as an exploitable channel that an adversarial user would pad on purpose, not just as a source of noise.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): when training gparse, a penalty against correlation between attachment decisions and raw token distance could stop the parser from defaulting to "attach to nearest head" beyond what the treebank itself shows.
  • I2 (learn): when auditing silver data from witness parsers, compare each parser's correlation with surface cues (sentence length, punctuation) against the gold treebank's, flagging parsers that exceed the human baseline.
  • I3 (learn): add this as a negative control to evaluation, so that any repair rule or retraining aimed at one bias is also checked for side effects on unrelated metrics.
  • I4 (answer): the readers on fables and school tests could be checked for answer confidence tracking question–passage word overlap, a bag-of-words shortcut, with rank correlation making overlap, length and position cues comparable.
  • I5 (frames): probe the role labeller with inputs padded or reordered to exploit linear position (for example, "first noun = agent") and measure whether predicted roles follow position more than gold VerbNet roles do.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom256 works · 123,373 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,379 works

Cite

Cite as Shilin Gao, Mark J. F. Gales and Kate M. Knill. 2026. Controlling Implicit Shortcut Reliance in L2 Spoken English Auto-markers. arXiv:2607.16085.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2607.16085 · arXiv 2607.16085

@misc{gao2026controlling,
  title = {{Controlling Implicit Shortcut Reliance in L2 Spoken English Auto-markers}},
  author = {Shilin Gao, Mark J. F. Gales and Kate M. Knill},
  year = {2026},
  doi = {10.48550/arxiv.2607.16085},
  eprint = {2607.16085},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2607.16085}
}

References

41 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. NAACL-HLT, 2019, pp. 4171–4186. [Online]. Available: https://aclanthology.org/N19-1423/ ↗
  2. B. Warner, A. Chaffin, B. Clavié, O. Weller, O. Hallström, S. Taghadouini, A. Gallagher, R. Biswas, F. Ladhak, T. Aarsen, G. T. Adams, J. Howard, and I. Poli, “Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference,” in Proc. the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 2526–2547. [Online]. Available: https://aclanthology.org/2025.acl-long.127/ ↗
  3. A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 12 449–12 460. [Online]. Available: https://proceedings.neurips.cc/paper/2020/hash/92d1e1eb1cd6f9fba3227870bb6d7f07-Abstract.html ↗
  4. K. Hermann, H. Mobahi, T. Fel, and M. Mozer, “On the foundations of shortcut learning,” in International Conference on Learning Representations, vol. 2024, 2024, pp. 43 832–43 868. [Online]. Available: https://proceedings.iclr.cc/paper files/paper/2024/ hash/bf215fa7fe70a38c5e967e59c44a99d0-Abstract-Conference.html ↗
  5. D. E. Powers, J. C. Burstein, M. Chodorow, M. E. Fowles, and K. Kukich, “Stumping e-rater: challenging the validity of automated essay scoring,” Computers in Human Behavior, vol. 18, no. 2, pp. 103–134, 2002. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0747563201000528 ↗
  6. M. Chodorow and J. Burstein, “Beyond essay length: Evaluating e-rater®’s performance on TOEFL® essays,” Educational Testing Service, Tech. Rep. RR-04-04, 2004. [Online]. Available: https://www.ets.org/Media/Research/pdf/RR-04-04.pdf ↗
  7. Y. Attali, “Construct validity of e-rater in scoring TOEFL essays,” Educational Testing Service, Tech. Rep. RR-07-21, 2007. [Online]. Available: https://www.ets.org/research/policy research reports/publications/report/2007/hsmn.html ↗
  8. J. Fleckenstein, J. Meyer, T. Jansen, S. Keller, and O. Köller, “Is a long essay always a good essay? The effect of text length on writing assessment,” Frontiers in Psychology, vol. 11, p. 562462, 2020. [Online]. Available: https://doi.org/10.3389/fpsyg.2020.562462 ↗
  9. S. Jeon and M. Strube, “Countering the influence of essay length in neural essay scoring,” in Proc. the Second Workshop on Simple and Efficient Natural Language Processing, N. S. Moosavi, I. Gurevych, A. Fan, T. Wolf, Y. Hou, A. Marasović, and S. Ravi, Eds. Virtual: Association for Computational Linguistics, Nov. 2021, pp. 32–38. [Online]. Available: https://aclanthology.org/2021.sustainlp-1.4/ ↗
  10. Y. Wang, R. Hu, and Z. Zhao, “Beyond agreement: Diagnosing the rationale alignment of automated essay scoring methods based on linguistically-informed counterfactuals,” in Findings of the Association for Computational Linguistics: EMNLP 2024, Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, Eds. Miami, Florida, USA: Association for Computational Linguistics, Nov. 2024, pp. 8906–8925. [Online]. Available: https://aclanthology.org/2024.findings-emnlp.520/ ↗
  11. Y. Farag, H. Yannakoudakis, and T. Briscoe, “Neural automated essay scoring and coherence modeling for adversarially crafted input,” in Proc. the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), M. Walker, H. Ji, and A. Stent, Eds. New Orleans, Louisiana: Association for Computational Linguistics, Jun. 2018, pp. 263–271. [Online]. Available: https://aclanthology.org/N18-1024/ ↗
  12. A. Chakravarty, M. Brenchley, T. Breakspear, I. Lewin, and Y. Huang, “Enhancing marker scoring accuracy through ordinal confidence modelling in educational assessments,” in Proc. the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), 2025, pp. 1498–1507. [Online]. Available: https://aclanthology.org/2025.acl-industry.106/ ↗
  13. Y. Wang, M. J. Gales, K. Knill, K. Kyriakopoulos, A. Malinin, R. C. van Dalen, and M. Rashid, “Towards automatic assessment of spontaneous spoken english,” Speech Communication, vol. 104, pp. 47–56, 2018. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167639317304545 ↗
  14. P. Bamdev, M. S. Grover, Y. K. Singla, P. Vafaee, M. Hama, and R. R. Shah, “Automated speech scoring system under the lens: evaluating and interpreting the linguistic cues for language proficiency,” International Journal of Artificial Intelligence in Education, vol. 33, no. 1, pp. 119–154, 2023. [Online]. Available: https://link.springer.com/content/pdf/10.1007/s40593-022-00291-5.pdf ↗
  15. Z. L. Handley and H. Wang, “What do the measures of utterance fluency employed in automatic speech evaluation (ase) tell us about oral proficiency?” Language Assessment Quarterly, vol. 21, no. 1, pp. 3–32, 2024. [Online]. Available: https://www.tandfonline.com/doi/pdf/10.1080/15434303.2023.2283839 ↗
  16. A. S. Ross, M. C. Hughes, and F. Doshi-Velez, “Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations,” in Proc. the Twenty-Sixth International Joint Conference on Artificial Intelligence, 2017, pp. 2662–2670. [Online]. Available: https://doi.org/10.24963/ijcai.2017/371 ↗
  17. X. Shao, A. Skryagin, W. Stammer, P. Schramowski, and K. Kersting, “Right for Better Reasons: Training Differentiable Models by Constraining their Influence Functions.” in Proc. the AAAI Conference on Artificial Intelligence, vol. 35, no. 11, 2021, pp. 9533–9540. [Online]. Available: https://doi.org/10.1609/aaai.v35i11.17148 ↗
  18. M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” in International conference on machine learning. PMLR, 2017, pp. 3319–3328. [Online]. Available: https://proceedings.mlr.press/v70/sundararajan17a.html ↗
  19. Y. Kumar, S. Parekh, S. Singh, J. J. Li, R. R. Shah, and C. Chen, “Automatic essay scoring systems are both overstable and oversensitive: Explaining why and proposing defenses,” Dialogue & Discourse, vol. 14, pp. 1–33, Apr. 2023. [Online]. Available: https://aclanthology.org/2023.dnd-14.7/ ↗
  20. K. Knill, D. Nicholls, M. J. Gales, M. Qian, and P. Stroinski, “The Speak & Improve Corpus 2025: an L2 English Speech Corpus for Language Assessment and Feedback,” 2025. [Online]. Available: https://doi.org/10.17863/CAM.114333 ↗
  21. ——, “Introducing the speak & improve corpus 2025: an l2 english speech corpus for language assessment and feedback,” in 10th Workshop on Speech and Language Technology in Education (SLaTE), 2025, pp. 167–171. [Online]. Available: https://www.isca-archive.org/slate 2025/ knill25 slate.html ↗
  22. V. Raina, M. J. Gales, and K. M. Knill, “Universal Adversarial Attacks on Spoken Language Assessment Systems,” in Interspeech 2020, 2020, pp. 3855–3859. find
  23. M. Qian, K. M. Knill, S. Bannò, S. Tang, P. Karanasou, M. J. Gales, and D. Nicholls, “Speak & Improve Challenge 2025,” in 10th Workshop on Speech and Language Technology in Education (SLaTE), 2025, pp. 41–45. find
  24. S. Bannò and M. Matassoni, “Proficiency assessment of L2 spoken English using wav2vec 2.0,” in 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 1088–1095. [Online]. Available: https://ieeexplore.ieee.org/document/10023019 ↗
  25. R. Ma, M. Qian, S. Tang, S. Bannò, K. Knill, and M. J. Gales, “Assessment of L2 oral proficiency using speech large language models,” in Proc. Interspeech 2025, 2025, pp. 5078–5082. [Online]. Available: https://www.isca-archive.org/interspeech 2025/ma25b interspeech.pdf ↗
  26. S. Bannò, K. Knill, M. Matassoni, V. Raina, and M. J. Gales, “Assessment of L2 oral proficiency using self-supervised speech representation learning,” in Proc. 9th Workshop on Speech and Language Technology in Education (SLaTE), 2023, pp. 126–130. [Online]. Available: https://www.isca-archive.org/slate 2023/banno23 slate.html ↗
  27. X. Xi, D. Higgins, K. Zechner, and D. M. Williamson, “Automated scoring of spontaneous speech using SpeechRater v1.0,” Educational Testing Service, Princeton, NJ, Tech. Rep. ETS RR-08-62, 2008. find
  28. K. Zechner, D. Higgins, X. Xi, and D. M. Williamson, “Automatic scoring of non-native spontaneous speech in tests of spoken english,” Speech communication, vol. 51, no. 10, pp. 883–895, 2009. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167639309000703 ↗
  29. D. Higgins, X. Xi, K. Zechner, and D. Williamson, “A three-stage approach to the automated scoring of spontaneous spoken responses,” Computer Speech and Language, vol. 25, no. 2, pp. 282–306, 2011. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0885230810000458 ↗
  30. L. Chen, K. Zechner, S.-Y. Yoon, K. Evanini, X. Wang, A. Loukina, J. Tao, L. Davis, C. M. Lee, M. Ma, R. Mundkowsky, C. Lu, C. W. Leong, and B. Gyawali, “Automated scoring of nonnative speech using the SpeechRater SM v. 5.0 engine,” ETS Research Report Series, vol. 2018, no. 1, pp. 1–31, 2018. [Online]. Available: https://doi.org/10.1002/ets2.12198 ↗
  31. J. Xu, E. Jones, V. Laxton, and E. Galaczi, “Assessing L2 English speaking using automated scoring technology: examining automarker reliability,” Assessment in Education: Principles, Policy & Practice, vol. 28, no. 4, pp. 411–436, 2021. ↗
  32. S. Gao, M. J. Gales, and J. Xu, “Detecting aberrant responses in automated L2 spoken English assessment,” in Exploring Artificial Intelligence in Applied Linguistics, C. A. Chapelle, G. H. Beckett, and J. Ranalli, Eds. Iowa State University Digital Press, 2024, pp. 96–117. [Online]. Available: https://www.iastatedigitalpress.com/plugins/books/154/chapter/1231 ↗
  33. K. Knill and M. J. Gales, “Building an auto-marker for assessing spontaneous L2 English speech,” in Language Test Validation in a Digital Age, ser. Studies in Language Testing, G. Yu and J. Xu, Eds. Cambridge University Press & Assessment, 2024, vol. 52, pp. 309–334. find
  34. I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” arXiv preprint arXiv:2004.05150, 2020. [Online]. Available: https://arxiv.org/abs/2004.05150 ↗
  35. S. W. McKnight, A. Civelekoglu, M. J. Gales, S. Bannò, A. Liusie, and K. Knill, “Automatic assessment of conversational speaking tests,” in Proc. SLaTE 2023, 2023, pp. 99–103. [Online]. Available: https://www.isca-archive.org/slate 2023/mcknight23 slate.html ↗
  36. S. Bannò, R. Ma, M. Qian, S. Tang, K. Knill, and M. J. Gales, “Natural Language-based Assessment of L2 Oral Proficiency using LLMs,” arXiv preprint arXiv:2507.10200, 2025. [Online]. Available: https://arxiv.org/abs/2507.10200 ↗
  37. L. Zhao, Q. Liu, L. Yue, W. Chen, L. Chen, R. Sun, and C. Song, “Comi: Correct and mitigate shortcut learning behavior in deep neural networks,” in Proc. the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, ser. SIGIR ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 218–228. [Online]. Available: https://doi.org/10.1145/3626772.3657729 ↗
  38. H. Chung, C. Han, Y. Choi, S. Ji, J. Kim, E.-J. Holden, and K. Han, “Filora: Focus-and-ignore lora for controllable feature reliance,” arXiv preprint arXiv:2602.02060, 2026. [Online]. Available: https://arxiv.org/abs/2602.02060 ↗
  39. Y. Fathullah, P. Radmard, A. Liusie, and M. J. Gales, “Who Needs Decoders? Efficient Estimation of Sequence-Level Attributes with Proxies,” in Proc. the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), Mar. 2024, pp. 1478–1496. [Online]. Available: https://aclanthology.org/2024.eacl-long.89/ ↗
  40. M. Blondel, O. Teboul, Q. Berthet, and J. Djolonga, “Fast Differentiable Sorting and Ranking,” in Proc. the 37th International Conference on Machine Learning, vol. 119. PMLR, 2020, pp. 950–959. [Online]. Available: https://proceedings.mlr.press/v119/blondel20a.html ↗
  41. Qwen Team, “Qwen2.5 technical report,” Alibaba Group, Tech. Rep., 2025. [Online]. Available: https://arxiv.org/abs/2412.15115 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gao-2026-controlling-implicit-shortcut-reliance-l2/