Elamur Research⌕
☰
home / papers / 2025 / Automatic Detection of Inauthentic Templated Responses in English Language Assessments

Automatic Detection of Inauthentic Templated Responses in English Language Assessments

Paper · 2025 · Yashad Samant, Lee Becker, Scott Hellman et al.

Summary

Test takers in high-stakes English proficiency exams sometimes submit memorized essay "templates": prewritten texts full of advanced vocabulary and discourse markers, with gaps they fill with words from the prompt. The goal is to inflate scores from automated essay scoring. The authors, working at Pearson, define this as the AuDITR task (automated detection of inauthentic, templated responses).

They annotated essays from the Pearson Test of English with a three-level templating scale, then collapsed it to a binary target: heavy templating versus everything else. Instead of fine-tuning BERT or using n-gram/TF-IDF classifiers, which tend to memorize the templates seen in training, they built a small set of interpretable features. These features measure how much of a response is not covered by fuzzy matches against known sub-templates or against the prompt text. A random forest on these features reached perfect precision on the test set with moderate recall.

The paper also reports that detection rates decay after each release as test takers switch templates. This makes template-list updates and drift monitoring a core part of operating such a system.

Key points

  • The corpus has 319 PTE essay responses, labeled by subject matter experts as None, Low or High templating. It was split into 213 training and 106 test responses, and the model target is High versus None/Low.
  • In a separate binary annotation effort over 4000 responses, raters agreed exactly 77.25% of the time, with Kappa 0.446. This shows how subjective partial templating is.
  • The model uses six features: the count and percentage of response tokens not matching sub-templates, not matching the prompt, and matching neither. The last of these is the "authentic" tokens. Matches are found with sliding n-gram windows and an efficient Levenshtein comparison.
  • A random forest (grid search over depth, max features and number of trees) was run at a threshold of 0.8.
    • Train: P 95.1, R 70.9, F1 81.2.
    • Test: P 100, R 58.2, F1 73.6, with no false positives.
  • After deployment, detection rates jumped at each model release and then decayed as new templates spread online. The authors call this adaptive adversarial drift and compare it to spam and fraud detection.

Interesting ideas

  • I1. Score a text by its leftover: count the tokens not explained by known external sources (templates, the prompt), so the feature directly expresses "how much did the writer produce themselves" and stays readable.
  • I2. Keep the knowledge the detector relies on (the template list) as an input configuration rather than baked into model weights, so new patterns can be added without retraining.
  • I3. Break long fixed texts into smaller sub-segments and match each with fuzzy windowed edit distance, which catches pieces stitched together from several sources and tolerates small edits.
  • I4. Annotate on a graded scale but train a binary decision at a tunable threshold, chosen for high precision, and hand-check a sample of flagged items before acting.
  • I5. Treat a falling detection rate on fresh data as a signal that inputs have shifted, not that the problem has disappeared.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): when grading a student's free-text answer, compute the share of tokens not copied from the question or source passage, to flag answers that merely echo the task.
  • I2 (learn): store self-induced repair rules and pattern lists as editable data consulted at run time, so fixes ship instantly and each decision can cite the exact entry that fired.
  • I3 (tokens): match a lexicon of multiword expressions and formulaic phrases with Levenshtein-tolerant n-gram windows, so slightly misspelled or recombined fixed phrases still get recognized as units.
  • I4 (learn): label silver data with graded confidence levels from witness-parser agreement, then pick precision-first cut-offs per use and audit what gets accepted.
  • I5 (learn): track over time how often specific rules or frames fire on new corpora, and treat sudden drops as drift alarms that trigger re-mining of patterns.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksInformation Systems · 1,730 worksSafety Research · 626 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,379 works

Cite

Cite as Yashad Samant, Lee Becker, Scott Hellman, Bradley Behan, Sarah Hughes and Joshua Southerland. 2025. Automatic Detection of Inauthentic Templated Responses in English Language Assessments. arXiv:2509.08355.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2509.08355 · arXiv 2509.08355

@misc{samant2025automatic,
  title = {{Automatic Detection of Inauthentic Templated Responses in English Language Assessments}},
  author = {Yashad Samant, Lee Becker, Scott Hellman, Bradley Behan, Sarah Hughes and Joshua Southerland},
  year = {2025},
  doi = {10.48550/arxiv.2509.08355},
  eprint = {2509.08355},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2509.08355}
}

References

19 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Burkhardt, A., & Lottridge, S. (2013). Examining the impact of training samples on identifying off-topic responses in automated essay scoring. find
  2. Cahill, A., Chodorow, M., & Flor, M. (2018). Developing an e-rater advisory to detect babel-generated essays. The Journal of Writing Analytics, 2 (1), 203–224. https://doi.org/10.37514/JWA-J.2018.2.1.08 ↗
  3. Dalvi, N., Domingos, P., Mausam, Sanghai, S., & Verma, D. (2004). Adversarial classification. Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, 99–108. https://doi.org/10.1145/1014052.1014066 ↗
  4. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019, June). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 4171–4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423 ↗
  5. Grosshans, M., Wenzel, S., & Morik, K. (2013). Detection of adversarial concept drift with drift-sensitive learners. European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN). find
  6. Hendersen, G., & Andrade, A. (2019). Gamification of automated scoring engines. find
  7. Higgins, D., Burstein, J., & Attali, Y. (2006). Identifying off-topic student essays without topic-specific training data. Natural Language Engineering, 12 (2), 145–159. https://doi.org/10.1017/S1351324906004189 ↗
  8. Higgins, D., & Heilman, M. (2014). Managing what we can measure: Quantifying the susceptibility of automated scoring systems to gaming behavior. Educational Measurement: Issues and Practice, 33 (3), 36–46. https://doi.org/10.1111/emip.12036 ↗
  9. Hughes, S. R., & Clesham, R. (2024, August). Maintaining test integrity through integration of human judgement and AI systems. Pearson. find
  10. Kim, J. (2023). Online high-stakes writing test preparation at popular cram schools in korea. https://doi.org/10.2139/ssrn.4353938 ↗
  11. Kim, S. (2021). Prepping for the TOEFL iBT writing test, gangnam style. Assessing Writing, 49, 100544. https://doi.org/10.1016/j.asw.2021.100544 ↗
  12. Levenshtein, V. I. (1966). Binary codes capable of correcting deletions, insertions, and reversals. Soviet physics doklady, 10 (8), 707–710. find
  13. Lottridge, S., Godek, B., & Patel, M. (2020). Comparing the Robustness of Deep Learning and Classical Automated Scoring Approaches to Gaming Strategies. Annual meeting of the National Council on Measurement in Education. find
  14. Perelman, L. (2020). The BABEL generator and e-rater: 21st century writing constructs and automated essay scoring (AES). Journal of Writing Assessment, 13 (1). find
  15. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems, 30.
  16. Xi, X., Mollaun, P., & Davis, L. (2024). Use of formulaic language in TOEFL iBT speaking responses. In Challenges and innovations in speaking assessment (1st). ↗
  17. Zhang, M., Chen, J., & Ruan, C. (2016). Evaluating the advisory flags and machine scoring difficulty in the e-rater ® automated scoring engine. ↗
  18. ETS Research Report Series, 2016 (2), 1–14. https://doi.org/10.1002/ets2.12116 ↗
  19. Zou, Y., Zheng, Y., & Wang, J. (2024). A case study of washback and test preparation of the new version of PTE academic. International Journal of Language Testing. https://doi.org/10.22034/ijlt.2024.447992.1337 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/samant-2025-automatic-detection-inauthentic-templated-responses/