Elamur Research⌕
☰
home / papers / 2026 / When the Target Domain Changes: AI-Mediated Construct Drift in High-Stakes English Language AssessmenW

When the Target Domain Changes: AI-Mediated Construct Drift in High-Stakes English Language AssessmenW

Paper · 2026 · Yi Gui

Summary

Gui examines a validity gap in high-stakes English tests such as TOEFL, IELTS, the Duolingo English Test (DET) and PTE. These tests treat performance without tools as evidence of readiness for English-medium study. Students now read, draft, translate and revise with generative AI, so the target language use (TLU) domain that scores are meant to generalize to is shifting. The paper is conceptual: it reports no new data, meta-analysis or systematic review. It draws on Messick's unified validity view, the 2014 Standards, Bachman and Palmer's Assessment Use Argument and Xi's work on AI-mediated competence, and it sorts recent literature into three unevenly populated layers. It names the resulting mismatch "AI-mediated construct drift," in which domain demands change while test constructs stay tied to an unaided model. As a remedy it proposes "bounded AI mediation," where every test taker uses the same institution-controlled assistant under fixed limits, with logged interactions. The argument matters because it reframes AI as a question of what scores mean, rather than only a matter of cheating or automated scoring.

Key points

  • The literature falls into three layers:
    • AI as operational infrastructure (scoring, feedback, proctoring, security), which is by far the largest.
    • AI as part of the authentic academic communication ecology, a smaller layer.
    • AI as a challenge to construct validity and extrapolation warrants, which is barely studied.
  • LLM rating of IELTS-style and L2 essays shows only partial agreement with human ratings, along with instability and subgroup fairness concerns. Matching existing ratings can carry over the biases already built into those ratings.
  • Bounded AI mediation is offered as a design principle for validation research, not as a finished test. It sits between banning AI outright and allowing it without limits.
  • Unaided scores remain meaningful as evidence of baseline linguistic resources. The paper calls for narrowing what scores are claimed to show, not for discarding the tests.
  • The proposed research agenda (Table 4) covers:
    • TLU analysis that reflects current AI use;
    • comparisons of no-AI, bounded-AI and broader-AI task conditions;
    • response-process studies using interaction logs;
    • scoring of skills such as hallucination detection;
    • DIF and other subgroup fairness analyses;
    • predictive validity, with richer outcomes than GPA;
    • score reports that state the assistance condition;
    • alignment with institutional AI policies.

Interesting ideas

  • I1. High agreement with an existing rater does not prove you measure the intended ability, because a system tuned to imitate a reference can inherit its blind spots.
  • I2. Have the same learners do parallel tasks with no help, bounded help and broad help, then compare structure, difficulty and outcomes.
  • I3. Separate help that supports comprehension from help that produces the answer, and design tasks where only the first kind is allowed.
  • I4. Log every assistant interaction, including what was asked, what was accepted or rejected, and how outputs were revised, so the process itself becomes evidence.
  • I5. Score reports should say what circumstances the performance was produced under and mark the point beyond which the evidence no longer holds.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver data agreed by several witness parsers should still be checked against independent external criteria and negative controls, since consensus may just replicate the witnesses' shared errors.
  • I2 (learn): Evaluate readers with the LLM teacher off, restricted and fully on, to measure how much accuracy truly comes from Mova's own pipeline.
  • I3 (answer): School-test and fable QA readers could allow an assistant to clarify word senses or references but never to supply answers, which keeps credit with the explainable chain.
  • I4 (learn): Record each LLM-teacher suggestion and whether repair rules or the parser adopted it, giving an auditable trace of where learned behaviour came from.
  • I5 (answer): Each answer could carry a tag saying which components contributed (pure pipeline vs. LLM fallback) and the domain where the reader was validated.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 worksDecision Sciences · 209 worksManagement Science and Operations Research · 108 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Yi Gui. 2026. When the Target Domain Changes: AI-Mediated Construct Drift in High-Stakes English Language AssessmenW. arXiv:2607.11213.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2607.11213 · arXiv 2607.11213

@misc{gui2026when,
  title = {{When the Target Domain Changes: AI-Mediated Construct Drift in High-Stakes English Language AssessmenW}},
  author = {Yi Gui},
  year = {2026},
  doi = {10.48550/arxiv.2607.11213},
  eprint = {2607.11213},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2607.11213}
}

References

22 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chapelle, C. A. (2001). Computer applications in second language acquisition: Foundations for teaching, testing, and research. Cambridge University Press. find
  2. Chapelle, C. A., Enright, M. K., & Jamieson, J. M. (Eds.). (2008). Building a validity argument for the Test of English as a Foreign Language. Routledge. find
  3. Chen, X., Zhou, Z., & Prado, M. (2025). ChatGPT-3.5 as an automatic scoring system and feedback provider in IELTS exams. International Journal of Assessment Tools in Education, 12(1), 62–77. https://doi.org/10.21449/ijate.1496193 ↗
  4. Chuang, P.-L., & Yan, X. (2025). Language assessment in the era of generative artificial intelligence: Opportunities, challenges, and future directions. System, 134, Article 103846. https://doi.org/10.1016/j.system.2025.103846 ↗
  5. Galaczi, E., & Pastorino-Campos, C. (2025). Ethical AI for language assessment: Principles, considerations, and emerging tensions. Annual Review of Applied Linguistics, 45, 294– 314. https://doi.org/10.1017/S0267190525100081 ↗
  6. Huang, Y., & Wilson, J. (2025). Evaluating LLM-based automated essay scoring: Accuracy, fairness, and validity. In Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con), Volume 2: Works in Progress (pp. 71-83). National Council on Measurement in Education. find
  7. International Test Commission & Association of Test Publishers. (2022). Guidelines for technology-based assessment. International Test Commission and Association of Test Publishers. find
  8. Isaacs, T., Hu, R., Trenkic, D., & Varga, J. (2023). Examining the predictive validity of the Duolingo English Test: Evidence from a major UK university. Language Testing, 40(3), 748–770. https://doi.org/10.1177/02655322231158550 ↗
  9. Isbell, D. R., Kremmel, B., & Kim, J. (2023). Remote proctoring in language testing: Implications for fairness and justice. Language Assessment Quarterly, 20(4–5), 469–487. https://doi.org/10.1080/15434303.2023.2288251 ↗
  10. Jin, Y., & Fan, J. (2023). Test-taker engagement in AI technology-mediated language assessment. Language Assessment Quarterly, 20(4–5), 488–500. https://doi.org/10.1080/15434303.2023.2291731 ↗
  11. Johnson, M. S. (2025). Responsible AI for measurement and learning: Principles and practices (Research Report No. RR-25-03). ETS Research Institute. find
  12. Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. https://doi.org/10.1111/jedm.12000 ↗
  13. Koraishi, O. (2024). The intersection of AI and language assessment: A study on the reliability of ChatGPT in grading IELTS Writing Task 2. Language Teaching Research Quarterly, 43, 22–42. https://doi.org/10.32038/ltrq.2024.43.02 ↗
  14. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13– 103). American Council on Education and Macmillan. find
  15. Perrotta, C., Knoch, U., Selwyn, N., & Mohammadi, S. (2025). Exploring the possibilities of integrating communicative AI into the IELTS test preparation process (IELTS Research Reports Online Series No. 2/25). British Council, IDP IELTS, and Cambridge University Press & Assessment. find
  16. Saricaoglu, A., & Bilki, Z. (2025). The capacity of ChatGPT-4 for L2 writing assessment: A closer look at accuracy, specificity, and relevance. Annual Review of Applied Linguistics, 45, 253–273. https://doi.org/10.1017/S0267190525100160 ↗
  17. Uyar, A. C., & Buyukahiska, D. (2025). Artificial intelligence as an automated essay scoring tool: A focus on ChatGPT. International Journal of Assessment Tools in Education, 12(1), 20–32. https://doi.org/10.21449/ijate.1517994 ↗
  18. Wilson, J., & Huang, Y. (2024). Validity of automated essay scores for elementary-age English language learners: Evidence of bias? Assessing Writing, 60, Article 100815. https://doi.org/10.1016/j.asw.2024.100815 ↗
  19. Xi, X. (2023). Advancing language assessment with AI and ML: Leaning into AI is inevitable, but can theory keep up? Language Assessment Quarterly, 20(4–5), 357–376. https://doi.org/10.1080/15434303.2023.2291488 ↗
  20. Xi, X. (2025). Revisiting communicative competence in the age of AI: Implications for large-scale testing. Annual Review of Applied Linguistics, 45, 200–221. https://doi.org/10.1017/S0267190525000078 ↗
  21. Xie, Q. (2024). Construct representation and predictive validity of integrated writing tasks: A study on the writing component of the Duolingo English Test. Assessing Writing, 61, Article 100846. https://doi.org/10.1016/j.asw.2024.100846 ↗
  22. Xu, J., Schmidt, E., Galaczi, E., & Somers, A. (2024). Automarking in language assessment: Key considerations for best practice. Cambridge University Press & Assessment. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gui-2026-target-domain-changes-ai-mediated/