Elamur Research⌕
☰
home / papers / 2025 / Artificial Intelligence Bias on English Language Learners in Automatic Scoring

Artificial Intelligence Bias on English Language Learners in Automatic Scoring

Paper · 2025 · Shuchen Guo, Yun Wang, Jichao Yu et al.

Summary

The paper asks whether automatic scoring of short written science answers treats English Language Learners (ELLs) unfairly, and in particular whether ELLs being underrepresented in training data is the cause. The authors used Grade 8 constructed-response items from the California Science Test (CAST), each scored 0–2. For every item they fine-tuned BERT four times: on ELL answers only, on non-ELL answers only, on a mix with the natural (unbalanced) ratio, and on a balanced mix. They downsampled so that the four variants could be compared fairly. Bias was measured as a difference in scoring accuracy, tested with Friedman tests. Disparity was measured by whether the model's mean score gap between ELLs and non-ELLs matched the gap in human scores. With plenty of ELL data (about 30,000 or about 1,000 per item) they found no bias and no distorted gaps. With only about 200 ELL answers, problems may appear. Their advice is to train on sufficiently large data that keeps real-world group proportions.

Key points

  • Data: 21 CAST Grade 8 items. 10 items have about 30,000 ELL responses, 5 have about 1,000, and 6 have about 200. ELLs are roughly 15% of all students.
  • Grouping: EL and ADEL were merged into "ELL". EO, RFEP and IFEP were merged into "non-ELL". The TBD and EPU categories were dropped as ambiguous.
  • There are four BERT training variants per item (ELL, Non-ELL, Unbalanced Mixed, Balanced Mixed), downsampled to the ELL sample size.
  • Two separate measures are used:
    • Accuracy (Acc) compared with Friedman tests, for bias.
    • The difference between human and AI Mean Score Gaps (MSGs), for disparity.
  • Training with the natural ELL/non-ELL ratio kept the gaps undistorted. Adding more ELL data narrowed the gap, but the authors could not tell whether this was fairer scoring or just lower non-ELL scores.

Interesting ideas

  • I1. The paper keeps two fairness questions apart: does the model score one group less accurately, and does it keep the gap between groups that human raters see. These two can fail independently, so checking both catches more problems.
  • I2. The ablation builds training sets from different group compositions (only A, only B, natural mix, balanced mix) but holds total size fixed. This isolates the effect of composition from the effect of quantity.
  • I3. The authors note that a shrinking gap might come from scores dropping for the majority group rather than real improvement. So a better-looking metric always needs a check of absolute per-group levels.
  • I4. Results are stratified by data scale (about 30,000 / 1,000 / 200 per group). Bias only showed up in the smallest tier, so averages over all items would have hidden it.
  • I5. The paper argues that short science answers are judged on content, not language polish, which reduces the effect of English proficiency on scoring.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): When grading school-course test answers, report per-group accuracy and also whether Mova's score gap between, e.g., learner-style and fluent answers matches human graders.
  • I2 (learn): Apply the same design to silver data from different sources or UD dialects, so a source's real effect is not confused with having more data.
  • I3 (learn): Add per-group absolute scores as a negative control in evaluation, so a "fairer" result caused by lowering everyone is flagged.
  • I4 (tree): Report parser LAS separately for small-data genres or converted treebanks, where weaknesses tend to hide behind large-corpus averages.
  • I5 (answer): Score student short answers by comparing meaning graphs rather than surface wording, so nonstandard but correct phrasing is not penalized.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🤝Educational Testing ServicePrinceton · United States27 works · 2,212 citations🎓University of GeorgiaAthens · United States17 works · 26,331 citations🎓George Mason UniversityFairfax · United States24 works · 1,288 citations🎓Nanjing Normal UniversityNanjing · China3 works · 195 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSafety Research · 543 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksGrammatical error correction & learner language · 156 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Artificial intelligence · 49 works

Cite

Cite as Shuchen Guo, Yun Wang, Jichao Yu, Xuansheng Wu, Bilgehan Ayik, Field M. Watts, Ehsan Latif, Ninghao Liu, Lei Liu and Xiaoming Zhai. 2025. Artificial Intelligence Bias on English Language Learners in Automatic Scoring. arXiv:2505.10643.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.10643 · arXiv 2505.10643

@misc{guo2025artificial,
  title = {{Artificial Intelligence Bias on English Language Learners in Automatic Scoring}},
  author = {Shuchen Guo, Yun Wang, Jichao Yu, Xuansheng Wu, Bilgehan Ayik, Field M. Watts, Ehsan Latif, Ninghao Liu, Lei Liu and Xiaoming Zhai},
  year = {2025},
  doi = {10.48550/arxiv.2505.10643},
  eprint = {2505.10643},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.10643}
}

References

29 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bablu, T.A., et al.: Machine learning in automated assessment: Enhancing objectivity and efficiency in educational evaluations. Journal of Advanced Computing Systems 4(7) (2024) find
  2. Cheuk, T.: Can AI be racist? Color-evasiveness in the application of machine learning to science assessments. Science Education 105(5), 825–836 (2021) find
  3. Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018), http://arxiv.org/abs/1810.04805 ↗
  4. Friedman, M.: The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association 32(200), 675–701 (1937). https://doi.org/10.1080/01621459.1937.10503522 ↗
  5. Gerard, L.F., Linn, M.C.: Using automated scores of student essays to support teacher guidance in classroom inquiry. Journal of Science Teacher Education 27(1), 111–129 (2016) find
  6. Grapin, S.E.: Multimodal Assessment of English Learners in Science: Expanding What "Counts" as Evidence of Content Learning. Ph.D. thesis, New York University, Steinhardt School of Culture, Education, and Human Development (2020), https://www.proquest.com/dissertations-theses/multimodal-assessment-english-learners-science/docview/2411140462/se-2 ↗
  7. Grapin, S.E., Llosa, L.: Multimodal tasks to assess english learners and their peers in science. Educational Assessment 27(1), 46–70 (2022). https://doi.org/10.1080/10627197.2022.2028139, https://doi.org/10.1080/10627197.2022.2028139, published online: January 17, 2022 ↗
  8. Guo, S., Latif, E., Zhou, Y., Huang, X., Zhai, X.: Using generative ai and multi-agents to provide automatic feedback. arXiv preprint arXiv:2411.07407 (2024) ↗
  9. Guo, S., Zheng, Y., Zhai, X.: Artificial intelligence in education research during 2013–2023: A review based on bibliometric analysis. Education and Information Technologies pp. 1–23 (2024) find
  10. Ifenthaler, D.: Automated essay scoring systems. In: Handbook of open, distance and digital education, pp. 1–15. Springer (2022) find
  11. Jeon, J., Lee, S., Coronel-Molina, S.M.: Rethinking ai: bias in speech-recognition chatbots for elt. ELT Journal 78(4), 435–445 (2024) find
  12. Jourdain, L., Sharma, S.V.: Language challenges in mathematics education for english language learners: A literature review. Waikato Journal of Education 21(2) (2016) find
  13. Latif, E., Zhai, X., Liu, L.: AI gender bias, disparities, and fairness: Does training data matter? arXiv preprint arXiv:2312.10833 (2023), https://arxiv.org/abs/2312.10833 ↗
  14. LeClair, C., Doll, B., Osborn, A., Jones, K.: English language learners’ and non– english language learners’ perceptions of the classroom environment. Psychology in the Schools 46(6), 568–577 (2009) find
  15. Lee, G.G., Latif, E., Wu, X., Liu, N., Zhai, X.: Applying large language models and chain-of-thought for automatic scoring. Computers and Education: Artificial Intelligence 6, 100213 (2024) find
  16. Lee, H.S., Pallant, A., Pryputniewicz, S., Lord, T., Mulholland, M., Liu, O.L.: Automated text scoring and real-time adjustable feedback: Supporting revision of scientific arguments involving uncertainty. Science Education 103(3), 590–622 (2019) S. Guo et al. find
  17. Li, C., Xing, W., Leite, W.: Using fair AI to predict students’ math learning outcomes in an online platform. Interactive Learning Environments 32(3), 1117–1136 (2024) find
  18. Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., Zou, J.: GPT detectors are biased against non-native English writers. Patterns 4(7) (2023)
  19. Nesbitt, J., Quesen, S.: Machines imitating humans: Scoring systems and the risk of bias (December 2023), https://www.wested.org/blog/machines-imitating-humans-scoring-systems-and-the-risk-of-bias/, accessed: February 19, 2025 ↗
  20. Reed, D.K., Mercer, S.H.: Potential scoring and predictive bias in interim and summative writing assessments. School Psychology 38(4), 215 (2023) find
  21. Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., Hall, P.: Towards a standard for identifying and managing bias in artificial intelligence, vol. 3. US Department of Commerce, National Institute of Standards and Technology (2022) find
  22. Weigle, S.C.: English language learners and automated scoring of essays: Critical considerations. Assessing Writing 18(1), 85–99 (2013) find
  23. WIDA Consortium: WIDA English Language Development Standards Framework, 2020 Edition. Tech. rep., Board of Regents of the University of Wisconsin-Madison (2020), https://wida.wisc.edu/resources/wida-english-language-development-standards-framework-2020-edition ↗
  24. Wilson, J., Huang, Y.: Validity of automated essay scores for elementary-age English language learners: Evidence of bias? Assessing Writing 60, 100815 (2024) find
  25. Zhai, X.: AI and machine learning for next generation science assessments. In: Machine Learning, Natural Language Processing, and Psychometrics. p. 201. IAP (2024) find
  26. Zhai, X., He, P., Krajcik, J.: Applying machine learning to automatically assess scientific models. Journal of Research in Science Teaching 59(10), 1765–1794 (2022) find
  27. Zhai, X., Krajcik, J.: Pseudo artificial intelligence bias p. 568–578 (2024), https://doi.org/10.1093/oso/9780198882077.003.0025 ↗
  28. Zhai, X., Nehm, R.H.: AI and formative assessment: The train has left the station. Journal of Research in Science Teaching 60(6), 1390–1398 (2023) find
  29. Zhai, X., Yin, Y., Pellegrino, J.W., Haudek, K.C., Shi, L.: Applying machine learning in science assessment: a systematic review. Studies in Science Education 56(1), 111–151 (2020) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/guo-2025-artificial-intelligence-bias-english-language/