Elamur Research⌕
☰
home / papers / 2026 / Individual and Combined Effects of English as a Second Language and Typos on LLM Performance

Individual and Combined Effects of English as a Second Language and Typos on LLM Performance

Paper · 2026 · Serena Liu; Yutong Yang; Prisha Sheth; Weixuan Dong; Mingjiao Diao; Xinru Zhu; Nikhil Banga; Oscar Melendez; Arnav Sharma; Minda Zhao; Marina Lin; and Mengyu Wang

Summary

Non-native users often talk to LLMs in English, and their prompts carry two kinds of deviation at once: learner grammar and typing mistakes. Earlier robustness studies looked at each in isolation. This paper measures them separately and together. Standard prompts are rewritten into ESL variants with Trans-EnV, using CEFR level A and eight L1 backgrounds. Keyboard-aware typos are then injected with MulTypo at three corruption rates. Four models (ChatGPT-5.2, DeepSeek-V3.2, Qwen2.5-7B-Instruct, LLaMA-3-8B-Instruct) are tested on three closed-ended and three open-ended benchmarks. Each benchmark is run under four matched conditions, and an interaction term checks whether the joint drop equals the sum of the single drops.

The combination usually hurts more than either factor alone, but the effects do not simply add up. On closed-ended tasks the joint drop tends to be subadditive and consistent. On open-ended tasks the picture is mixed, and sometimes the joint drop is superadditive. The practical lesson is that clean-English benchmarks likely overstate real-world quality.

Key points

  • Perturbations:
    • ESL rewriting uses Trans-EnV at CEFR A and covers eight L1 backgrounds: Spanish, Russian, Portuguese, Mandarin, Japanese, German, French and Arabic.
    • Typos come from MulTypo at rates r ∈ {0.1, 0.4, 0.7}, using insertion, deletion, substitution and transposition.
  • Benchmarks: GSM8K, MMLU and HellaSwag (closed-ended); MT-Bench, IFEval and AlpacaFarm (open-ended).
  • Scoring:
    • IFEval uses rule-based scoring.
    • MT-Bench and AlpacaFarm are judged by gpt-4o.
  • Interaction metric: δ = (ΔESL + ΔTypo,r) − ΔComb,r. A positive value means subadditive, a negative value means superadditive.
  • IFEval:
    • Typos dominate: ΔTypo at r=0.7 ranges from 0.13 to 0.26, while ΔESL ranges from 0.006 to 0.141.
    • The mean interaction is near zero (δ̄ = 0.010), but its sign differs by model: DeepSeek is superadditive, while ChatGPT-5.2 and Qwen are subadditive.
  • AlpacaFarm:
    • The interaction is weakly superadditive (δ̄ = −0.018).
    • LLaMA-3-8B has the largest combined drop (≈0.244).
  • MT-Bench:
    • Under severe combined noise, the drop is about the same on every scoring dimension (~0.75–0.77).
    • DeepSeek was the most robust model.
  • L1 effects: On closed-ended tasks, Arabic, Spanish and Portuguese variants hurt most. This ordering largely disappears on open-ended tasks.

Interesting ideas

  • I1. A 2×2 factorial design (clean / factor A / factor B / both) with an explicit additivity term shows whether two noise sources compound or overlap, which a single robustness score would hide.
  • I2. Rewriting inputs in a meaning-preserving way keeps gold labels fixed, so any score change is attributable to the input form alone.
  • I3. Graded, keyboard-aware typo injection gives a controllable noise dial instead of one fixed noisy test set.
  • I4. L1-specific grammar transformations, such as dropping articles or breaking subject-verb agreement, simulate the systematic errors of particular learner groups.
  • I5. Splitting the quality score into separate dimensions helps locate the damage: a uniform drop across all dimensions suggests the failure lies in understanding the input rather than in later reasoning.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Run parser and pipeline evaluations under typo-only, learner-grammar-only and combined inputs, and report δ on LAS alongside the raw drops.
  • I2 (answer): Rephrase school-test questions and maths word problems into learner English with the same gold answers, to check that readers rely on meaning rather than surface wording.
  • I3 (tokens): Stress the tokenizer, tagger and lemmatizer at several corruption rates to find where morphology breaks, and to motivate a spelling-normalization step.
  • I4 (tree): Produce learner-style variants of treebank sentences to test whether the parser still finds the right heads when determiners are missing or agreement is wrong.
  • I5 (frames): Check whether who-did-what-to-whom frames stay identical under perturbation, so that degradation can be traced to the stage where it first appears.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harvard UniversityCambridge · United States126 works · 45,130 citations

Also on the byline: Harvard University Press.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEnglish · 2,379 worksGrammatical error correction & learner language · 162 works

Cite

Cite as Serena Liu; Yutong Yang; Prisha Sheth; Weixuan Dong; Mingjiao Diao; Xinru Zhu; Nikhil Banga; Oscar Melendez; Arnav Sharma; Minda Zhao; Marina Lin; and Mengyu Wang. 2026. Individual and Combined Effects of English as a Second Language and Typos on LLM Performance. arXiv:2604.04723.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2604.04723 · arXiv 2604.04723

@misc{lin2026individual,
  title = {{Individual and Combined Effects of English as a Second Language and Typos on LLM Performance}},
  author = {Serena Liu; Yutong Yang; Prisha Sheth; Weixuan Dong; Mingjiao Diao; Xinru Zhu; Nikhil Banga; Oscar Melendez; Arnav Sharma; Minda Zhao; Marina Lin; and Mengyu Wang},
  year = {2026},
  doi = {10.48550/arxiv.2604.04723},
  eprint = {2604.04723},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2604.04723}
}

References

21 references, 15 of them with a card of ours. ↗ leads to the work.

  1. Amirhossein Aliakbarzadeh, Lucie Flek, and Akbar Karimi. Exploring robustness of multilingual LLMs on real-world noisy data, 2025. ↗
  2. Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations, 2018. ↗
  3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020. URL https://arxiv.org/abs/2005.14165. ↗
  4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. 2021. URL https://arxiv.org/abs/2110.14168. ↗
  5. DeepSeek-AI. Deepseek-V3 technical report. 2025. URL https://arxiv.org/abs/2412.19437. ↗
  6. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. 2023. URL https://arxiv.org/abs/2305.14387. ↗
  7. Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, and Mikel Artetxe. Do multilingual language models think better in english? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pp. 550–564, 2024. ↗
  8. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. ↗
  9. Jiyoung Lee, Seungho Kim, Jieun Han, Jun-Min Lee, Kitaek Kim, Alice Oh, and Edward Choi. Trans-EnV: A framework for evaluating the linguistic robustness of LLMs against english varieties. In NeurIPS 2025 Datasets and Benchmarks Track, 2025a.
  10. Jiyoung Lee, Seungho Kim, Jieun Han, Jun-Min Lee, Kitaek Kim, Alice Oh, and Edward Choi. Trans-env: A framework for evaluating the linguistic robustness of llms against english varieties. 2025b. URL https://arxiv.org/abs/2505.20875. ↗
  11. Fangru Lin, Shaoguang Mao, Emanuele La Malfa, Valentin Hofmann, Adrian de Wynter, Xun Wang, Si-Qing Chen, Michael J. Wooldridge, Janet B. Pierrehumbert, and Furu Wei. Assessing dialect fairness and robustness of large language models in reasoning tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6317–6342, 2025. ↗
  12. Yihong Liu, Raoyuan Zhao, Lena Altinger, Hinrich Schütze, and Michael A. Hedderich. Evaluating robustness of large language models against multilingual typographical errors. 2025. doi: 10.48550/arXiv.2510.09536. URL https://arxiv.org/abs/2510.09536. ↗
  13. Meta AI. The Llama 3 herd of models. 2024. URL https://arxiv.org/abs/2407.21783. ↗
  14. OpenAI. GPT-4 technical report. 2024. URL https://arxiv.org/abs/2303.08774. ↗
  15. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, 2022.
  16. Qwen Team. Qwen2.5 technical report. 2025. URL https://arxiv.org/abs/2412.15115. ↗
  17. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, 2019. ↗
  18. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. 2023a. URL https://arxiv.org/abs/2306.05685. ↗
  19. Lianmin Zheng et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023b. ↗
  20. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yu Luan, Dengyong Zhou, and Le Hou. Instruction-following evaluation for large language models. 2023. URL https://arxiv.org/abs/2311.07911. ↗
  21. Runtao Zhou, Guangya Wan, Saadia Gabriel, Sheng Li, Alexander J Gates, Maarten Sap, and Thomas Hartvigsen. Disparities in llm reasoning accuracy and explanations: A case study on african american english. arXiv preprint arXiv:2503.04099, 2025. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-individual-combined-effects-english-second/