Elamur Research⌕
☰
home / papers / 2026 / Does Continued Pretraining on a Learner Corpus Improve Automated Essay Scoring on English Proficiency Tests? Evidence from EFCAMDAT

Does Continued Pretraining on a Learner Corpus Improve Automated Essay Scoring on English Proficiency Tests? Evidence from EFCAMDAT

Paper · 2026 · Duy Anh Nguyen

Summary

Pretrained encoders used for Automated Essay Scoring (AES) learn mostly from native-written English, while proficiency-exam essays come from L2 learners. The study asks whether domain-adaptive continued pretraining (DAPT) on learner writing closes this gap. BERT, RoBERTa and DistilBERT were further pretrained with masked language modeling on the cleaned EFCAMDAT learner corpus. They were then fine-tuned for two settings: scoring within a single dataset (FCE or IELTS Task 2), and few-shot transfer from one test to the other. Each adapted model was compared with its original checkpoint.

Pretraining on the whole corpus gave inconsistent results across models, datasets and metrics. Lexical and syntactic comparisons showed that EFCAMDAT differs from the exam data in proficiency level, genre and communicative purpose. The author therefore repeated DAPT on proficiency-specific subsets. These often beat full-corpus DAPT and sometimes the unadapted baseline, though not across the board, and they did not reliably help cross-test transfer. The practical lesson is that matching the pretraining data to the target matters more than the sheer amount of in-domain text.

Key points

  • Pretraining data: the cleaned EFCAMDAT release (Shatz 2020).
    • About 45.39M word tokens.
    • CEFR levels A1–C1.
    • Writers of 198 nationalities.
    • Texts longer than 512 tokens were filtered out.
    • 15% of subword tokens were masked, following Stearns et al. (2024).
  • Downstream data:
    • CLC FCE: 1244 scripts. Response-level grades were mapped to a 0–20 scale using the Zhang et al. (2018) scheme.
    • IELTS Writing Scored Essays (Kaggle): 1274 essays, reduced to 710 Task 2 essays scored 0–9. Task 1 was dropped because the referenced charts were missing.
  • Transfer setup: a model is fine-tuned on the source test, then adapted with 50, 100 or 200 target samples, in both directions (IELTS→FCE and FCE→IELTS).
  • Main ablation result: proficiency-specific DAPT outperformed full-EFCAMDAT DAPT in 17 of 24 metric comparisons. The clearest gains over the baseline were for BERT on FCE, with partial gains for RoBERTa on FCE.
  • Limitations:
    • Only one random seed was used.
    • The IELTS set is small.
    • The transfer ablation covers BERT only.
    • Checkpoints are not released because of EFCAMDAT licensing.

Interesting ideas

  • I1. Before adapting, compare the candidate corpus with the target data on lexical overlap and syntactic complexity, and treat a good match as a filter rather than a guarantee of gains. This cheaply explains why "more in-domain text" can fail.
  • I2. Slicing a heterogeneous corpus by a meaningful attribute (here learner proficiency) and pretraining on the best-matching slice can beat using everything. Targeted subsets can be worth more than volume.
  • I3. Measuring transfer as a curve over a few target sizes (50/100/200) shows how quickly a model adapts to a new test format, not just whether it does.
  • I4. Items whose prompt depends on unavailable material (IELTS charts) were removed rather than scored blindly, which keeps labels meaningful.
  • I5. Summarising many metric-level comparisons as a win count (17/24), while noting the single-seed caveat, gives an honest picture of a conditional effect.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): before mixing a new silver-data source into training, profile its UD trees (depth, relation distribution) against the target test texts and flag mismatched sources.
  • I2 (tree): train or fine-tune gparse on treebank subsets chosen by genre or register closeness to the target, instead of always pooling every treebank.
  • I3 (answer): when adding a new school-course test style, report reader accuracy after 50, 100 and 200 seen items to see how quickly the reader adapts to it.
  • I4 (answer): detect questions that refer to missing figures or tables and mark them unanswerable instead of letting the reader guess.
  • I5 (learn): report repair-rule or training changes as win counts across metrics and seeds, so that small single-run differences are not over-read.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Duy Anh Nguyen. 2026. Does Continued Pretraining on a Learner Corpus Improve Automated Essay Scoring on English Proficiency Tests? Evidence from EFCAMDAT. arXiv:2605.25924.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2605.25924 · arXiv 2605.25924

@misc{nguyen2026does,
  title = {{Does Continued Pretraining on a Learner Corpus Improve Automated Essay Scoring on English Proficiency Tests? Evidence from EFCAMDAT}},
  author = {Duy Anh Nguyen},
  year = {2026},
  doi = {10.48550/arxiv.2605.25924},
  eprint = {2605.25924},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2605.25924}
}

References

27 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jill Burstein and Martin Chodorow. 1999. Automated essay scoring for nonnative English speakers. In Computer Mediated Language Assessment and Evaluation in Natural Language Processing.
  2. Yue Cao, Hanqi Jin, Xiaojun Wan, and Zhiwei Yu. 2020. Domain-adaptive neural automated essay scoring. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, page 1011–1020, New York, NY, USA. Association for Computing Machinery. find
  3. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4):213– 220. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  5. Gabrielle Gaudeau. 2025. Beyond the gold standard in analytic automated essay scoring. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 18–39, Vienna, Austria. Association for Computational Linguistics. ↗
  6. Jeroen Geertzen, Theodora Alexopoulou, and Anna Korhonen. 2014. Automatic linguistic annotation of large scale L2 databases: The EF-Cambridge open language database (EFCamDat). In Selected Proceedings of the 2012 Second Language Research Forum, pages 240–254, Somerville, MA. Cascadilla Proceedings Project. find
  7. Yi Gui. 2025. Develop a generic essay scorer for practice writing tests of statewide assessments. In Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Full Papers, pages 58–81, Wyndham Grand Pittsburgh, Downtown, Pittsburgh, Pennsylvania, United States. National Council on Measurement in Education (NCME). find
  8. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. Association for Computational Linguistics. ↗
  9. Yun Huang, Jeroen Geertzen, Rachel Baker, Anna Korhonen, and Theodora Alexopoulou. 2017. The EF Cambridge Open Language Database (EFCAMDAT): Information for users. https://ef-lab.mmll.cam.ac.uk/EFCAMDAT.html. Accessed 2026-02-19. ↗
  10. Zhiwei Jiang, Tianyi Gao, Yafeng Yin, Meng Liu, Hua Yu, Zifeng Cheng, and Qing Gu. 2023. Improving domain generalization for prompt-aware essay scoring via disentangled representation learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12456–12470, Toronto, Canada. Association for Computational Linguistics. find
  11. S. Kullback and R. A. Leibler. 1951. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79–86. ↗
  12. Pei Yee Liew and Ian K. T. Tan. 2025. On automated essay grading using large language models. In Proceedings of the 2024 8th International Conference on Computer Science and Artificial Intelligence, CSAI ’24, page 204–211, New York, NY, USA. Association for Computing Machinery. find
  13. J. Lin. 1991. Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory, 37(1):145–151. ↗
  14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. Preprint, arXiv:1907.11692. ↗
  15. Deryle Lonsdale and Diane Strong-Krause. 2003. Automated rating of ESL essays. In Proceedings of the HLT-NAACL 03 Workshop on Building Educational Applications Using Natural Language Processing, pages 61–67. find
  16. Jinghui Lu, Maeve Henchion, and Brian Mac Namee. 2020. Diverging divergences: Examining variants of Jensen Shannon divergence for corpus comparison tasks. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6740– 6744, Marseille, France. European Language Resources Association. find
  17. Xiaofei Lu. 2010. Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics, 15(4):474–496. ↗
  18. Peter Phandi, Kian Ming A. Chai, and Hwee Tou Ng. 2015. Flexible domain adaptation for automated essay scoring using correlated linear regression. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 431– 439, Lisbon, Portugal. Association for Computational Linguistics. find
  19. Zhuang Qiu, Peizhi Yan, and Zhenguang Cai. 2024. Large language models for second language English writing assessments: An exploratory comparison. In Proceedings of the 38th Pacific Asia Conference on Language, Information and Computation, pages 363– 370, Tokyo, Japan. Tokyo University of Foreign Studies.
  20. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. Preprint, arXiv:1910.01108. ↗
  21. Veronica Juliana Schmalz and Alessio Brutti. 2021. Automatic assessment of English CEFR levels using BERT embeddings. In Proceedings of the Eighth Italian Conference on Computational Linguistics (CLiC-it 2021), pages 295–301, Milan, Italy. CEUR Workshop Proceedings.
  22. Itamar Shatz. 2020. Refining and modifying the EF-CAMDAT: Lessons from creating a new corpus from an existing large-scale english learner language database. International Journal of Learner Corpus Research, 6(2):220–236. find
  23. Bernardo Stearns, Nicolas Ballier, Thomas Gaillat, Andrew Simpkin, and John P. McCrae. 2024. Evaluating the generalisation of an artificial learner. In Proceedings of the 13th Workshop on Natural Language Processing for Computer Assisted Language Learning, pages 199–208, Rennes, France. LiU Electronic Press. find
  24. Yongjie Wang, Chuang Wang, Ruobing Li, and Hui Lin. 2022. On the use of Bert for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3416–3425, Seattle, United States. Association for Computational Linguistics. find
  25. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 180–189, Portland, Oregon, USA. Association for Computational Linguistics.
  26. Helen Yannakoudakis and Ronan Cummins. 2015. Evaluating the performance of automated text scoring systems. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 213–223, Denver, Colorado. Association for Computational Linguistics. find
  27. Meng Zhang, Xie Chen, Ronan Cummins, Øistein E. Andersen, and Ted Briscoe. 2018. The effect of adding authorship knowledge in automated text scoring. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 305–314, New Orleans, Louisiana. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nguyen-2026-continued-pretraining-learner-corpus-improve/