Elamur Research⌕
☰
home / papers / 2021 / Automatic Assessment of English CEFR Levels Using BERT Embeddings

Automatic Assessment of English CEFR Levels Using BERT Embeddings

Paper · 2021 · Veronica Juliana Schmalz and Alessio Brutti

Summary

The paper asks whether pre-trained BERT representations can sort English learners' written exams into CEFR levels automatically. It also tests whether supplying a corrected version of each text improves the prediction. The corrections come either from human evaluators or from the LanguageTool checker. The authors keep BERT-base-uncased frozen and train only a small feed-forward classifier on its embeddings. They compare three input setups:

  • the learner text alone;
  • the learner text joined with its correction before encoding;
  • a two-streams variant that encodes both texts separately and averages the embeddings.

Experiments use EFCAMDAT (up to 100K essays, levels A1–C1) and the much smaller CLC-FCE (four levels). On EFCAMDAT, accuracy is very high, close to 98%, and it levels off at around 50K training essays. Concatenating corrections gives small gains: automatic corrections help most with little data, and manual ones help most with lots of data. On CLC-FCE, accuracy falls to roughly 61% and the differences between setups are small. The work shows that cheap frozen embeddings plus correction signals can support scalable proficiency assessment when enough labelled exams exist.

Key points

  • The model is a frozen BERT-base-uncased encoder followed by a classifier. The classifier has a Dense 768 layer and a Dense 128 layer, both with ReLU and dropout 0.2, and a softmax output. It was trained with Adam and batch size 32. Maximum input length was 450 tokens for EFCAMDAT and 512 for CLC-FCE.
  • EFCAMDAT contains 1,180,310 essays, of which a 100K subset was used. The text-only baseline scored 95.2%, 97.1% and 97.4% at 10K, 50K and 100K training essays. The best result was 97.7%, from concatenating manual corrections at 100K.
  • In EFCAMDAT, errors per word drop as CEFR level rises. LanguageTool flags more errors than humans do, especially at low levels.
  • CLC-FCE has 2,469 texts, with scores mapped to A2–C1. Results were averaged over several random seeds:
    • text-only: 61.5% ± 2.0;
    • manual corrections: 60.7%;
    • automatic corrections: 61.7%;
    • two-streams: 61.5%.
  • Averaging the two embeddings (two-streams) did slightly worse than concatenation. The authors suspect the averaged vector falls between the two texts and loses discriminative power.

Interesting ideas

  • I1. Instead of labelling error types, the system is given the original text plus its corrected version and left to infer the errors from the contrast; this avoids costly error taxonomies.
  • I2. A plain length-normalized error rate already tracks proficiency level monotonically, which makes it a transparent, cheap baseline feature for grading.
  • I3. Noisier but consistent automatic corrections beat more accurate yet variable human ones when data is scarce, and the order flips as data grows.
  • I4. Merging two representations by averaging blurred the signal, while keeping both inputs side by side preserved it.
  • I5. The evaluation draws learning-curve subsets that keep the class distribution fixed, and reports mean ± std over seeds when data is small, so that tiny gaps are not over-read.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): parse both a learner sentence and its LanguageTool-corrected form, then diff the two UD trees to locate and explain the exact structural fault.
  • I2 (answer): when checking student answers in school-course tests, report errors per word alongside the verdict as an explainable fluency indicator.
  • I3 (learn): prefer consistent silver annotations from witness parsers for small training sets, and add heterogeneous LLM-teacher labels only once the volume can absorb their variability.
  • I4 (learn): when combining outputs of several witness parsers, keep each witness's evidence separate or use voting rather than averaging scores into an uninformative middle.
  • I5 (learn): adopt class-stratified data-size sweeps and multi-seed reporting when judging whether a new repair rule or training tweak truly helps.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Fondazione Bruno KesslerTrento · Italy35 works · 43,945 citations🎓KU LeuvenLeuven · Belgium42 works · 5,839 citations🎓Free University of Bozen-BolzanoBolzano · Italy3 works · 344 citations🤝IMECLeuven · Belgium3 works · 215 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Veronica Juliana Schmalz and Alessio Brutti. 2021. Automatic Assessment of English CEFR Levels Using BERT Embeddings. Proceedings of the Eighth Italian Conference on Computational Linguistics (CLiC-it 2021).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{schmalz2021automatic,
  title = {{Automatic Assessment of English CEFR Levels Using BERT Embeddings}},
  author = {Veronica Juliana Schmalz and Alessio Brutti},
  booktitle = {Proceedings of the Eighth Italian Conference on Computational Linguistics (CLiC-it 2021)},
  year = {2021},
  doi = {10.4000/books.aaccademia.10828},
  url = {https://doi.org/10.4000/books.aaccademia.10828}
}

References

23 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yigal Attali. 2004. Exploring the feedback and revision features of criterion. Journal of Second Language Writing, 14:191–205. find
  2. Jill Burstein, Joel Tetreault, and Nitin Madnani. 2013. The e-rater® automated essay scoring system. In Handbook of automated essay evaluation, pages 77– 89. Routledge. find
  3. Carol A Chapelle and Erik Voss. 2008. Utilizing technology in language assessment. Encyclopedia of language and education, 7:123–134. ↗
  4. Carol A Chapelle. 2017. Evaluation of technology and language learning. The handbook of technology and second language teaching and learning, pages 378– 392. ↗
  5. Chi-Fen Emily Chen and Wei-Yuan Eugene Cheng Cheng. 2008. Beyond the design of automated writing evaluation: Pedagogical practices and perceived learning effectiveness in efl writing classes. Language Learning & Technology, 12(2):94–112. ↗
  6. Education Committee Council of Europe, Council for Cultural Co-operation. 2001. Common European Framework of Reference for Languages: learning, teaching, assessment. Cambridge University Press. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  8. Neus Figueras. 2012. The impact of the cefr. ELT journal, 66(4):477–485. ↗
  9. Jeroen Geertzen, Theodora Alexopoulou, Anna Korhonen, et al. 2013. Automatic linguistic annotation of large scale l2 databases: The ef-cambridge open language database (efcamdat). In Proceedings of the 31st Second Language Research Forum. Somerville, MA: Cascadilla Proceedings Project, pages 240– 254. Citeseer. find
  10. Luke William Harding and Tim McNamara. 2017. Language assessment: The challenge of elf. In Routledge Handbook of English as a Lingua Franca. Routledge. find
  11. Carl James. 2005. Contrastive analysis and the language learner. Linguistics, language teaching and language learning, 120. find
  12. Eunice Eunhee Jang. 2017. Cognitive aspects of language assessment. Language Testing and Assessment,, pages 163–177. ↗
  13. Hossein Karami. 2013. The quest for fairness in language testing. Educational Research and Evaluation, 19(2-3):158–169. ↗
  14. Thomas K Landauer. 2003. Automatic essay assessment. Assessment in education: Principles, policy & practice, 10(3):295–308. ↗
  15. Marcin Miłkowski. 2010. Developing an open-source, rule-based proofreading tool. Software: Practice and Experience, 40(7):543–566. ↗
  16. Daniel Naber et al. 2003. A rule-based style and grammar checker. find
  17. Carsten Roever and Tim McNamara. 2006. Language testing: The social dimension. International Journal of Applied Linguistics, 16(2):242–258. ↗
  18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
  19. Joshua Wilson and Rod D Roscoe. 2020. Automated writing evaluation and feedback: Multiple metrics of efficacy. Journal of Educational Computing Research, 58(1):87–125. ↗
  20. Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pier-ric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45.
  21. Xiaoming Xi, Derrick Higgins, Klaus Zechner, and David M Williamson. 2008. Automated scoring of spontaneous speech using speechratersm v1. 0. ETS Research Report Series, 2008(2):i–102. find
  22. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading esol texts. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 180–189.
  23. Wenxin Zhu. 2019. A study on the application of automated essay scoring in college english writing based on pigai. In 2019 5th International conference on social science and higher education (ICSSHE 2019), pages 451–454. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schmalz-2021-automatic-assessment-english-cefr-levels/