Elamur Research⌕
☰
home / papers / 2026 / Using machine learning and large language models for the automatic assessment of Celpe-Bras exam texts

Using machine learning and large language models for the automatic assessment of Celpe-Bras exam texts

Original (Portuguese): Uso de técnicas de Aprendizado de Máquina e Modelos de Língua de Larga Escala para avaliação automática de textos do exame Celpe-Bras

Paper · 2026 · Rafael Oleques Nunes, Bernardo Cobalchini Zietolie, Ricardo Zanini De Costa et al.

Summary

The paper tackles automated essay scoring (AES) for the written part of Celpe-Bras, Brazil's official proficiency exam in Portuguese as an additional language. Teachers preparing candidates face heavy grading loads, and learners have few affordable resources aligned with the exam's construct. The authors therefore offer a first systematic comparison of scoring approaches for this exam. They used texts from the Corpus Celpe-Bras (CorCel) and framed scoring as six-class classification over grades 0–5. Three model families were compared:

  • Classical baselines: Naive Bayes, SVM and Random Forest on TF-IDF features.
  • Portuguese encoder models: BERT variants.
  • Encoder-decoder models: mBART-50 and a Portuguese T5.

The BERT-based models, especially BERTimbau (large), scored best and reached moderate agreement with official raters, though at a much higher computational cost. SVM came fairly close for a fraction of the compute. The work shows that automated scoring of Celpe-Bras texts is feasible. It also sets a baseline for future feedback tools for learners of Portuguese.

Key points

  • Data: CorCel holds 15,315 rated texts (about 3M words) from four exam editions (2015-2 to 2017-1). The experiments used one task's subset of 2,293 texts with integer grades, which is heavily imbalanced (only 25 texts scored 0; most scored 2–3).
  • Models:
    • Naive Bayes, SVM and RF, with stopword removal and snowball stemming.
    • BERTimbau (base and large), Albertina (100m) and BERTugues (base).
    • mBART-50 and ptt5-v2-base.
  • Evaluation: stratified 5-fold cross-validation, with 10% of each training split held out for validation. Metrics were reported in macro and weighted forms, and models were trained on Colab T4 GPUs.
  • Best results: BERTimbau (large) on the optimized data reached accuracy 0.4128 and F1 0.4049. BERT variants achieved QWK of about 0.59 on the full dataset and about 0.57 on the optimized one.
  • Weaker models:
    • Albertina over-predicted grades 2 and 3 and struggled with minority classes.
    • Naive Bayes, RF and mBART-50 performed poorly.
    • SVM and T5 performed similarly to each other.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Rafael Oleques Nunes, Bernardo Cobalchini Zietolie, Ricardo Zanini De Costa, Rodrigo Brock da Silva, João Victor Piardi Pacheco, Rafaela Dall'Agnol da Rocha, Dennis Giovani Balreira, Elisa Marchioro Stumpf and Juliana Roquele Schoffen. 2026. Uso de técnicas de Aprendizado de Máquina e Modelos de Língua de Larga Escala para avaliação automática de textos do exame Celpe-Bras. Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{nunes2026uso,
  title = {{Uso de técnicas de Aprendizado de Máquina e Modelos de Língua de Larga Escala para avaliação automática de textos do exame Celpe-Bras}},
  author = {Rafael Oleques Nunes, Bernardo Cobalchini Zietolie, Ricardo Zanini {De Costa}, Rodrigo Brock {da Silva}, João Victor Piardi Pacheco, Rafaela Dall'Agnol {da Rocha}, Dennis Giovani Balreira, Elisa Marchioro Stumpf and Juliana Roquele Schoffen},
  booktitle = {Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1},
  year = {2026},
  url = {https://aclanthology.org/2026.propor-1.85/}
}

References

27 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yigal Attali and Jill Burstein. 2004. Automated essay scoring with e-rater® v. 2.0. ETS Research Report Series, 2004(2):i–21. find
  2. Claire Barale, Michael Rovatsos, and Nehal Bhuta. 2023. Do language models learn about legal entity types during pretraining? In Proceedings of the Natural Legal Language Processing Workshop, volume 2023, pages 25–37. find
  3. Jing Chen, James H Fife, Isaac I Bejar, and André A Rupp. 2016. Building e-rater® scoring models using machine learning methods. ETS Research Report Series, 2016(1):1–12. find
  4. Tobias Deußer, Cong Zhao, Lorenz Sparrenberg, Daniel Uedelhoven, Armin Berger, Maren Pielka, Lars Hillebrand, Christian Bauckhage, and Rafet Sifa. 2024. A comparative study of large language models for named entity recognition in the legal domain. In 2024 IEEE International Conference on Big Data (B… find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  6. Luiza Sarmento Divino. 2024. Contribuições da linguística de corpus para a definição de níveis de proficiência escrita no exame celpe-bras. Master’s thesis, Universidade Federal do Rio Grande do Sul. find
  7. INEP. 2020. Documento base do exame Celpe-Bras. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. find
  8. Yingying Liu, Huilei Qi, and Xiaofei Lu. 2025. Enhancing gpt-based automated essay scoring: the impact of fine-tuning and linguistic complexity measures. Computer Assisted Language Learning, pages 1–20. find
  9. Josephine Lukito, Bin Chen, Gina M. Masullo, and Natalie Jomini Stroud. 2024. Comparing a BERT classifier and a GPT classifier for detecting connective language across multiple social media. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19140–19153,… find
  10. Atsushi Mizumoto and Masaki Eguchi. 2023. Exploring the potential of using an ai language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2):100050. find
  11. Ellen Yurika Nagasawa. 2023. O conteúdo de insumo em tarefas que integram leitura e escrita no celpe-bras: uma abordagem informada por corpus. Doctoral dissertation. find
  12. Austin Pack, Alex Barrett, and Juan Escalante. 2024. Large language models and automated essay scoring of english language learner writing: Insights into validity and reliability. Computers and Education: Artificial Intelligence, 6:100234. find
  13. Youri Peskine, Damir Korenčić, Ivan Grubisic, Paolo Papotti, Raphael Troncy, and Paolo Rosso. 2023. Definitions matter: Guiding gpt for multi-label classification. In EMNLP 2023, Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
  14. Marcos Piau, Roberto Lotufo, and Rodrigo Nogueira. 2024. ptt5-v2: A closer look at continued pretraining of t5 models for the portuguese language. find
  15. Amanda Pontes Rassi and Priscilla de Abreu Lopes. 2024. Correção automática de redação. In Helena de Medeiros Caseli and Maria das Graças Volpe Nunes, editors, Processamento de Linguagem Natural: conceitos, técnicas e aplicações em português, 3 edition, pages 610–633. BPLN, São Carlos. find
  16. Eugénio Ribeiro, Nuno Mamede, and Jorge Baptista. 2024. Avaliação automática do nível de complexidade de textos em português europeu. Linguamática, 16(2):115–139. find
  17. João Rodrigues, Luís Gomes, João Silva, António Branco, Rodrigo Santos, Henrique Lopes Cardoso, and Tomás Osório. 2023. Advancing neural encoding of portuguese with transformer albertina pt. In EPIA Conference on Artificial Intelligence, pages 441–453. Springer. find
  18. Juliana Schoffen, Margarete Schlatter, Simone Paula Kunrath, Ellen Yurika Nagasawa, Gabrielle Rodrigues Sirianni, Kaiane Mendel, Luana Ramos Truyllio, and Luiza Sarmento Divino. 2018. Estudo descritivo das tarefas da parte escrita do exame celpe-bras: Edições de 1998 a 2017. Technical report, Porto… find
  19. Juliana Roquele Schoffen, Elisa Marchioro Stumpf, Luiza Sarmento Divino, Isadora Dahmer Hanauer, Deise Amaral, Amanda Michel Raupp, and Brenda de Souza Xavier. 2025. Corcel: A brazilian portuguese \<i\>corpus\</i\> of celpe-bras exam written texts. Revista Brasileira de Linguística Aplicada, 25(1):e5003… find
  20. Mark D Shermis and Joshua Wilson. 2024. Introduction to automated essay evaluation. In The Routledge international handbook of automated essay evaluation, pages 3–22. Routledge. find
  21. Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2020. Bertimbau: pretrained bert models for brazilian portuguese. In Intelligent Systems: 9th Brazilian Conference, BRACIS 2020, Rio Grande, Brazil, October 20–23, 2020, Proceedings, Part I 9, pages 403–417. Springer. find
  22. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. find
  23. Erik Voss. 2025. Comparison of traditional machine learning and neural network approaches for automated scoring of second language english essays. Language Testing, 42(4):369–396. find
  24. Jorge A. Wagner Filho, Rodrigo Wilkens, Marco Idiart, and Aline Villavicencio. 2018. The brWaC corpus: A new open resource for Brazilian Portuguese. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources… find
  25. Rodrigo Wilkens, Alice Pintard, David Alfter, Vincent Folny, and Thomas François. 2023. Tcfle-8: a corpus of learner written productions for french as a foreign language and its application to automated essay scoring. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro… find
  26. Steven Yeung. 2025. A comparative study of rule-based, machine learning and large language model approaches in automated writing evaluation (awe). In Proceedings of the 15th International Learning Analytics and Knowledge Conference, pages 984–991. find
  27. Ricardo Mazza Zago and Luciane Agnoletti dos Santos Pedotti. 2024. Bertugues: A novel bert transformer model pre-trained for brazilian portuguese. Semina: Ciências Exatas e Tecnológicas, 45:e50630– e50630. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nunes-2026-uso-de-tecnicas-de-aprendizado/