Elamur Research⌕
☰
home / papers / 2026 / Evaluating Automated Scoring Models on Official ENEM Essays

Evaluating Automated Scoring Models on Official ENEM Essays

Paper · 2026 · Laís Nuto Rossman, Igor Cataneo Silveira and Denis Deratani Mauá

Summary

Most automated essay scoring (AES) systems for Brazilian Portuguese ENEM essays are trained on mock-exam essays scraped from practice websites. It has been unclear whether these systems transfer to real exam essays, because official essays are almost never released. The authors built a new labeled dataset of 157 official ENEM essays. Volunteers submitted scans of their essays along with their official score sheets. The handwritten texts were transcribed with LLM help, checked manually and anonymized.

The authors first used NILC-Metrix features and per-competence linear regression to compare the corpus with mock-exam data. They then tested three encoders previously trained on mock essays by Barbosa et al. (2025) under three settings: zero-shot, fine-tuning the pretrained models on official essays, and training only on official essays. Mock-trained models already generalized reasonably well, and fine-tuning usually improved them. Pretraining on mock essays gave a large advantage over official-only training. The practical takeaway is to pretrain on plentiful mock essays and then adapt with a small set of real ones.

Key points

  • Dataset: 173 essays were collected from eight ENEM editions (2016 and 2018–2024), and 157 complete ones were kept. Each record has the text, the exam year, scores for all five competencies (C1–C5, 0–200 in steps of 40), the prompt and the supporting texts. The dataset is released on HuggingFace, but it is skewed toward high scores because many contributors are university students.
  • Linguistic comparison: Official essays are longer than mock essays (about 335 vs. 264 words on average), while sentence structure is similar. The most predictive NILC-Metrix features match those reported for mock data (Silveira et al., 2025).
  • Models and metrics: BERTimbau-base, BERTuguês and mBERT were evaluated with QWK and weighted F1. After fine-tuning, the best gain was +0.316 QWK (BERTimbau, C5) and the worst change was −0.04 QWK (mBERT, C1).
  • Ablation: Mock-pretrained then fine-tuned models beat official-only models by about 0.27 QWK on average.
  • GPT-4o: Despite being far larger, GPT-4o did not outperform the ~110M-parameter encoders, making it less cost-effective for this task.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Laís Nuto Rossman, Igor Cataneo Silveira and Denis Deratani Mauá. 2026. Evaluating Automated Scoring Models on Official ENEM Essays. Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{rossman2026evaluating,
  title = {{Evaluating Automated Scoring Models on Official ENEM Essays}},
  author = {Laís Nuto Rossman, Igor Cataneo Silveira and Denis Deratani Mauá},
  booktitle = {Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1},
  year = {2026},
  url = {https://aclanthology.org/2026.propor-1.16/}
}

References

15 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Evelin Amorim and Adriano Veloso. 2017. A multi-aspect analysis of automatic essay scoring for Brazilian Portuguese. In Proceedings of the Student Research Workshop at the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 94–102. Association for Computat… find
  2. André Barbosa, Igor Cataneo Silveira, and Denis Deratani Mauá. 2025. An empirical analysis of large language models for automated cross-prompt essay trait scoring in brazilian portuguese. Journal of the Brazilian Computer Society, 31(1):858–871. find
  3. Jeziel Marinho, Rafael Anchiêta, and Raimundo Moura. 2022a. Essay-br: a brazilian corpus to automatic essay scoring task. Journal of Information and Data Management, 13(1):65–76. find
  4. Jeziel Marinho, Fábio Cordeiro, Rafael Anchiêta, and Raimundo Moura. 2022b. Automated essay scoring: An approach based on enem competencies. In Anais do XIX Encontro Nacional de Inteligência Artificial e Computacional, pages 49–60. find
  5. Ricardo Mazza Zago and Luciane Agnoletti dos Santos Pedotti. 2024. Bertugues: A novel bert transformer model pre-trained for brazilian portuguese. Semina: Ciências Exatas e Tecnológicas, 45:e50630. find
  6. Rafael Ferreira Mello, Hilário Oliveira, Moésio Wenceslau, Hyan Batista, Thiago Cordeiro, Ig Ibert Bittencourt, and Seiji Isotanif. 2024. PROPOR’24 Competition on Automatic Essay Scoring of Portuguese Narrative Essays. In Proceedings of the 16th International Conference on Computational Processing o… find
  7. Rodrigo Cavalcanti, Gabriela Casini, Gabriel Assis, Livy Real, Daniela Vianna, Paulo Mann, and Aline Paes. 2025. Diplomatrix-br: Um corpus paralelo de redaçoes de autoria humana e de llms no concurso de diplomacia brasileira. In Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (… find
  8. Rogério F de Sousa, Jeziel C Marinho, Francisco AR Neto, Rafael Anchiêta, and Raimundo S Moura. 2024. PiLN at PROPOR: A BERT-Based Strategy for Grading Narrative Essays. In Proceedings of the 16th International Conference on Computational Processing of Portuguese-Vol. 2, pages 10–13. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics.
  10. Afrizal Doewes, Nughthoh Arfawi Kurdhi, and Akrati Saxena. 2023. Evaluating quadratic weighted kappa as the standard performance metric for automated essay scoring. In Proceedings of the 16th International Hilário Oliveira, Rafael Ferreira Mello, Péricles Miranda, Hyan Batista, Moésio Wenceslau Silv… find
  11. Ellis B. Page. 1966. The imminence of... grading essays by computer. The Phi Delta Kappan, pages 238–243. find
  12. Eugénio Ribeiro, Nuno Mamede, and Jorge Baptista. 2024. Exploring the Automated Scoring of Narrative Essays in Brazilian Portuguese using Transformer Models. In Proceedings of the 16th International Conference on Computational Processing of Portuguese-Vol. 2, pages 14–17. find
  13. Igor Cataneo Silveira, André Barbosa, Daniel Silva Lopes da Costa, and Denis Deratani Mauá. 2025. Investigating Universal Adversarial Attacks Against Transformers-Based Automatic Essay Scoring Systems. In Intelligent Systems, pages 169–183, Cham. Springer Nature Switzerland. find
  14. Igor Cataneo Silveira, André Barbosa, and Denis Deratani Mauá. 2024. A new benchmark for automatic essay scoring in Portuguese. In Proceedings of the 16th International Conference on Computational Processing of Portuguese - Vol. 1, pages 228–237. find
  15. Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2020. Bertimbau: pretrained bert models for brazilian portuguese. In Brazilian conference on intelligent systems, pages 403–417. Springer. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rossman-2026-evaluating-automated-scoring-models-official/