Elamur Research⌕
☰
home / papers / 2026 / Automated Reformulation of Argumentative Essays to Improve Argument Organization and Development

Automated Reformulation of Argumentative Essays to Improve Argument Organization and Development

Paper · 2026 · Naomi James Sutcliffe de Moraes and Denis Deratani Mauá

Summary

The paper investigates whether an LLM can act as a feedback tool for Brazilian Portuguese argumentative essays. The tool works by rewriting an essay, a technique known as reformulation. The rewrite should improve only argument quality and leave spelling, cohesion and style mostly untouched. The authors used ENEM-style essays and OpenAI's GPT 4o in two connected steps:

  • Scoring (AES): prompts split ENEM's argument-quality trait into three criteria: stance, organization and development.
  • Reformulation (AER): prompt variants were tested, and the scorer measured how much each rewrite changed.

With careful prompt design, the scorer agreed with human raters as well as the raters agreed with each other. The criterion-level scores also make the judgments easier to interpret.

For reformulation, a generic request to improve the essay produced only cosmetic edits. Substantive changes appeared only when the model was told to add supporting facts or examples. The model added accurate support even without background material. It could not fully separate organization from development. Unconstrained or "maximize the score" prompts changed texts heavily and sometimes even softened the author's stance.

The work matters because it shows that targeted, minimally invasive automated feedback is possible, though it needs careful prompting.

Key points

  • Data: 381 essays from UOL mock ENEM exams (corpus by Silveira et al., 2024), each scored by two expert raters.
    • Four zero-scored essays were removed, giving EA-AES.
    • The 168 essays where both raters agreed form EAC-AES.
    • A 20-essay subset was used for AER.
  • Human baseline (rater A vs. B): QWK 0.57, MAE 0.73, RMSE 1.05.
  • Best LLM scorer: reached QWK up to 0.63 and MAE as low as 0.58. A basic rubric-only prompt had 31% worse MAE than the best three-criteria prompt with background material.
  • Unrestricted reformulation (P22, P23): compared with the restricted versions (P2, P3), Levenshtein distance rose by 63% and 37%, and cosine similarity fell by about 6%. The model also tended to add English-style introduction and conclusion sentences.
  • Limitations: only one LLM (GPT 4o), small datasets, Portuguese only, and no human scores for the three sub-criteria.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Naomi James Sutcliffe de Moraes and Denis Deratani Mauá. 2026. Automated Reformulation of Argumentative Essays to Improve Argument Organization and Development. Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{moraes2026automated,
  title = {{Automated Reformulation of Argumentative Essays to Improve Argument Organization and Development}},
  author = {Naomi James Sutcliffe {de Moraes} and Denis Deratani Mauá},
  booktitle = {Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1},
  year = {2026},
  url = {https://aclanthology.org/2026.propor-1.42/}
}

References

27 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. André Barbosa, Igor Caetano Silveira, and Denis Deratani Mauá. 2025. An empirical analysis of large language models for automated cross-prompt essay trait scoring in Brazilian portuguese. Journal of the Brazilian Computer Society, 31(1):858–871. find
  2. Brazilian Ministry of Education. 2024. Exame Nacional de Ensino Médio. Accessed on October 12, 2025. find
  3. OpenAI. 2024. GPT-4o system card. Accessed on April 12, 2025.
  4. Maria Santos, Sonia López Serrano, and Rosa M Manchón. 2010. The differential effect of two types of direct written corrective feedback on noticing and uptake: Reformulation vs. error correction. International Journal of English Studies, 10(1):131–154. find
  5. Lei Shu, Liangchen Luo, Jayakumar Hoskere, Yun Zhu, Yinxiao Liu, Simon Tong, Jindong Chen, and Lei Meng. 2024. Rewritelm: An instruction-tuned large language model for text rewriting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18970–18980. find
  6. Yingzhao Chen. 2025. Evaluating the potential of ChatGPT-reformulated essays as written feedback in L2 writing. Computers and Education: Artificial Intelligence, page 100500. find
  7. J. Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37–46. find
  8. Valerie J Shute. 2008. Focus on formative feedback. Review of educational research, 78(1):153–189.
  9. Afrizal Doewes, Nughthoh Kurdhi, and Akrati Saxena. 2023. Evaluating quadratic weighted kappa as the standard performance metric for automated essay scoring. In Proceedings of the 16th International Conference on Educational Data Mining, EDM 2023, pages 103–113. International Educational Data Mining… find
  10. Igor Cataneo Silveira, André Barbosa, and Denis Deratani Mauá. 2024. A new benchmark for automatic essay scoring in Portuguese. In Proceedings of the 16th International Conference on Computational Processing of Portuguese, pages 228–237, Santiago de Compostela, Galicia/Spain. Association for Computa… find
  11. Erick Fonseca, Ivo Medeiros, Dayse Kamikawachi, and Alessandro Bokan. 2018. Automatically grading Brazilian student essays. In Computational Processing of the Portuguese Language. find
  12. PROPOR 2018, pages 170–179. Springer International Publishing. find
  13. Yishen Song, Qianta Zhu, Huaibo Wang, and Qinhua Zheng. 2024. Automated essay scoring and revising based on open-source large language models. IEEE Transactions on Learning Technologies, 17:1880– 1890. find
  14. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (Inep). 2019. Enem redações - material de leitura, módulo 5, competência III. Accessed on 2025-10-01. find
  15. Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Wieting, Nanyun Peng, and Xuezhe Ma. 2023. Evaluating large language models on controlled generation tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3155–3168, Singapore… find
  16. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Compututing Surveys, 55(12).
  17. Joan Karbach. 1987. Using toulmin’s model of argumentation. Journal of Teaching Writing, 6(1):81–92. find
  18. Shubhra Kanti Karmaker Santu and Dongji Feng. 2023. TELeR: A general taxonomy of LLM prompts for benchmarking complex tasks. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14197–14203. Association for Computational Linguistics. find
  19. Ha Ram Kim and Melissa Bowles. 2019. How deeply do second language learners process written corrective feedback? Insights gained from think-alouds. TESOL Quarterly, 53(4):913–938. find
  20. Beata Beigman Klebanov and Nitin Madnani. 2022. Automated Essay Scoring. Springer Nature, Switzerland. find
  21. Atsushi Mizumoto and Masaki Eguchi. 2023. Exploring the potential of using an AI language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2):100050. find
  22. Xiaoyi Tang, Hongwei Chen, Daoyu Lin, and Kexin Li. 2024. Harnessing LLMs for multi-dimensional writing assessment: Reliability and alignment with human judgments. Heliyon, 10(14). find
  23. Agustina Tocalli-Beller and Merrill Swain. 2005. Reformulation: The cognitive conflict and L2 learning it generates. International Journal of Applied Linguistics, 15(1):5–28. find
  24. Universo Online (UOL). 2020. Uol Educação Banco de Redações. Accessed June 11, 2025. find
  25. Qiao Wang and John Maurice Gayed. 2024. Effectiveness of large language models in automated evaluation of argumentative essays: finetuning vs. zero-shot prompting. Computer Assisted Language Learning, pages 1–29. find
  26. Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with ChatGPT. arXiv preprint arXiv:2302.11382.
  27. Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. 2023. A survey of controllable text generation using transformer-based pre-trained language models. ACM Compututing Surveys, 56(3). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/moraes-2026-automated-reformulation-argumentative-essays-improve/