Elamur Research⌕
☰
home / papers / 2026 / EduBench: A Portuguese Benchmark for Open-Ended Discursive Question Answering

EduBench: A Portuguese Benchmark for Open-Ended Discursive Question Answering

Summary

Most LLM benchmarks, particularly for Portuguese, test option selection or short span extraction. They reveal little about whether a model can write an extended, well-argued answer. EduBench fills this gap with 3,149 essay-style questions from the second-phase entrance exams of UNESP, USP and UNICAMP, covering 2015–2025 and spanning Humanities, Exact and Natural Sciences, and Languages. Reference answers come from official commented solutions or, when those are unavailable, from resolutions published by Colégio Objetivo.

A three-step pipeline builds the dataset. It extracts text and figures from the exam PDFs, replaces images with GPT-4o-mini captions so text-only models can be tested, and uses GPT-4o to segment exams into items, clean prompts, attach answers and assign subject labels. Quality checks follow.

Nine models, from 4B-parameter SLMs to reasoning-oriented LLMs, were scored with BLEU, BERTScore and G-Eval. The metrics frequently contradict one another, and models reach roughly 54–71% agreement with expert answers. This shows how hard it is to judge discursive output automatically. The release gives Portuguese NLP a public testbed for long-form generation and its evaluation.

Key points

  • Dataset scope: 3,149 open-ended questions from Brazilian university entrance exams (UNESP, USP, UNICAMP, 2015–2025), each with institution, year and subject metadata, plus short-form answers where applicable.
  • Construction: GPT-4o-mini generated the figure descriptions and GPT-4o handled structuring. Reference answers were taken from official or Colégio Objetivo commented solutions.
  • Baselines: Nine models ranging from 4B SLMs to state-of-the-art reasoning LLMs were evaluated with BLEU, BERTScore and G-Eval. Alignment with expert answers ranged from 54% to 71%, and the metrics showed notable disagreement.
  • Availability: The data is released on Hugging Face (recogna-nlp/EduBench) and the code on GitHub (pedropaiola/EduBench).
  • Limitations: The benchmark covers Brazilian Portuguese only and may contain residual extraction errors. Evaluation is fully automatic, with no large-scale human study, which the authors list as future work alongside expansion to other Portuguese varieties.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Question answering · Evaluation & benchmarks

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4nocig8f36r9

@inproceedings{paiola2026edubench,
  title = {{EduBench: A Portuguese Benchmark for Open-Ended Discursive Question Answering}},
  author = {Paiola, Pedro Henrique and Mendes, Luís Gabriel Damiati and Monchelato, Bruno de Oliveira and Schuck, André da Fonseca and Garcia, Gabriel Lino and Rodrigues, Douglas and Caseli, Helena de Medeiros and Papa, João Paulo},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4nocig8f36r9},
  url = {https://doi.org/10.63317/4nocig8f36r9}
}

References

17 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Thales Sales Almeida, Thiago Laitz, Giovana K. Bonás, and Rodrigo Nogueira. 2023. Bluex: A benchmark based on brazilian leading universities entrance exams. find
  2. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. Ms marco: A human generated machine reading comprehension dataset. find our card
  3. Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. Humans or LLMs as the judge? a study on judgement bias. find
  4. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. find our card
  5. Rudolf Flesch. 1948. A New Readability Yardstick, volume 32. American Psychological Association. find
  6. Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328. find our card
  7. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Lin… find
  8. Marcelo Sartori Locatelli et al. 2025. Examining the Behavior of LLM Architectures within the Framework of Standardized National Exams in Brazil, page 879–890. AAAI Press. find
  9. Teresa Bertholdo Freire Martins, Claudete Moreno Ghiraldelo, Maria das Graças Volpe Nunes, and Osvaldo Novais de Oliveira Junior. 1996. Readability Formulas Applied to Textbooks in Brazilian Portuguese. Ph.D. thesis, Instituto de Ciências Matemáticas de São Carlos, Universidade de São Paulo, São Car… find
  10. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, page 311–318, USA. Association for Computational Linguistics. find our card
  11. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. find our card
  12. Hélio Fonseca Sayama, Anderson Viçoso Araujo, and Eraldo Rezende Fernandes. 2019. Faquad: Reading comprehension dataset in the domain of brazilian higher education. In 2019 8th Brazilian Conference on Intelligent Systems (BRACIS), pages 443–448. find
  13. Aarohi Srivastava et al. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research. Featured Certification. find our card
  14. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark… find our card
  15. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTscore: Evaluating text generation with BERT. In Proceedings of the 8th International Conference on Learning Representations. find
  16. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. find
  17. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/paiola-2026-edubench-portuguese-benchmark-open-ended/