Elamur Research⌕
☰
home / papers / 2026 / PHEB: An European Portuguese High School-Level LLM Benchmark

PHEB: An European Portuguese High School-Level LLM Benchmark

Summary

Evaluation of LLMs in European Portuguese (PT-PT) mostly relies on benchmarks machine-translated from English or built for Brazilian Portuguese. These miss Portugal's linguistic, historical and educational context and can contain translation artefacts. The authors address this with PHEB (Portuguese Highschool Exams Benchmark). It is built from official Portuguese national secondary-school exams published by IAVE, the state assessment institute. The team crawled the IAVE archive and processed the exams with automatic PDF parsing plus manual transcription. They then hand-checked every question–answer pair, converted mathematical notation to LaTeX, and manually extracted the official grading rubrics so open-ended answers can be scored by an LLM-as-a-judge. The benchmark mixes multiple-choice items with free-text answers ranging from short completions to essays, so it tests both knowledge and generation. The authors evaluate general-purpose, multilingual and Portuguese-specific open LLMs, examining knowledge gaps, language coverage, answer-format bias and robustness to machine translation. The detailed results were omitted from the supplied text. PHEB matters because it is a native, curriculum-aligned PT-PT resource that covers both reasoning and long-form generation.

Key points

  • Size and coverage: 3,575 questions drawn from 279 national exams over 18 years (2006–2023).
  • Subjects: six disciplines are covered: Mathematics, Biology/Geology, History, Philosophy, Geography, and Portuguese Language and Literature.
  • Question split: 1,819 multiple-choice items, always with four options, and 1,756 open-ended items with official scoring rubrics.
  • Comparison with prior work: the closest earlier resource, EXAMS, had only 920 PT-PT multiple-choice questions across 4 subjects and no open-ended answers.
  • Availability: the dataset is released at github.com/AMALIA-LLM/pheb (NOVA School of Science and Technology, LREC 2026).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2o3fvueefvwj

@inproceedings{tavares2026pheb,
  title = {{PHEB: An European Portuguese High School-Level LLM Benchmark}},
  author = {Tavares, Diogo C. and Ferreira, Rafael and Simplício, Afonso and Vinagre, Gonçalo and Condez, Ana Carolina and Calvo, Inês and Vieira, Inês and Semedo, David and Magalhaes, Joao},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2o3fvueefvwj},
  url = {https://doi.org/10.63317/2o3fvueefvwj}
}

References

31 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hugo Queiroz Abonizio, Thales Sales Almeida, Thiago Soares Laitz, Roseval Malaquias Junior, Giovana K. Bonás, Rodrigo Nogueira, and Ramon Pires. 2024. Sabiá-3 Technical Report. CoRR, abs/2410.12049. — [arXiv:2410.12049](https://arxiv.org/abs/2410.12049) find
  2. Thales Sales Almeida, Hugo Queiroz Abonizio, Rodrigo Nogueira, and Ramon Pires. 2024. Sabiá-2: A New Generation of Portuguese Large Language Models. CoRR, abs/2403.09887. — [arXiv:2403.09887](https://arxiv.org/abs/2403.09887) find
  3. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. CoRR, abs/1803.05457. find
  4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. find our card
  5. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, et al. 2024. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. find our card
  6. Nicholas Kluge Corrêa, Aniket Sen, Sophia Falk, and Shiza Fatimah. 2025. Tucano: Advancing Neural Text Generation for Portuguese. Patterns. find
  7. José Ángel González, Ian Borrego-Obrador, Álvaro Romo Herrero, Areg Mikael Sarvazyan, Mara Chinea-Rios, Angelo Basile, and Marc Franco-Salvador. 2026. IberBench: LLM evaluation on Iberian languages. Comput. Speech Lang., 96:101899. find
  8. Aitor Gonzalez-Agirre, Marc Pàmies, Joan Llop, Irene Baucells, Severino Da Dalt, Daniel Tamayo, José Javier Saiz, Ferran Espuña, Jaume Prats, Javier Aula-Blasco, Mario Mina, Adrián Rubio, Alexander Shvets, Anna Sallés, Iñaki Lacunza, Iñigo Pikabea, Jorge Palomar, Júlia Falcão, Lucía Tormo, Luis Vasq… find
  9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, et al. 2024. The Llama 3 Herd of Models. find
  10. Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslav Nakov. 2020. EXAMS: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Proc… find
  11. Helia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme, and Chris Kedzie. 2024. LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap… find
  12. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. find our card
  13. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  14. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683. find our card
  15. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving Quantitative Reasoning Problems with Language Models. In Advances in… find our card
  16. Ricardo Lopes, Joao Magalhaes, and David Semedo. 2024. GlórIA: A generative and open large language model for Portuguese. In Proceedings of the 16th International Conference on Computational Processing of Portuguese - Vol. 1, pages 441–453, Santiago de Compostela, Galicia/Spain. Association for Comp… find
  17. Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. 2024. EuroLLM: Multilingual Language M… find our card
  18. Hugo Gonçalo Oliveira, Ricardo Rodrigues, Bruno Ferreira, Purificação Silvano, and Sara Carvalho. 2024. BATS-PT: Assessing Portuguese masked language models in lexico-semantic analogy solving and relation completion. In Proceedings of the 16th International Conference on Computational Processing of… find
  19. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, et al. 2025. 2 OLMo 2 Furious. find
  20. Tomás Freitas Osório, Bernardo Leite, Henrique Lopes Cardoso, Luís Gomes, João Rodrigues, Rodrigo Santos, and António Branco. 2024. PORTULAN ExtraGLUE datasets and models: Kick-starting a benchmark for the neural processing of Portuguese. In Proceedings of the 17th Workshop on Building and Using Com… find
  21. Miguel Angel Peñaloza Perez, Bruno Lopez Orozco, Jesus Tadeo Cruz Soto, Michelle Bruno Hernandez, Miguel Angel Alvarado Gonzalez, and Sandra Malagon. 2025. AI4Math: A Native Spanish Benchmark for University-Level Mathematical Reasoning in Large Language Models. arXiv preprint arXiv:2505.18978. — [arXiv:2505.18978](https://arxiv.org/abs/2505.18978) find
  22. Ricardo Rei, Nuno Miguel Guerreiro, José Pombal, João Alves, Pedro Teixeirinha, M. Amin Farajian, and André F. T. Martins. 2025. Tower+: Bridging Generality and Translation Specialization in Multilingual LLMs. CoRR, abs/2506.17080. find
  23. Sagi Shaier, George Arthur Baker, Chiranthan Sridhar, Lawrence Hunter, and Katharina Von Der Wense. 2025. MALAMUTE: A Multilingual, Highly-granular, Template-free, Education-based Probing Dataset. In Findings of the Association for Computational Linguistics: ACL 2025, pages 4051–4069, Vienna, Austri… find
  24. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, André F. T. Martins, Leshem… find our card
  25. Klaudia Thellmann, Bernhard Stadler, Michael Fromm, Jasper Schulze Buschhoff, Alex Jude, Fabio Barth, Johannes Leveling, Nicolas Flores-Herr, Joachim Köhler, René Jäkel, and Mehdi Ali. 2024. Towards Multilingual LLM Evaluation for European Languages. CoRR, abs/2410.08928. — [arXiv:2410.08928](https://arxiv.org/abs/2410.08928) find
  26. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural… find our card
  27. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRevi… find our card
  28. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark… find our card
  29. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processi… find our card
  30. Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023. M3exam: a multilingual, multimodal, multilevel benchmark for examining large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY,… find
  31. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large Language Models Are Not Robust Multiple Choice Selectors. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tavares-2026-pheb-european-portuguese-high-school/