Elamur Research⌕
☰
home / papers / 2026 / ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs

ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs

Paper · 2026 · Inês Vieira, Inês Calvo, Iago Paulo et al.

Summary

Most Portuguese-language training data and benchmarks reflect Brazilian Portuguese (pt-BR). As a result, evaluations of European Portuguese (pt-PT) are often skewed or rely on machine-translated English material, which cannot capture rhyme, wordplay or idioms. ALBA is a generation-oriented benchmark written natively in pt-PT by two linguistics experts. Its 800 open-ended prompts are divided among eight linguistic dimensions: syntax, morphology, lexicology, phonetics and phonology, word plays, discourse analysis, semantics tied to culture, and language variety. Model answers are scored on a 1–5 scale by an LLM-as-a-judge, which was calibrated against expert reference answers at several quality levels. The authors tested 7B–12B open models alongside two proprietary systems. Open models do reasonably well on discourse and syntax tasks but perform poorly on phonology, wordplay and character-level morphology. The authors link these weaknesses to tokenization. Proprietary models score around 90/100. The benchmark gives pt-PT a variety-aware diagnostic tool and offers a template for other under-resourced language varieties.

Key points

  • The dataset has 800 expert-authored questions, with 100 per dimension. They were built in two rounds: 30 per dimension with graded reference answers for judge validation, then 70 more. Each item was written by one expert and checked by the other.
  • Judge scores (1–5) are rescaled to 0–100. Overall results: GPT-5 scored 91.0, Gemini 2.5-Pro 90.1, Gemma 3-12B 51.1 (best open-weight), Qwen 3-8B 49.8, and AMALIA-9B 43.6 (best fully open). OLMo 2-7B was lowest at 16.9.
  • Discourse Analysis is the easiest dimension for open models, with Gemma 3-12B reaching 85.5. Word Plays and Phonetics & Phonology are the hardest, with most open models scoring below 35.
  • Common errors include invented slang, mixing up pt-BR and pt-PT, missing irony and cultural humour, and unintended language mixing.
  • Other models evaluated include EuroLLM-9B, Salamandra-7B, Apertus-8B, Gervasio-8B, Mistral-7B, Ministral-8B, LLaMA 3.1-8B, Qwen 2.5-7B and Gemma 2-9B. The resource is released at github.com/AMALIA-LLM/alba-benchmark.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universidade Nova de LisboaLisbon · Portugal5 works · 1,901 citations🔬Laboratório para a Ciência da Computação e InformáticaCosta da Caparica · Portugal3 works · 11 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Inês Vieira, Inês Calvo, Iago Paulo, James Furtado, Rafael Ferreira, Diogo Tavares, Diogo Glória-Silva, David Semedo and João Magalhães. 2026. ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs. Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{vieira2026alba,
  title = {{ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs}},
  author = {Inês Vieira, Inês Calvo, Iago Paulo, James Furtado, Rafael Ferreira, Diogo Tavares, Diogo Glória-Silva, David Semedo and João Magalhães},
  booktitle = {Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1},
  year = {2026},
  url = {https://aclanthology.org/2026.propor-1.69/}
}

References

55 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Thales Sales Almeida, Giovana Kerche Bonás, and João Guilherme Alves Santos. 2025. Broverbs – measuring how much llms understand portuguese proverbs. Preprint, arXiv:2509.08960. — [arXiv:2509.08960](https://arxiv.org/abs/2509.08960) find
  2. Project Apertus. 2025. Apertus: Democratizing open and compliant llms for global language environments. CoRR, abs/2509.14233.
  3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, and 26 others. 2020. Language models are few-shot learners. In NeurIPS 2020, December 6-12, 2020, virtual.
  4. Florian Cafiero and Marie Puren. 2025. A Riddle in a Haystack: LLM Detection of Intricate Wordplays in Colette and Willy’s Novels for Authorship Attribution. In Digital Humanities 2025, Lisbonne, Portugal. find
  5. Universidade Nova de Lisboa and Universidade de Coimbra and Universidade de Aveiro. Mukund Choudhary, KV Aditya Srivatsa, Gaurja Aeron, Antara Raaghavi Bhattacharya, Dang Khoa Dang Dinh, and 4 others. 2025. Unveiling: What makes linguistics olympiad puzzles tricky for llms? Preprint, arXiv:2508.1126… find
  6. David Crystal. 2010. A Little Book of Language. Yale University Press, Cornwall. find
  7. DeepSeek-AI. 2024. Deepseek-v3 technical report. CoRR, abs/2412.19437. find
  8. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. CoRR, abs/2501.12948.
  9. James Paul Gee. 2018. Introducing Discourse Analysis - From Grammar to Society. Routledge, Oxford. find
  10. Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. 2016. Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. In Proceedings of the Student Research Workshop, SRW@HLT-NAACL 2016, NAACL: Human Language Technologies, USA, pages 8–15. The…
  11. José Ángel González, Ian Borrego-Obrador, Álvaro Romo Herrero, Areg Mikael Sarvazyan, Mara Chinea-Rios, and 2 others. 2026. Iberbench: LLM evaluation on iberian languages. Comput. Speech Lang., 96:101899. find
  12. Aitor Gonzalez-Agirre, Marc Pàmies, Joan Llop, Irene Baucells, Severino Da Dalt, and 19 others. 2025. Salamandra technical report. CoRR, abs/2502.08489. — [arXiv:2502.08489](https://arxiv.org/abs/2502.08489) find
  13. Satyam Goyal and Soham Dan. 2025. Iolbench: Benchmarking llms on linguistic reasoning. Preprint, arXiv:2501.04249.
  14. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, and 7 others. 2024. A survey on llm-as-a-judge. CoRR, abs/2411.15594. find
  15. Mete Ismayilzada, Defne Circi, Jonne Sälevä, Hale Sirin, Abdullatif Köksal, and 4 others. 2025. Evaluating morphological compositional generalization in large language models. Preprint, arXiv:2410.12656. find
  16. Howard Jackson and Etienne Zé Amvela. 2000. Words, Meaning and Vocabulary: An Introduction to Modern English Lexicology. Continuum, London. find
  17. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, and 13 others. 2023. Mistral 7b. CoRR, abs/2310.06825. find
  18. Eunsu Kim, Juyoung Suk, Philhoon Oh, Haneul Yoo, James Thorne, and Alice Oh. 2024. Click: A benchmark dataset of cultural and linguistic intelligence in korean. Preprint, arXiv:2403.06412. find
  19. Olaf Koeneman and Hedde Zeijlstra. 2017. Introducing Syntax. Cambridge University Press, Cambridge. find
  20. Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, and 2 others. 2023. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. In Proceedings of EMNLP 2023 - System Demonstrations, Singapore, pages 318– 327…
  21. Bingru Li and Han Wang. 2024. Tacomore: Leveraging the potential of llms in corpus-based discourse analysis with prompt engineering. Preprint, arXiv:2412.10139. — [arXiv:2412.10139](https://arxiv.org/abs/2412.10139) find
  22. Rochelle Lieber. 2009. Introducing Morphology. Cambridge University Press, Cambridge. find
  23. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In EMNLP 2023, Singapore, pages 2511–2522. ACL. models in lexico-semantic analogy solving and relation completion. In PROPOR, pages 207–217, Santiago de…
  24. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, and 35 others. 2025. 2 olmo 2 furious. CoRR, abs/2501.00656. find
  25. OpenAI. 2025. Gpt-5 system card. find
  26. Tomás Freitas Osório, Bernardo Leite, Henrique Lopes Cardoso, Luís Gomes, João Rodrigues, and 2 others. 2024. PORTULAN ExtraGLUE datasets and models: Kick-starting a benchmark for the neural processing of Portuguese. In Workshop on Building and Using Comparable Corpora (BUCC) @ LREC-COLING 2024, pag… find
  27. Sachin Pawar, Manoj Apte, Kshitij Jadhav, Girish Keshav Palshikar, and Nitin Ramrakhiyani. 2025. Broken words, broken performance: Effect of tokenization on performance of llms. Preprint, arXiv:2512.21933. find
  28. Paloma Piot, José Ramom Pichel Campos, and Javier Parapar. 2025. Bridging gaps in hate speech detection: Meta-collections and benchmarks for low-resource iberian languages. CoRR, abs/2510.11167. — [arXiv:2510.11167](https://arxiv.org/abs/2510.11167) find
  29. Livy Real, Erick Fonseca, and Hugo Goncalo Oliveira. 2020. The assin 2 shared task: a quick overview. In PROPOR, pages 406–412. Springer. find
  30. Parker Riley, Timothy Dozat, Jan A. Botha, Xavier Garcia, Dan Garrette, and 3 others. 2023. FRMT: A benchmark for few-shot region-aware machine translation. Trans. Assoc. Comput. Linguistics, 11:671– 685.
  31. Morgane Rivière, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, and 94 others. 2024. Gemma 2: Improving open language models at a practical size. CoRR, abs/2408.00118. find
  32. Ricardo Lopes, João Magalhães, and David Semedo. 2024. Glória: A generative and open large language model for portuguese. In Proceedings of the 16th International Conference on Computational Processing of Portuguese, PROPOR 2024, Santiago de Compostela, Galicia/Spain, March 12-15, 2024, Volume 1, pa… find
  33. Rodrigo Santos, João Silva, Luís Gomes, João Rodrigues, and António Branco. 2024. Advancing generative AI for portuguese with open decoder gervásio PT. CoRR, abs/2402.18766. find
  34. Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno Miguel Guerreiro, Ricardo Rei, and 10 others. 2024. Eurollm: Multilingual language models for europe. CoRR, abs/2409.16235.
  35. Afonso Simplício, Gonçalo Vinagre, Miguel Ramos, Diogo Tavares, Rafael Ferreira, and 17 others. 2026. AMALIA: An open source large language model for european portuguese. In PROPOR, Salvador, Bahia, Brazil. find
  36. Nathalie Norman, Sanni Nimb, Sussi Olsen, Nina Schneidermann, and Bolette S. Pedersen. 2025. Electronic lexicography in the 21st century (eLex). find
  37. David Odden. 2005. Introducing Phonology. Cambridge University Press, Cambridge. find
  38. Hugo Gonçalo Oliveira, Ricardo Rodrigues, Bruno Ferreira, Purificação Silvano, and Sara Carvalho. 2024. BATS-PT: Assessing Portuguese masked language Andrew Shin and Kunitake Kaneko. 2024. Large language models lack understanding of character composition of words. Preprint, arXiv:2405.11357. — [arXiv:2405.11357](https://arxiv.org/abs/2405.11357) find
  39. Ashima Suvarna, Harshita Khandelwal, and Nanyun Peng. 2024. Phonologybench: Evaluating phonological skills of large language models. Preprint, arXiv:2404.02456. find
  40. Gemini Team. 2025a. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. CoRR, abs/2507.06261.
  41. Gemma Team. 2025b. Gemma 3 technical report. CoRR, abs/2503.19786. find
  42. Llama Team. 2024a. The llama 3 herd of models. CoRR, abs/2407.21783.
  43. Mistral AI Team. 2024b. Ministral 8b instruct 2410. https://huggingface.co/mistralai/ Ministral-8B-Instruct-2410. ↗
  44. Released October 2024. find
  45. Qwen Team. 2025c. Qwen3 technical report. Preprint, arXiv:2505.09388.
  46. Klaudia Thellmann, Bernhard Stadler, Michael Fromm, Jasper Schulze Buschhoff, Alex Jude, and 6 others. 2024. Towards multilingual LLM evaluation for european languages. CoRR, abs/2410.08928. — [arXiv:2410.08928](https://arxiv.org/abs/2410.08928) find
  47. Andreas Waldis, Yotam Perlitz, Leshem Choshen, Yufang Hou, and Iryna Gurevych. 2024. Holmes: A benchmark to assess the linguistic competence of language models. Preprint, arXiv:2404.18923. — [arXiv:2404.18923](https://arxiv.org/abs/2404.18923) find
  48. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, and 3 others. 2020. Superglue: A stickier benchmark for general-purpose language understanding systems. Preprint, arXiv:1905.00537.
  49. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. Preprint, arXiv:1804.07461.
  50. Yicheng Wang, Jiayi Yuan, Yu-Neng Chuang, Zhuoer Wang, Yingchi Liu, and 5 others. 2025. DHP benchmark: Are llms good NLG evaluators? In Findings of the ACL: NAACL 2025, Albuquerque USA, April 29 - May 4, 2025, pages 8079–8094. ACL. find
  51. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, and 2 others. 2023. Blimp: The benchmark of linguistic minimal pairs for english. Preprint, arXiv:1912.00582. — [arXiv:1912.00582](https://arxiv.org/abs/1912.00582) find
  52. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, and 4 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS 2022.
  53. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, and 36 others. 2024. Qwen2.5 technical report. CoRR, abs/2412.15115. find
  54. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, and 7 others. 2025. Justice or prejudice? quantifying biases in llm-as-a-judge. In ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.
  55. Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. 2023. Don’t trust chatgpt when your question is not in english: A study of multilingual abilities and types of llms. Preprint, arXiv:2305.16339. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vieira-2026-alba-european-portuguese-benchmark-evaluating/