Elamur Research⌕
☰
home / papers / 2026 / GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek

GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek

Paper · 2026 · Yang Zhang, Mersin Konomi, Christos Xypolopoulos et al.

Summary

Greek is often present in the training data of multilingual LLMs, but there are few trustworthy ways to measure how well these models handle it. Most existing tests are machine translations of English datasets, which carry translation artifacts and US-centric cultural assumptions. The authors address this with GreekMMLU, a multiple-choice benchmark built only from questions originally written in Greek. The questions come from openly licensed school, university, vocational, civil-service, citizenship and driving examinations. They are organized into a new taxonomy of 45 subjects and labeled by educational level. Part of the data is held back for a private leaderboard to limit test contamination. The team tested more than 80 open and proprietary models. Commercial frontier systems clearly lead open-weight ones, and accuracy rises with model size. Topics tied to Greek culture are hardest for generic multilingual models, while Greek- and Europe-focused models handle them better. The resource gives Greek NLP a native, culturally grounded yardstick. It is also a template for other under-resourced languages, Ukrainian among them.

Key points

  • Size and structure: 21,805 questions across 45 subjects, grouped into STEM, Humanities, Social Sciences and Other.
    • Eight subjects are specifically Greek, such as Greek History, Greek Mythology and Greek Traditions.
    • Each question has 2–4 options with exactly one correct answer.
    • Difficulty levels run from primary school to professional exams.
  • Public and private splits: 16,857 items are publicly released and 4,948 are reserved for a hidden leaderboard hosted on Hugging Face Spaces.
  • Scale and prompting: Models below roughly 2B parameters perform near chance. Five-shot prompting helps mid-size and large models, notably the Gemma and Llama families, but not the smallest ones.
  • Greek-adapted models: Llama-Krikri-8B, Meltemi and EuroLLM are compared with general models such as Qwen, Gemma, GPT-5.2 and Gemini 3 Flash. The Greek-adapted models show consistent advantages on culturally specific subjects.
  • Calibration: The correlation between confidence and accuracy is strongest for Llama-Krikri-8B (r = 0.93). It is weaker for Qwen2.5-7B (r = 0.74) and very poor for Llama-2-7B (r = 0.13). Question length barely affects model confidence.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates138 works · 6,590 citations🎓École PolytechniquePalaiseau · France15 works · 6,688 citations🎓National Technical University of AthensAthens · Greece8 works · 461 citations🔬Laboratoire d'Informatique de l'École PolytechniquePalaiseau · France3 works · 105 citations🎓University of IoanninaIoannina · Greece2 works · 2 citations

Also on the byline: University of Peloponnese.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Yang Zhang, Mersin Konomi, Christos Xypolopoulos, Konstantinos Divriotis, Konstantinos Skianis, Giannis Nikolentzos, Giorgos Stamou, Guokan Shang and Michalis Vazirgiannis. 2026. GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.448

@inproceedings{zhang2026greekmmlu,
  title = {{GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek}},
  author = {Yang Zhang, Mersin Konomi, Christos Xypolopoulos, Konstantinos Divriotis, Konstantinos Skianis, Giannis Nikolentzos, Giorgos Stamou, Guokan Shang and Michalis Vazirgiannis},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.448},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.448}
}

References

1 references, 1 of them with a card of ours. ↗ leads to the work.

  1. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-greekmmlu-native-sourced-multitask-benchmark/