Elamur Research⌕
☰
home / papers / 2026 / Charting the European LLM Benchmarking Landscape: A New Taxonomy and Registry

Charting the European LLM Benchmarking Landscape: A New Taxonomy and Registry

Paper · 2026 · Spela Vintar, Mojca Brglez, Taja Kuzman Pungeršek et al.

Summary

Evaluation of large language models is still heavily English-centric. The common workaround is machine-translating English benchmarks, but this carries over Western cultural assumptions and misses phenomena specific to individual languages. Native single-language suites avoid that problem but rarely allow cross-lingual comparison. The authors, from the Jožef Stefan Institute and the University of Ljubljana, first survey the field:

  • widely used English "parent" benchmarks (MMLU, SuperGLUE, HellaSwag, BIG-bench);
  • their parallel multilingual derivatives;
  • dynamic, contamination-resistant benchmarks such as LiveBench;
  • European language- and culture-specific suites.

From this survey they propose a new taxonomy for classifying benchmarks, built around multilingual and non-English use cases. They also introduce a pilot registry of European benchmarks that applies the taxonomy and documents each entry with a rich set of metadata descriptors. Finally, they give methodological recommendations for future benchmark design and review current trends. The work matters because a shared, structured catalogue could coordinate benchmark development across European languages and push evaluation toward greater linguistic and cultural sensitivity.

Key points

  • Global-MMLU analysis cited by the authors: 28% of MMLU questions need culturally sensitive knowledge, and 84.9% of geography questions concern North America or Europe.
  • Parallel benchmarks differ in how they were built. The EU20 sets (MMLU, HellaSwag, ARC, TruthfulQA, GSM8K) are machine-translated. FLORES-200, Belebele and BenchMax rely on professional translation or repeated revision.
  • Native and localized European examples include:
    • BenCzechMark: 50 tasks, 14 newly created, only about 10% of instances machine-translated;
    • ITALIC: 10,000 exam-based items across 12 domains;
    • HuLu (Hungarian), IberBench, IberoBench and SloBENCH.
  • CUS-QA (Czech, Ukrainian, Slovak) tests cultural competence with textual and visual questions. Each question is based on a Wikipedia article available in just one of the three languages.
  • Dialects remain hard. On DIALECT-COPA-style dialectal data, top proprietary models score only about halfway between random and optimal. On LiveBench, the best models stay below 80% accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Jožef Stefan InstituteLjubljana · Slovenia17 works · 2,165 citations🎓Jožef Stefan International Postgraduate SchoolLjubljana · Slovenia11 works · 1,776 citations🎓University of LjubljanaLjubljana · Slovenia13 works · 38,693 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 works

Cite

Cite as Spela Vintar, Mojca Brglez, Taja Kuzman Pungeršek and Nikola Ljubešić. 2026. Charting the European LLM Benchmarking Landscape: A New Taxonomy and Registry. Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4kixe3c9zmde

@inproceedings{vintar2026charting,
  title = {{Charting the European LLM Benchmarking Landscape: A New Taxonomy and Registry}},
  author = {Spela Vintar, Mojca Brglez, Taja Kuzman Pungeršek and Nikola Ljubešić},
  booktitle = {Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026},
  year = {2026},
  doi = {10.63317/4kixe3c9zmde},
  url = {https://doi.org/10.63317/4kixe3c9zmde}
}

References

52 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI Verify Foundation. 2023. Catalogue of LLM Evaluations. find
  2. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants. In Proceedings of the 62nd Annual M…
  3. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.
  4. Irene Baucells, Javier Aula-Blasco, Iria de Dios-Flores, Silvia Paniagua Suárez, Naiara Perez, Anna Salles, Susana Sotelo Docio, Júlia Falcão, Jose Javier Saiz, Robiert Sepulveda Torres, Jeremy Barnes, Pablo Gamallo, Aitor Gonzalez-Agirre, German Rigau, and Marta Villegas. 2025. IberoBench: A benchm…
  5. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2023. A survey on evaluation of large language models. arXiv preprint arXiv:2307.03109.
  6. Adrian-Gabriel Chifu, Goran Glavaš, Radu Tudor Ionescu, Nikola Ljubešić, Aleksandra Miletić, Filip Miletić, Yves Scherrer, and Ivan Vulić. 2024. VarDial evaluation campaign 2024: Commonsense reasoning in dialects and multi-label similar language identification. In Proceedings of the Eleventh Worksho… find
  7. Pavel Chizhov, Mattia Nee, Pierre-Carl Langlais, and Ivan P Yamshchikov. 2025. What the Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In Proceedings of the 2019 Co…
  8. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv preprint arXiv:1803.05457.
  9. Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2022. Fleurs: Few-shot learning evaluation of universal representations of speech. arXiv preprint arXiv:2205.12446.
  10. Sławomir Dadas, Małgorzata Grębowiec, Michał Perełkiewicz, and Rafał Poświata. 2025. Evaluating Polish linguistic and cultural competency in large language models. arXiv preprint arXiv:2503.00995. — [arXiv:2503.00995](https://arxiv.org/abs/2503.00995) find
  11. Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The CommitmentBank: Investigating projection in naturally occurring discourse. In Proceedings of Sinn und Bedeutung, volume 23, pages 107–124.
  12. Fahim Faisal, Orevaoghene Ahia, Aarohi Srivastava, Kabir Ahuja, David Chiang, Yulia Tsvetkov, and Antonios Anastasopoulos. 2024. DIALECT-BENCH: An NLP benchmark for dialects, varieties, and closely-related languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Ling… find
  13. Martin Fajcik, Martin Docekal, Jan Dolezal, Karel Ondrej, Karel Beneš, Jan Kapsa, Pavel Smrz, Alexander Polok, Michal Hradis, Zuzana Neverilova, et al. 2025. BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism. Transactions of the… find
  14. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1–9.
  15. José Ángel González, Ian Borrego Obrador, Álvaro Romo Herrero, Areg Mikael Sarvazyan, Mara Chinea-Ríos, Angelo Basile, and Marc Franco-Salvador. 2025. IberBench: LLM Evaluation on Iberian Languages. arXiv preprint arXiv:2504.16921. — [arXiv:2504.16921](https://arxiv.org/abs/2504.16921) find
  16. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computat… find
  17. Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong, et al. 2023. Evaluating Large Language Models: A Comprehensive Survey. arXiv preprint arXiv:2310.19736.
  18. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. Proceedings of the International Conference on Learning Representations (ICLR).
  19. Xu Huang, Wenhao Zhu, Hanxu Hu, Conghui He, Lei Li, Shujian Huang, and Fei Yuan. 2025. BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models. arXiv preprint arXiv:2502.07346. find
  20. Thomas Huber and Christina Niklaus. 2025. LLMs Meet Bloom’s Taxonomy: A Cognitive View on Large Language Model Evaluations. In Proceedings of the 31st International Conference on Computational Linguistics, pages 5211–5246. find
  21. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences. In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL).
  22. Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, and Jimmy Huang. 2024. A systematic survey and critical review on evaluating large language mode…
  23. Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd Schema Challenge. In 13th International Conference on the Principles of Knowledge Representation and Reasoning, KR 2012, pages 552–561. Institute of Electrical and Electronics Engineers Inc. find
  24. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hu… find
  25. Jindřich Libovickỳ, Jindřich Helcl, Andrei Manea, and Gianluca Vico. 2025. CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Dataset. arXiv preprint arXiv:2507.22752. find
  26. Noémi Ligeti-Nagy, Gergő Ferenczi, Enikő Héja, László János Laki, Noémi Vadász, Zijian Győző Yang, and Tamás Váradi. 2024. HuLU: Hungarian language understanding benchmark kit. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (… find
  27. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  28. Nikola Ljubešić, Nada Galant, Sonja Benčina, Jaka Čibej, Stefan Milosavljević, Peter Rupnik, and Taja Kuzman. 2024. DIALECT-COPA: Extending the standard translations of the COPA causal commonsense reasoning dataset to South Slavic dialects. In Proceedings of the Eleventh Workshop on NLP for Similar… find
  29. Nikola Ljubešić and Davor Lauc. 2021. BERTić-The Transformer Language Model for Bosnian, Croatian, Montenegrin and Serbian. In Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing, pages 37–42. find
  30. Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, Víctor Gutiérrez-Basulto, Yazmín Ibáñez García, Hwaran Lee, Shamsuddeen Hassan Muhammad, Kiwoong Park, Anar Sabuhi Rzayev, Nina White, Seid Muhie… — BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages
  31. Shiwen Ni, Guhong Chen, Shuaimin Li, Xuanang Chen, Siyi Li, Bingli Wang, Qiyao Wang, Xingjian Wang, Yifan Zhang, Liyang Fan, et al. 2025. A Survey on Large Language Model Benchmarks. arXiv preprint arXiv:2508.15361. — [arXiv:2508.15361](https://arxiv.org/abs/2508.15361) find
  32. Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum…
  33. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376, Online. Association…
  34. David MW Powers. 2020. Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation. arXiv preprint arXiv:2010.16061. — [arXiv:2010.16061](https://arxiv.org/abs/2010.16061) find
  35. Nishat Raihan, Antonios Anastasopoulos, and Marcos Zampieri. 2025. mHumanEval - a multilingual benchmark to evaluate large language models for code generation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Languag… find
  36. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning. In Proceedings of the 2011 AAAI Spring Symposium on Logical Formalizations of Commonsense Reasoning. find
  37. Andrea Seveso, Daniele Potertì, Edoardo Federici, Mario Mezzanzanica, and Fabio Mercorio. 2025. ITALIC: An Italian culture-aware natural language benchmark. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language T… find
  38. Jiatong Shi, Dan Berrebbi, William Chen, Ho-Lam Chung, En-Pei Hu, Wei-Ping Huang, Xuankai Chang, Shang-Wen Li, Abdelrahman Mohamed, Hung yi Lee, and Shinji Watanabe. 2023. Ml-superb: Multilingual speech universal performance benchmark. In Proceedings of the Annual Conference of the International Spe… find
  39. Dimitar Shterionov, Mirella De Sisto, Vincent Vandeghinste, Aoife Brady, Mathieu De Coster, Lorraine Leeson, Josep Blat, Frankie Picron, Marcello Paolo Scipioni, Aditya Parikh, et al. 2022. Sign language translation: Ongoing development, challenges and innovations in the signon project. In Proceedin… find
  40. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, et al. 2024. Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation. arXiv…
  41. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, et al. 2023. Beyond the imitation game: Quantify…
  42. Klaudia Thellmann, Bernhard Stadler, Michael Fromm, Jasper Schulze Buschhoff, Alex Jude, Fabio Barth, Johannes Leveling, Nicolas Flores-Herr, Joachim Köhler, René Jäkel, et al. 2024. Towards Multilingual LLM Evaluation for European Languages. arXiv preprint arXiv:2410.08928. — [arXiv:2410.08928](https://arxiv.org/abs/2410.08928) find
  43. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. Advances in neural information processing systems, 32.
  44. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.12885.
  45. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In Proceedings of the International Conference on Learning Representations. find
  46. Chengzhi Zhong, Fei Cheng, Qianying Liu, Junfeng Jiang, Zhen Wan, Chenhui Chu, Yugo Murawaki, and Sadao Kurohashi. 2024. Beyond English-centric LLMs: What language do multilingual language models think in? arXiv preprint arXiv:2408.10811.
  47. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. 2024. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. Advances in Neural Information Processing Systems, 37:95266–95290.
  48. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, et al. 2024. LiveBench: A Challenging, Contamination-Limited LLM Benchmark. arXiv preprint arXiv:2406.19314. find
  49. Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Aosong Feng, Dairui Liu, Yun Xing, Junjue Wang, Fan Gao, et al. 2025. MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation. arXiv preprint arXiv:2503.10497.
  50. Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Wei-Cheng Huang, Wei-Cheng Tseng, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li Wang, Zhaoheng Ni Ma, Benjamin Chen, Chih-Liang Chang, Kevin Lin, W… find
  51. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800. find
  52. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vintar-2026-charting-european-llm-benchmarking-landscape/