Elamur Research⌕
☰
home / papers / 2025 / Testing Low-Resource Language Support in LLMs Using Language Proficiency Exams: the Case of Luxembourgish

Testing Low-Resource Language Support in LLMs Using Language Proficiency Exams: the Case of Luxembourgish

Paper · 2025 · Cedric Lothritz, Jordi Cabot and Laura Bernardy

Summary

Evaluation resources for Luxembourgish, a language with about 600,000 speakers, are scarce. Most LLMs are built mainly for English, so it is unclear how well they handle it. The authors ask whether official language proficiency exams can serve as a benchmark. They obtained unpublished exams from the Institut National des Langues Luxembourg (INLL), covering CEFR levels A1 to C2. Open fill-in-the-blank items were rewritten as multiple-choice questions so that every item could share one prompt. They then tested 53 LLMs zero-shot with an identical pipeline. To see whether exam results carry over to generation, they compared exam scores with performance on two LuxGen summarisation tasks. The largest systems, such as Claude and DeepSeek-R1, scored well across all levels. Smaller models often struggled even with beginner exams. Items needing knowledge specific to Luxembourgish, and items whose answer options looked alike, were the hardest. Exam scores correlated clearly with headline-generation quality. This suggests that exams are a cheap proxy for choosing a model, for example a locally deployable one when privacy rules forbid cloud services.

Key points

  • Exam data: 629 questions from INLL, roughly 100–114 per CEFR level. They fall into four categories: vocabulary (155), grammar (164), reading comprehension (155) and conversation comprehension (155). Conversation comprehension uses transcripts of listening tasks. About 17% of the items were originally open and were converted to MCQs. The dataset itself cannot be shared.
  • Generation tasks: two tasks from LuxGen (Plum et al., 2025).
    • Headline Generation on RTL news articles, using 300 of 13,852 test samples.
    • Short Description generation for Luxembourgish Wikipedia articles, with Wikidata descriptions as ground truth.
  • Pipeline: zero-shot prompting with an identical prompt for every model and samples in random order. Post-processing repairs minor typos. Answers that deviate strongly or match several options are counted as invalid.
  • Correlation with headline quality: across all models, Pearson correlations between overall exam score and headline quality were 0.837 (Claude-G), 0.794 (Claude-A), 0.667 (METEOR) and 0.622 (BERTScore). Within the group of large models alone, the correlations were much weaker (about 0.09–0.30).
  • Leaderboard: the authors publish an interactive public leaderboard on the LIST AI Sandbox and plan to extend it as new models appear.

Interesting ideas

  • I1. Graded human proficiency exams (A1→C2) work as a ready-made difficulty ladder, which shows not just whether a system fails but at what level it breaks down.
  • I2. The authors wrote distractors on purpose: near-miss spellings, wrongly conjugated verbs, wrong prepositions, and near-synonyms that do not fit the context. Each wrong option targets one specific competence.
  • I3. Before trusting a cheap proxy benchmark, the authors checked whether it predicts the expensive downstream task, and found the link weakens within the strongest group.
  • I4. Scoring is tolerant of small typos, but an answer that fits several options is ruled invalid rather than given credit.
  • I5. The authors advise dropping exam categories that are irrelevant to the target use, instead of letting them drag down the overall score.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-course tests could be organised by grade or difficulty level, so we can report the highest level each reader passes reliably instead of one flat accuracy.
  • I2 (learn): Evaluation sets could pair every gold item with distractors of this kind as negative controls, checking that the system rejects minimal-error variants rather than surface-similar guesses.
  • I3 (learn): We could correlate quick probe scores, such as parse accuracy on a small set, with end-task QA results, both across all model variants and within the top tier, before using the probe to pick checkpoints.
  • I4 (answer): Graders for word problems and fable QA could normalise trivial surface noise while explicitly counting answers that fit several options as wrong, so vague answers are never rewarded.
  • I5 (frames): Pragmatics evaluations, for example on speech acts, irony and coreference, could be scored per category so that each component is judged on the phenomena it is actually responsible for.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of LuxembourgLuxembourg · Luxembourg27 works · 337 citations🔬Luxembourg Institute of Science and TechnologyBelvaux · Luxembourg6 works · 9 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksLow-resource languages · 267 works

Cite

Cite as Cedric Lothritz, Jordi Cabot and Laura Bernardy. 2025. Testing Low-Resource Language Support in LLMs Using Language Proficiency Exams: the Case of Luxembourgish. arXiv:2504.01667.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2504.01667 · arXiv 2504.01667

@misc{lothritz2025testing,
  title = {{Testing Low-Resource Language Support in LLMs Using Language Proficiency Exams: the Case of Luxembourgish}},
  author = {Cedric Lothritz, Jordi Cabot and Laura Bernardy},
  year = {2025},
  doi = {10.48550/arxiv.2504.01667},
  eprint = {2504.01667},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.01667}
}

References

54 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ali Abbas, Mahad S Rehman, and Syed S Rehman. 2024. Comparing the performance of popular large language models on the national board of medical examiners sample questions. Cureus, 16(3). ↗
  2. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024a. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219. ↗
  3. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. 2024b. Phi-4 technical report. arXiv preprint arXiv:2412.08905. ↗
  4. Anthropic. 2024. Claude 3.5 Sonnet. https://www.anthropic.com/. Large Language Model. ↗
  5. Anthropic. 2025. Claude 3.7 sonnet. Anthropic. AI assistant. find
  6. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
  7. Tu Dinh, Carlos Mullov, Leonard Bärmann, Zhaolin Li, Danni Liu, Simon Reiß, Jueun Lee, Nathan Lerzer, Jianfeng Gao, Fabian Peller-Konrad, et al. 2024. Sciex: Benchmarking large language models on scientific exams with human expert grading and automatic grading. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11592–11610. ↗
  8. Michael E Ellis, K Mike Casey, and Geoffrey Hill. 2024. Chatgpt and python programming homework. Decision Sciences Journal of Innovative Education, 22(2):74–87. ↗
  9. Protection European Union. 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council. Regulation (eu), 679:2016. find
  10. Peter Gilles. 2006. Phonologie der n-tilgung im mosel-fränkischen (’eifler regel’). ein beitrag zur dialektolo-gischen prosodieforschung. find
  11. Google. 2024. Gemini 2.0 flash. Large language model. find
  12. Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Kelly Marchisio, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. 2024. Aya 23: Open weight releases to further multilingual progress. ↗
  13. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.
  14. Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al. 2024. Stable lm 2 1.6 b technical report. arXiv preprint arXiv:2402.17834. ↗
  15. Cohere For AI. 2024. c4ai-command-r-08-2024. find
  16. Council of Europe. 2001. Common european framework of reference for languages: Learning, teaching, assessment. find
  17. Xuan-Quy Dao, Ngoc-Bich Le, Xuan-Dung Phan, Bac-Bien Ngo, and TD Vo. 2023. Evaluation of chatgpt and microsoft bing ai chat performances on physics exams of vietnamese national high school graduation examination. arXiv Prepr. arXiv2306, 4538. find
  18. Roberts Darg̀is, Guntis Barzdins, Inguna Skadin , a, and Baiba Saulı̄te. 2024. Evaluating open-source llms in low-resource languages: Insights from latvian high school exams. In Proceedings of the 4th International Conference on Natural Language Processing for Digital Humanities, pages 289–293. find
  19. Jennifer Gotta, Quang Anh Le Hong, Vitali Koch, Leon D Gruenewald, Tobias Geyer, Simon S Martin, Jan-Erik Scholtz, Christian Booz, Daniel Pinto Dos Santos, Scherwin Mahmoudi, et al. 2024. Large language models (llms) in radiology exams for medical students: Performance and consequences. In RöFo-Fortschritte auf dem Gebiet der Röntgen-strahlen und der bildgebenden Verfahren. Georg Thieme Verlag KG. ↗
  20. Grand-Duché de Luxembourg. 2018. Loi du 1er août 2018 portant organisation de la Commission nationale pour la protection des données et mise en œuvre du règlement (UE) 2016/679. Journal officiel du Grand-Duché de Luxembourg, Mémorial A n°686. find
  21. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  22. Oskar Hallström, Amélie Chatelain, Clément Thiriet, Julien Séailles, Adrien Cavaillès, and Axel Marmet. 2023. Alfred-40b-1023. find
  23. Krzysztof Jassem, Michał Ciesiółka, Filip Graliński, Piotr Jabłoński, Jakub Pokrywka, Marek Kubis, Monika Jabłońska, and Ryszard Staruch. 2025. Llmzsz Ł: a comprehensive llm benchmark for polish. arXiv preprint arXiv:2501.02266. ↗
  24. Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. ↗
  25. Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2024. Gpt-4 passes the bar exam. Philosophical Transactions of the Royal Society A, 382(2270):20230254. ↗
  26. Fajri Koto. 2024. Cracking the code: Multi-domain llm evaluation on real-world professional exams in indonesia. arXiv preprint arXiv:2409.08564. ↗
  27. Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. 2023. Large language models only pass primary school exams in indonesia: A comprehensive test on indommlu. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12359–12374. find
  28. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2023. Bloom: A 176b-parameter open-access multilingual language model. ↗
  29. Nero Li, Shahar Broner, Yubin Kim, Katrina Mizuo, Elijah Sauder, Claire To, Albert Wang, Ofek Gila, and Michael Shindler. 2025. Investigating the capabilities of generative ai in solving data structures, algorithms, and computability problems. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1, pages 659–665. find
  30. Marcelo Sartori Locatelli, Matheus Prado Miranda, Igor Joaquim da Silva Costa, Matheus Torres Prates, Victor Thomé, Mateus Zaparoli Monteiro, Tomas Lacerda, Adriana Pagano, Eduardo Rios Neto, Wagner Meira Jr, et al. 2024. Examining the behavior of llm architectures within the framework of standardized national exams in brazil. In Proceedings of the 2024 AAAI/ACM Conference on AI, Ethics, and Society, pages 879–890. find
  31. Cedric Lothritz, Bertrand Lebichot, Kevin Allix, Lisa Veiber, Tegawendé François D Assise Bissyande, Jacques Klein, Andrey Boytsov, Anne Goujon, and Clément Lefebvre. 2022. Luxembert: Simple and practical data augmentation in language model pre-training for luxembourgish. In 13th Language Resources and Evaluation Conference (LREC 2022). find
  32. Eric Martínez. 2024. Re-evaluating gpt-4’s bar exam performance. Artificial intelligence and law, pages 1–24. ↗
  33. Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M Guerreiro, Ricardo Rei, Duarte M Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, et al. 2024. Eurollm: Multilingual language models for europe. arXiv preprint arXiv:2409.16235. ↗
  34. Marina Mayor-Rocher, Nina Melero, Elena Merino-Gómez, María Grandury, Javier Conde, and Pedro Reviriego. 2024. Evaluating large language models with tests of spanish as a foreign language: Pass or fail? arXiv preprint arXiv:2409.15334. ↗
  35. Fabio Mercorio, Mario Mezzanzanica, Daniele PotertĂŹ, Antonio Serino, and Andrea Seveso. 2024. Disce aut deficere: Evaluating llms proficiency on the invalsi italian benchmark. arXiv preprint arXiv:2406.17535. ↗
  36. Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, and et al. 2024. Gemma. find
  37. Mistral AI. 2023a. Lechat. Large language model. find
  38. Mistral AI. 2023b. Mistral 7b. Large language model. find
  39. Mistral AI. 2024. Mixtral 8x7b. Large language model. find
  40. OpenAI. 2023. Chatgpt-3.5. Large language model. Version: ChatGPT-3.5, URL: https://openai.com/chatgpt. ↗
  41. OpenAI. 2024a. Chatgpt-4o. Large language model. Version: ChatGPT-4o, URL: https://openai.com/ chatgpt. ↗
  42. OpenAI. 2024b. Chatgpt-4o mini. Large language model. Version: ChatGPT-4o Mini, URL: https://openai.com/chatgpt. ↗
  43. Alistair Plum, Tharindu Ranasinghe, and Christoph Purschke. 2025. Text generation models for luxembourgish with limited data: A balanced multilingual strategy. VarDial 2025, page 93. ↗
  44. Qwen. 2025. Qwq-32b: Embracing the power of reinforcement learning. find
  45. Mehmet Fatih Şahin, Çağrı Doğan, Erdem Can Top-kaç, Serkan Şeramet, Furkan Batuhan Tuncer, and Cenk Murat Yazıcı. 2025. Which current chatbot is more competent in urological theoretical knowledge? a comparative analysis by the european board of urology in-service assessment. World Journal of Urology, 43(1):116. ↗
  46. Daniel Stribling, Yuxing Xia, Maha K Amer, Kiley S Graim, Connie J Mulligan, and Rolf Renne. 2024. The model student: Gpt-4 performance on graduate biomedical science exams. Scientific Reports, 14(1):5670. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ↗
  47. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. ↗
  48. Arya VarastehNezhad, Reza Tavasoli, Mostafa Masumi, and Fattaneh Taghiyareh. 2024. Llm performance assessment in computer science graduate entrance exams. In 2024 11th International Symposium on Telecommunications (IST), pages 232–237. IEEE. find
  49. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2024. WizardLM: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations. ↗
  50. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. 2024a. Qwen2 technical report. ↗
  51. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2024b. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. ↗
  52. Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools.
  53. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. ↗
  54. Hui Zong, Rongrong Wu, Jiaxue Cha, Jiao Wang, Erman Wu, Jiakun Li, Yi Zhou, Chi Zhang, Weizhe Feng, and Bairong Shen. 2024. Large language models in worldwide medical exams: Platform development and comprehensive analysis. Journal of Medical Internet Research, 26:e66114. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lothritz-2025-testing-low-resource-language-support/