Elamur Research⌕
☰
home / papers / 2025 / TounsiBench: Benchmarking Large Language Models for Tunisian Arabic

TounsiBench: Benchmarking Large Language Models for Tunisian Arabic

Paper · 2025 · Souha Ben Hassine, Asma Arrak, Marouene Addhoum et al.

Summary

Tunisian Arabic (Derja) is spoken by more than 12 million people. It differs strongly from MSA, borrows heavily from French, Italian, Turkish and Berber, and has no fixed spelling. Until this paper, nobody had systematically checked how well chat LLMs handle it.

The authors built a set of 744 Tunisian Arabic instructions. Some were collected from native speakers describing how they would actually query a chatbot. The rest were rendered into Derja from an existing English instruction set. Native-speaker authors also wrote a reference answer for every item. Ten Arabic-capable LLMs were told to answer strictly in Tunisian Arabic. Their outputs were then compared in pairs by human annotators and by LLM judges on four axes: quality, correctness, relevance and dialectal adherence. GPT-4o tracked human preferences most closely, so it was adopted as the automatic judge for the final leaderboard.

The central finding is that most models fail to recognise the dialect or to stay within it. Data, gold answers and evaluation code are public, so other teams can test their own models.

Key points

  • Dataset composition: 744 instructions in total.
    • 417 were crowdsourced from 48 native speakers (32 female, 16 male, aged 15–48, from various regions).
    • 327 were translated from the HuggingFace Helpful Instructions dataset.
    • Every item carries topic labels and a human-written gold response.
  • Models tested: GPT-4o, GPT-4o-mini, GPT-4, GPT-3.5 Turbo, LLaMA-3.3-70B, LLaMA-3.1-8B-Instruct, Aya-23-8B, jais-13B-chat, SILMA-9B-Instruct-v1.0 and Mixtral-8x7B-Instruct-v0.1.
  • Evaluation design: all 45 possible model pairs were compared, giving 900 pairwise evaluation instances. The order of the two responses was randomised to counter position bias.
  • Length bias:
    • Humans chose the longer answer 45% of the time.
    • Among LLM judges, the rate ranged from 47% (Aya-23-8B) and 64% (GPT-4o) up to 75% (Mixtral).
    • Yet GPT-4o's rankings correlated only weakly with average response length (Spearman 0.42 for quality down to 0.115 for Tunisian usage).
  • Other judge and model failures:
    • Models picked their own outputs more often than humans did. The gap was largest for Mixtral (78 vs 29) and LLaMA-3.1-8B (94 vs 46).
    • Annotators flagged Mixtral 30 times out of 180 for answers that were not in Arabic at all.

Interesting ideas

  • I1. Before trusting an automatic judge at scale, the authors scored several candidate judges against human ratings and kept the one that agreed best; this turns "which judge?" into a measured choice rather than an assumption.
  • I2. A self-preference audit compares how often a model picks its own output with how often humans pick it, exposing judges that reward their own style.
  • I3. Length bias was checked in two ways: the share of "longer wins" per judge, and the correlation between final rankings and mean response length.
  • I4. Language/dialect adherence was scored as its own criterion, separate from correctness and relevance, so a factually right answer in the wrong variety still loses points.
  • I5. The benchmark mixes genuinely user-written queries with adapted items from an existing dataset, balancing realism against coverage.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Each LLM teacher tier could be certified by its agreement with a small hand-checked gold set before its labels are allowed into training data.
  • I2 (learn): When witness parsers vote on silver trees, we could measure whether any witness systematically backs analyses identical to its own output and down-weight it.
  • I3 (answer): When grading Mova's readers on school tests or fable QA, a length-only baseline could serve as a negative control, ensuring verbose answers are not mistaken for better ones.
  • I4 (back): Generation from the meaning graph could be scored separately on content fidelity and on register or style, revealing cases where facts survive but the wording drifts.
  • I5 (answer): Test sets for the readers could combine naturally phrased questions from real users with converted standard items, catching failures that clean curated questions hide.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Oakland UniversityRochester · United States7 works · 180 citations🎓University of Michigan–FlintFlint · United States3 works · 1 citation

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Souha Ben Hassine, Asma Arrak, Marouene Addhoum and Steven R. Wilson. 2025. TounsiBench: Benchmarking Large Language Models for Tunisian Arabic. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.1756

@inproceedings{hassine2025tounsibench,
  title = {{TounsiBench: Benchmarking Large Language Models for Tunisian Arabic}},
  author = {Souha Ben Hassine, Asma Arrak, Marouene Addhoum and Steven R. Wilson},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1756},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1756}
}

References

31 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2025. Demographics of tunisia. find
  2. David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba Oluwadara Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Ijeoma Chukwuneke, Happy Buzaaba, Blessing Kudzaishe Sibanda, Godson Koffi Kalipe, Jonathan Mukiibi, Salomon Kabongo Kabenamualu, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela, Nkiruka Odu, Rooweither Mabuya, Salomey Osei, Shamsuddeen Hassan Muhammad, Sokhar Samb, Tadesse Kebede Guge, Tombekai Vangoni Sherman, and Pontus Stenetorp. 2025. IrokoBench: A new benchmark for African languages in the age of large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2732–2757, Albuquerque, New Mexico. Association for Computational Linguistics. ↗
  3. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475. ↗
  4. Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Kelly Marchisio, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. 2024. Aya 23: Open weight releases to further multilingual progress. Preprint, arXiv:2405.15032. ↗
  5. Elisa Gugliotta, Marco Dinarelli, and Olivier Kraif. 2020. Multi-task sequence prediction for tunisian arabizi multi-level annotation. arXiv preprint arXiv:2011.05152. ↗
  6. Gagan Bhatia, El Moatez Billah Nagoudi, Abdellah El Mekki, Fakhraddin Alwajih, and Muhammad Abdul-Mageed. 2024. Swan and arabicmteb: Dialect-aware, arabic-centric, cross-lingual, and cross-cultural embedding models and benchmarks. arXiv preprint arXiv:2411.01192. ↗
  7. Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, and Marzieh Fadaee. 2023. Elo uncovered: Robustness and best practices in language model evaluation. In Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 339–352, Singapore. Association for Computational Linguistics. find
  8. Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics. find
  9. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132. ↗
  10. Daniel Deutsch, Rotem Dror, and Dan Roth. 2022. On the limitations of reference-free evaluations of generated text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10960–10977, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  11. David M. Eberhard, Gary F. Simons, and Charles D. Fennig. 2024. Ethnologue: Languages of the world -tunisian arabic (aeb). Accessed: 2025-02-14. find
  12. Arpad E Elo and Sam Sloan. 1978. The rating of chessplayers: Past and present. find
  13. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166. ↗
  14. Imane Guellil, Houda Saâdane, Faical Azouaou, Billel Gueni, and Damien Nouvel. 2021. Arabic natural language processing: An overview. Journal of King Saud University-Computer and Information Sciences, 33(5):497–507.
  15. Hatem Haddad, Hala Mulki, and Asma Oueslati. 2019. T-hsab: A tunisian hate speech and abusive dataset. In International conference on Arabic language processing, pages 251–263. Springer. find
  16. Hatem Haddad, Ahmed Cheikh Rouhou, Abir Messaoudi, Abir Korched, Chayma Fourati, Amel Sellami, Moez Ben HajHmida, and Faten Ghriss. 2023. Tunbert: pretraining bert for tunisian dialect understanding. SN Computer Science, 4(2):194. ↗
  17. Tannon Kew, Florian Schottmann, and Rico Sennrich. 2024. Turning English-centric LLMs into polyglots: How much multilinguality is needed? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13097–13124, Miami, Florida, USA. Association for Computational Linguistics. find
  18. Md Tawkat Islam Khondaker, Numaan Naeem, Fatimah Khan, Abdelrahim Elmadany, and Muhammad Abdul-Mageed. 2024. Benchmarking llama-3 on arabic language generation tasks. In Proceedings of The Second Arabic Natural Language Processing Conference, pages 283–297.
  19. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2023. Benchmarking cognitive biases in large language models as evaluators. arXiv preprint arXiv:2309.17012. ↗
  20. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939. ↗
  21. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval. ↗
  22. Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. 2024. Wildbench: Benchmarking llms with challenging tasks from real users in the wild. arXiv preprint arXiv:2406.04770. ↗
  23. Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024. How do large language models handle multilingualism? arXiv preprint arXiv:2402.18815. ↗
  24. AI @ Meta Llama Team. 2024. The llama 3 herd of models.
  25. Salima Mdhaffar, Haroun Elleuch, Fethi Bougares, and Yannick Estève. 2024. Performance analysis of speech encoders for low-resource slu and asr in tunisian dialect. arXiv preprint arXiv:2407.04533. ↗
  26. MistralAI. 2024. Mixtral-8x7b-instruct-v0.1. find
  27. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.
  28. Hala Mulki, Hatem Haddad, Chedi Bechikh Ali, and Ismail Babaoğlu. 2018. Tunisian dialect sentiment analysis: a natural language processing-based approach. Computación y Sistemas, 22(4):1223–1232. ↗
  29. Fatiha Sadat, Fatma Mallek, Mohamed Mahdi Boud-abous, Rahma Sellami, and Atefeh Farzindar. 2014. Collaboratively constructed linguistic resources for language variants and their exploitation in nlp application–the case of tunisian arabic and the social media. In Proceedings of workshop on Lexical and grammatical resources for language processing, pages 102–110. ↗
  30. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Alham Fikri Aji, Zhengzhong Liu, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Preslav Nakov, Timothy Baldwin, and Eric Xing. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. Preprint, arXiv:2308.16149. ↗
  31. Uri Shaham, Jonathan Herzig, Roee Aharoni, Idan Szpektor, Reut Tsarfaty, and Matan Eyal. 2024. Multilingual instruction tuning with just a pinch of multilinguality. arXiv preprint arXiv:2401.01854. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hassine-2025-tounsibench-benchmarking-large-language-models/