Elamur Research⌕
☰
home / papers / 2025 / ArabicSense: A Benchmark for Evaluating Commonsense Reasoning in Arabic with Large Language Models

ArabicSense: A Benchmark for Evaluating Commonsense Reasoning in Arabic with Large Language Models

Paper · 2025 · Salima Lamsiyah, Kamyar Zeinalipour, Samir El amrany et al.

Summary

Commonsense benchmarks exist mostly for English, and the few Arabic ones were machine- or hand-translated from English sets such as SemEval-2020 ComVE, ECQA and OMCS. Translation tends to lose cultural and linguistic fit. The authors build ArabicSense, which they present as the first Arabic commonsense benchmark created natively rather than translated.

They take opening sections of Arabic Wikipedia articles as seeds, focusing on the bolded keywords and collecting article metadata. Task-specific prompts make GPT-4 turn these seeds into examples, and humans then check the output. The benchmark has three tasks:

  • Task A: pick the nonsensical sentence from a pair of similar sentences.
  • Task B: pick the correct reason for the nonsense from three options.
  • Task C: write an explanation, scored with BERTScore.

Six encoder models and three causal LLMs were tested zero-shot and after fine-tuning. Untuned BERT-style models score around 0.33. Fine-tuning lifts them to roughly 0.75–0.87, and fine-tuned causal LLMs reach about 0.93–0.95 F1. The resource and code are public, which supports Arabic NLU evaluation and offers a recipe for other under-resourced languages.

Key points

  • Resources: the dataset is ArabicSense on Hugging Face. The fine-tuned models Mistral-7b-CS-AR, gemma2-9b-CS-AR and P-Llama3-8B are released, with code on GitHub.
  • Encoder models without fine-tuning: AraBERT v2, MarBERT, CamelBERT, ArabicBERT base/large and mBERT all sit near 0.33 accuracy on Tasks A and B.
  • Encoder models after fine-tuning:
    • On Task A, AraBERT v2 reaches F1 0.87 in Table 4; the abstract attributes this score to the second task.
    • On Task B, ArabicBERT large is best at F1 0.84.
  • Causal LLMs, Tasks A and B:
    • Zero-shot Gemma already scores well: F1 0.867 on A and 0.921 on B.
    • After fine-tuning, Mistral-7b leads Task A at 0.948 and Gemma leads Task B at 0.944.
    • The abstract quotes slightly different figures (95.5 / 94.8).
  • Task C (BERTScore F1 after fine-tuning):
    • LLaMA-3 scores 0.773, Mistral-7b 0.771 and Gemma 0.759.
    • Zero-shot scores were 0.656, 0.747 and 0.728 respectively.
    • Stated limitations are synthetic origin, possible bias, and no causal or temporal reasoning dimensions.

Interesting ideas

  • I1. A graded ladder of three tasks (spot the absurd statement, choose why it is absurd, then explain it in your own words) separates mere detection from real understanding of the violated fact.
  • I2. Seeding synthetic data from the lead paragraphs and bolded key terms of encyclopedia articles keeps generated items anchored to real entities and spread across many domains.
  • I3. Pairing two nearly identical sentences, only one of which is sensible, isolates the commonsense violation from surface wording.
  • I4. Reporting untuned baselines next to fine-tuned and zero-shot results shows that the benchmark is not trivially solvable and that gains come from training.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): A reader can be scored on all three rungs, with the explanation produced from the specific world-state conflict that caused the rejection.
  • I2 (learn): Similar seeds drawn from Wikidata or WordNet entries could drive an LLM teacher to produce grounded test items, rather than letting it invent topics freely.
  • I3 (world): Such minimal pairs make clean negative controls for the world builder, which should flag exactly the sentence whose entities or states clash with compiled knowledge.
  • I4 (learn): Every new evaluation set should ship with a deliberately uninformed baseline run, so improvements are measured against a known floor.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,426 worksSemitic languages · 2,908 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Salima Lamsiyah, Kamyar Zeinalipour, Samir El amrany, Matthias Brust, Marco Maggini, Pascal Bouvry and Christoph Schommer. 2025. ArabicSense: A Benchmark for Evaluating Commonsense Reasoning in Arabic with Large Language Models. Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{lamsiyah2025arabicsense,
  title = {{ArabicSense: A Benchmark for Evaluating Commonsense Reasoning in Arabic with Large Language Models}},
  author = {Salima Lamsiyah, Kamyar Zeinalipour, Samir {El amrany}, Matthias Brust, Marco Maggini, Pascal Bouvry and Christoph Schommer},
  booktitle = {Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4)},
  year = {2025},
  url = {https://aclanthology.org/2025.wacl-1.1/}
}

References

35 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  3. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234. ↗
  4. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  5. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. ↗
  6. Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, and Minlie Huang. 2020. A knowledge-enhanced pre-training model for commonsense story generation. Transactions of the Association for Computational Linguistics, 8:93–108. find
  7. Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. 2021. Explanations for CommonsenseQA: New Dataset and Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3050–3065, Online. Association for Computational Linguistics.
  8. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2023. Unnatural instructions: Tuning language models with (almost) no human labor. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14409–14428, Toronto, Canada. Association for Computational Linguistics.
  9. Saja Al-Tawalbeh and Mohammad Al-Smadi. 2020. A benchmark arabic dataset for commonsense explanation. arXiv preprint arXiv:2012.10251. ↗
  10. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. ↗
  11. Farah Alshanik, Ibrahim Al-Sharif, and Mohammad W Abdullah. 2023. Commonsense validation and explanation for arabic sentences. In International Conference on Emerging Trends and Applications in Artificial Intelligence, pages 101–112. Springer. find
  12. Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. (comet-) atomic 2020: On symbolic and neural commonsense knowledge graphs. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 6384–6392. ↗
  13. Lutz Prechelt. 2002. Early stopping-but when? In Neural Networks: Tricks of the trade, pages 55–69. Springer. find
  14. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online). International Committee for Computational Linguistics.
  15. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. ↗
  16. M Moneb Khaled, Aghyad Al Sayadi, and Ashraf Elnagar. 2023. Commonsense validation and explanation in arabic text: A comparative study using arabic bert models. In 2023 24th International Arab Conference on Information Technology (ACIT), pages 1–6. IEEE. find
  17. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463– 4473, Hong Kong, China. Association for Computational Linguistics. ↗
  18. Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019. KagNet: Knowledge-aware graph networks for commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2829–2839, Hong Kong, China. Association for Computational Linguistics. ↗
  19. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, volume 31. ↗
  20. Roderick JA Little. 1993. Statistical analysis of masked data. Journal of Official statistics, 9(2):407. find
  21. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958. find
  22. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950–1965.
  23. Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. 2019. Does it make sense? and why? a pilot study for sense making and explanation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4020–4026, Florence, Italy. Association for Computational Linguistics.
  24. Ye Liu, Yao Wan, Lifang He, Hao Peng, and S Yu Philip. 2021. Kg-bart: Knowledge graph-augmented bart for generative commonsense reasoning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 6418–6425. find
  25. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508, Toronto, Canada. Association for Computational Linguistics. ↗
  26. I Loshchilov. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. ↗
  27. Pratyush Maini, Skyler Seto, Richard Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. 2024. Rephrasing the web: A recipe for compute and data-efficient language modeling. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14044– 14072, Bangkok, Thailand. Association for Computational Linguistics. ↗
  28. Da Yin, Xiao Liu, Fan Yin, Ming Zhong, Hritik Bansal, Jiawei Han, and Kai-Wei Chang. 2023. Dynosaur: A dynamic growth paradigm for instruction-tuning data curation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4031–4047, Singapore. Association for Computational Linguistics. find
  29. Arindam Mitra, Luciano Del Corro, Guoqing Zheng, Shweti Mahajan, Dany Rouhana, Andres Codas, Yadong Lu, Wei-ge Chen, Olga Vrousgos, Corby Rosset, et al. 2024. Agentinstruct: Toward generative teaching with agentic flows. arXiv preprint arXiv:2407.03502. ↗
  30. Kamyar Zeinalipour, Achille Fusco, Asya Zanollo, Marco Maggini, and Marco Gori. 2024a. Harnessing llms for educational content-driven italian crossword generation. arXiv preprint arXiv:2411.16936. ↗
  31. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001, Florence, Italy. Association for Computational Linguistics. ↗
  32. Kamyar Zeinalipour, Yusuf Gökberk Keptiğ, Marco Maggini, and Marco Gori. 2024b. Automating turkish educational quiz generation using large language models. arXiv preprint arXiv:2406.03397. ↗
  33. Kamyar Zeinalipour, Yusuf Gökberk Keptiğ, Marco Maggini, Leonardo Rigutini, and Marco Gori. 2024c. A turkish educational crossword puzzle generator. In International Conference on Artificial Intelligence in Education, pages 226–233. Springer. find
  34. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. ↗
  35. Andrea Zugarini, Kamyar Zeinalipour, Surya Sai Kadali, Marco Maggini, Marco Gori, and Leonardo Rigutini. 2024. Clue-instruct: Text-based clue generation for educational crossword puzzles. arXiv preprint arXiv:2404.06186. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lamsiyah-2025-arabicsense-benchmark-evaluating-commonsense-reasoning/