Elamur Research⌕
☰
home / papers / 2025 / MarsadLab at PalmX Shared Task: An LLM Benchmark for Arabic Culture and Islamic Civilization

MarsadLab at PalmX Shared Task: An LLM Benchmark for Arabic Culture and Islamic Civilization

Paper · 2025 · Md. Rafiul Biswas, Shimaa Ibrahim, Kais Attia et al.

Summary

The paper describes a MarsadLab entry to PalmX 2025, a shared task that tests how well LLMs handle Arabic cultural and Islamic knowledge through four-option multiple-choice questions in Modern Standard Arabic. The authors first compared several open models: NileChat-3B, LLaMA3.1 8B, Fanar and Qwen2.5-7B-Instruct. They then adapted the strongest one, Qwen2.5-7B-Instruct, with LoRA on top of a 4-bit NF4 quantized backbone and trained only on the official training splits. Each training example is a chat-formatted prompt, and the target is just the correct option letter. Ablations varied LoRA rank, the modules being adapted, and the prompt style. The fine-tuned model reached 67.55% on General Culture and 74.13% on Islamic Culture, placing 7th and 4th on the blind test sets. The whole setup runs on a single consumer GPU with about 8 GB of memory, which shows that cheap parameter-efficient tuning can noticeably improve culture-specific QA. The distance to the leading systems, especially on the Islamic subtask, suggests that more knowledge grounding is still needed.

Key points

  • Data: the General Culture task has 4,500 questions about 22 Arab countries (2,000 train / 500 dev / 2,000 test). The Islamic Culture task has 1,900 questions (600 / 300 / 1,000). No external data augmentation was used.
  • Setup: Qwen2.5-7B-Instruct with LoRA (rank 16, α = 32, dropout 0.1) over all attention and FFN projections. Training ran for 3 epochs at learning rate 2×10⁻⁴ with an effective batch of 16 and maximum length 512. It took about 3 hours on one RTX 3090.
  • Fine-tuning gains for Qwen were +7.75 points on Task 1 (59.80 → 67.55) and +8.73 points on Task 2 (65.40 → 74.13). The top systems scored 72.15% and 84.22%, leaving gaps of 4.60 and 10.09 points.
  • Ablations:
    • Rank 32 was only marginally better than rank 16 but used twice the parameters (41.94M vs 20.97M).
    • Adapting attention alone beat adapting FFN alone, and adapting both was best.
    • Expert-persona framing was the best prompt (67.6 / 74.1). Adding few-shot examples lowered scores slightly.
  • Each subtask gets its own LoRA adapter on a shared quantized base, so adapters can be swapped at inference. An adapter checkpoint is about 1.2 GB, versus about 15 GB for the full model.

Interesting ideas

  • I1. The supervision target is only the answer letter, which removes verbose output, keeps labels consistent and makes answer extraction trivial.
  • I2. One frozen backbone carries several small task-specific adapters that can be hot-swapped, which avoids negative transfer between domains while keeping a single deployable model.
  • I3. The rank ablation reports accuracy together with trainable parameters and training hours, which makes the point of diminishing returns visible.
  • I4. Prompt add-ons that seem helpful, such as few-shot examples, were tested against a simpler instruction and turned out slightly harmful once the model was tuned.
  • I5. Comparing where adaptation helps (attention versus FFN modules) is used as a cheap probe of which part of the network the task depends on.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-test and fable-QA readers could produce a constrained answer slot (option ID or entity ID) alongside their explanation trace, so scoring never depends on parsing free text.
  • I2 (tree): gparse could keep one shared encoder plus lightweight adapters per treebank genre or UD dialect, switching them according to input type without retraining the core.
  • I3 (learn): Training runs could always log cost next to the score, so model size or feature-set choices are justified by measured marginal gains.
  • I4 (learn): Each LLM teacher tier's prompt elements (persona, examples, reasoning requests) should be ablated individually against the silver-data agreement rate instead of being assumed useful.
  • I5 (tree): A similar module-wise ablation on gparse could show whether errors on specific relations come from attention patterns or feed-forward layers, guiding targeted repair.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Northwestern University in QatarDoha · Qatar24 works · 148 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Md. Rafiul Biswas, Shimaa Ibrahim, Kais Attia, Firoj Alam and Wajdi Zaghouani. 2025. MarsadLab at PalmX Shared Task: An LLM Benchmark for Arabic Culture and Islamic Civilization. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.112

@inproceedings{biswas2025marsadlab,
  title = {{MarsadLab at PalmX Shared Task: An LLM Benchmark for Arabic Culture and Islamic Civilization}},
  author = {Md. Rafiul Biswas, Shimaa Ibrahim, Kais Attia, Firoj Alam and Wajdi Zaghouani},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.112},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.112}
}

References

28 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. This paper presented MarsadLab’s approach to the PalmX 2025 shared task on Arabic Islamic and Cultural understanding. Through parameter-efficient fine-tuning using LoRA adaptation of Qwen2.5-7B-Instruct, we achieved competitive performance across both tasks. Our work demonstrates that parameter-efficient methods can effectively adapt LLMs for culturally-nuanced tasks without requiring extensive computational resources. By training only 0.27% of model parameters through LoRA while employing 4-bit quantization, we reduced memory requirements by approximately 75% compared to full fine-tuning, making our approach ac-Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Youssef Elshahawy, Ahmed Ali, Nadir Durrani, Natasa Milic-Frayling, and Firoj Alam. 2024. LAraBench: Benchmarking Arabic AI with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 487–520, St. Julian’s, Malta. Association for Computational Linguistics.
  2. Rawan Al-Matham, Kareem Darwish, Raghad Al-Rasheed, Waad Alshammari, Muneera Alhoshan, Amal Almazrua, Asma Al Wazrah, Mais Alheraki, Firoj Alam, Preslav Nakov, and 1 others. 2025. BAL-SAM: A platform for benchmarking arabic large language models. arXiv preprint arXiv:2507.22603. ↗
  3. F. Alam, Md Asif Hasan, S. R. Laskar, M. Kutlu, Kareem Darwish, and S. A. Chowdhury. 2025. NativQA framework: Enabling llms with native, local, and everyday knowledge. arXiv preprint arXiv:2504.05995. ↗
  4. Badr AlKhamissi, Muhammad ElNokrashy, Mai Alkhamissi, and Mona Diab. 2024. Investigating cultural alignment of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12404–12422, Bangkok, Thailand. Association for Computational Linguistics.
  5. Fakhraddin Alwajih, Gagan Bhatia, and Muhammad Abdul-Mageed. 2024. Dallah: A dialect-aware multimodal large language model for arabic. In Proceedings of The Second Arabic Natural Language Processing Conference, pages 320–336.
  6. Fakhraddin Alwajih, Abdellah El Mekki, Hamdy Mubarak, Majd Hawasly, Abubakr Mohamed, and Muhammad Abdul-Mageed. 2025a. PalmX 2025: The first shared task on benchmarking LLMs on arabic culture. In Proceedings of the Third Arabic Natural Language Processing Conference (ANLP 2025), Suzhou, China. Association for Computational Linguistics. Fakhraddin Alwajih, Abdellah El Mekki, Samar Mohamed Magdy, Abdelrahim A Elmadany, Omer Nacar, El Moatez Billah Nagoudi, Reem Abdel-Salam, Hanin Atwany, Youssef Nafea, Abdulfattah Mohammed Yahya, and 1 others. 2025b. Palm: A culturally inclusive and linguistically diverse dataset for arabic llms. arXiv preprint arXiv:2503.00151. ↗
  7. Shane Arora, Marzena Karpinska, Hung-Ting Chen, Ipsita Bhattacharjee, Mohit Iyyer, and Eunsol Choi. 2024. CaLMQA: Exploring culturally specific long-form question answering across 23 languages. arXiv preprint arXiv:2406.17761. ↗
  8. Lama Ayash, Hassan Alhuzali, Ashwag Alasmari, and Sultan Aloufi. 2025. Saudiculture: A benchmark for evaluating large language models’ cultural competence within saudi arabia. Journal of King Saud University Computer and Information Sciences, 37(6):123. ↗
  9. Fahim Dalvi, Maram Hasanain, Sabri Boughorbel, Basel Mousi, Samir Abdaljalil, Nizi Nazar, Ahmed Abdelali, Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Ali, Majd Hawasly, Nadir Durrani, and Firoj Alam. 2024. LLMeBench: A flexible framework for accelerating LLMs benchmarking. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 214–222, St. Julians, Malta. Association for Computational Linguistics. ↗
  10. Rocktim Das, Simeon Hristov, Haonan Li, Dimitar Dimitrov, Ivan Koychev, and Preslav Nakov. 2024. EXAMS-V: A multi-discipline multilingual multimodal exam benchmark for evaluating vision language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7768–7791, Bangkok, Thailand. Association for Computational Linguistics. ↗
  11. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, and 37 others. 2021. The GEM benchmark: Natural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 96–120, Online. Association for Computational Linguistics. Md. find
  12. Arid Hasan, Maram Hasanain, Fatema Ahmad, Sahinur Rahman Laskar, Sunaya Upadhyay, Vrunda N Sukhadia, Mucahid Kutlu, Shammur Absar Chowdhury, and Firoj Alam. 2025. NativQA: Multilingual culturally-aligned natural query for LLMs. In Findings of the Association for Computational Linguistics: ACL 2025, pages 14886–14909, Vienna, Austria. Association for Computational Linguistics. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3. ↗
  13. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pages 4411–4421. PMLR. find
  14. Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. GPTAraEval: A comprehensive evaluation of ChatGPT on Arabic NLP. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 220–247, Singapore. Association for Computational Linguistics.
  15. Mohamed Bayan Kmainasi, Rakif Khan, Ali Ezzat Shahroor, Boushra Bendou, Maram Hasanain, and Firoj Alam. 2025. Native vs non-native language prompting: A comparative analysis. In Web Information Systems Engineering – WISE 2024, pages 406–420, Singapore. Springer Nature Singapore. find
  16. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, and 5 others. 2020. XGLUE: A new benchmark dataset for cross-lingual pre-training, understanding and generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6008–6018, Online. Association for Computational Linguistics.
  17. Abdellah El Mekki, Houdaifa Atou, Omer Nacar, Shady Shehata, and Muhammad Abdul-Mageed. 2025. Nilechat: Towards linguistically diverse and culturally aware llms for local communities. arXiv preprint arXiv:2505.18383. ↗
  18. Basel Mousi, Nadir Durrani, Fatema Ahmad, Md Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2024. Aradice: Benchmarks for dialectal and cultural capabilities in llms. arXiv preprint arXiv:2409.11404. ↗
  19. Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, and 1 others. 2024. BLEnD: A benchmark for llms on everyday knowledge in diverse cultures and languages. arXiv preprint arXiv:2406.09948. ↗
  20. El Moatez Billah Nagoudi, AbdelRahim Elmadany, Ahmed El-Shangiti, and Muhammad Abdul-Mageed. 2023. Dolphin: A challenging and diverse benchmark for Arabic NLG. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1404–1422, Singapore. Association for Computational Linguistics.
  21. Tarek Naous, Michael Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16366–16393, Bangkok, Thailand. Association for Computational Linguistics.
  22. Zhaozhi Qian, Faroq Altam, Muhammad Alqurishi, and Riad Souissi. 2024. Cameleval: Advancing culturally aligned arabic language models and benchmarks. arXiv preprint arXiv:2409.12623. ↗
  23. Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021. XTREME-R: Towards more challenging and nuanced multilingual evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10215–10245, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  24. A. Sadallah, J. C. Tonga, K. Almubarak, S. Almheiri, F. Atif, C. Qwaider, K. Kadaoui, S. Shatnawi, Y. Alesh, and F. Koto. 2025a. Commonsense reasoning in arab culture. Preprint, arXiv:2502.12788. ↗
  25. Abdelrahman Sadallah, Junior Cedric Tonga, Khalid Almubarak, Saeed Almheiri, Farah Atif, Chatrine Qwaider, Karima Kadaoui, Sara Shatnawi, Yaser Alesh, and Fajri Koto. 2025b. Commonsense reasoning in arab culture. arXiv preprint arXiv:2502.12788. ↗
  26. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, and 1 others. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, and 23 others. Fanar: An arabic-centric multimodal generative ai platform. ↗
  27. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. ↗
  28. Silvana Yakhni and Ali Chehab. 2025. Can llms translate cultural nuance in dialects? a case study on lebanese arabic. In Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script, pages 114–135.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/biswas-2025-marsadlab-palmx-shared-task-llm/