Elamur Research⌕
☰
home / papers / 2025 / ADAPT–MTU HAI at PalmX 2025: Leveraging Full and Parameter-Efficient LLM Fine-Tuning for Arabic Cultural QA

ADAPT–MTU HAI at PalmX 2025: Leveraging Full and Parameter-Efficient LLM Fine-Tuning for Arabic Cultural QA

Paper · 2025 · Shehenaz Hossain and Haithem Afli

Summary

The authors describe their PalmX 2025 entry, a shared task that tests whether LLMs handle Arabic cultural knowledge, posed as four-option multiple-choice questions in Modern Standard Arabic. The task has two parts: general culture (Subtask 1) and Islamic knowledge (Subtask 2). They chose a different adaptation strategy for each part.

  • Subtask 1: they fully fine-tuned NileChat-3B, a Qwen-2.5-based model tuned on Egyptian and Moroccan Arabic.
  • Subtask 2: they adapted ALLaM-7B-Instruct-preview with LoRA, loading the model in 8-bit so it fits on one consumer GPU. Its training set merged the small Islamic split with the general-culture training data.

Both systems were trained with plain causal language modeling on prompts that follow the organizers' official template, so the model outputs a single answer letter. The general-culture system ranked first and the Islamic system ranked third.

The work matters because it shows that modest, carefully formatted fine-tuning of mid-sized Arabic-centric models can be competitive on culture-heavy QA, even on limited hardware. However, the paper reports no ablations that separate the effect of each design choice.

Key points

  • Data:

    • Subtask 1 has 2,000 train, 500 dev and 2,000 blind test MCQs.
    • Subtask 2 has 300 dev and 1,000 test MCQs. Its training set combined 600 Subtask 2 items with the 2,000 Subtask 1 items.

  • NileChat-3B (Subtask 1): full fine-tuning on a single A100 (40GB) in BF16, with these settings:

    • 3 epochs, effective batch size 16, max length 512;
    • AdamW at LR 2e-5;
    • the checkpoint was chosen by validation loss.

  • ALLaM-7B (Subtask 2): LoRA on q_proj and v_proj only, run on an RTX 4090 (24GB), with these settings:

    • rank 16, alpha 32, dropout 0.05, under 1% of weights trained;
    • 8-bit loading via bitsandbytes and FP16 training;
    • 5 epochs, effective batch size 32, max length 256, LR 3e-5 with cosine decay.

  • Results:

    SystemDev accuracyBlind test accuracyRank
    Subtask 178.60%72.15%1st
    Subtask 275.60%82.52%3rd

  • Limitations named by the authors:

    • Full fine-tuning is costly.
    • Both datasets are narrow in scope.
    • Results are sensitive to prompt format and initialisation.

Interesting ideas

  • I1. Training examples were built in exactly the same template the official scorer reads, ending in the answer cue, so the model learns to emit one letter and nothing else; this removes parsing failures at evaluation time.
  • I2. The small, specialised Islamic training split was enlarged by merging in the larger general-culture set, on the grounds that both share question style and reasoning shape.
  • I3. The fine-tuning prompt included a step-by-step reasoning instruction that is missing at test time, assuming the cue shapes internal behaviour without breaking the plainer inference format.
  • I4. The error analysis surfaced a distinct failure type: on a composite option ("both B and C are correct"), the model picked one partially correct component.
  • I5. Dev and blind-test accuracy moved in opposite directions for the two systems, so dev scores alone would have misled about which setup generalised.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Mova's test readers could render each school-test item in one fixed, scorer-identical layout before reasoning, so that a wrong format never counts as a wrong answer.
  • I2 (learn): When a target domain (e.g., fable QA) has few labelled items, Mova could pool silver data from a structurally similar task and check on a held-out set whether pooling actually helps.
  • I3 (learn): LLM teacher tiers could be prompted with richer instructions than the student ever sees, and negative controls could measure whether this train/test prompt mismatch helps or hurts.
  • I4 (answer): The MCQ reader could detect compound options explicitly and verify each referenced component in the world model before choosing a single option.
  • I5 (learn): Mova's evaluation could always report a blind split next to the dev split, and flag large dev–test gaps as a signal of style shift rather than real progress.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Munster Technological UniversityCork · Ireland4 works · 11 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksQuestion answering · 441 works

Cite

Cite as Shehenaz Hossain and Haithem Afli. 2025. ADAPT–MTU HAI at PalmX 2025: Leveraging Full and Parameter-Efficient LLM Fine-Tuning for Arabic Cultural QA. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.110

@inproceedings{hossain2025adapt,
  title = {{ADAPT–MTU HAI at PalmX 2025: Leveraging Full and Parameter-Efficient LLM Fine-Tuning for Arabic Cultural QA}},
  author = {Shehenaz Hossain and Haithem Afli},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.110},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.110}
}

References

35 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hani A. Aleid and Aqil M. Azmi. 2025. Hajj-fqa: A benchmark arabic dataset for developing question-answering systems on hajj fatwas. Journal of King Saud University – Computer and Information Sciences, 37(6):135. ↗
  2. Sarah Alnefaie, Eric Atwell, and Mohammad Ammar Alsalka. 2023. Is GPT-4 a good islamic expert for answering Quran questions? In Proceedings of the 35th Conference on Computational Linguistics and Speech Processing (ROCLING 2023), pages 124–133, Taipei City, Taiwan. The Association for Computational Linguistics and Chinese Language Processing (ACLCLP). find
  3. Fakhraddin Alwajih, Abdellah El Mekki, Hamdy Mubarak, Majd Hawasly, Abubakr Mohamed, and Muhammad Abdul-Mageed. 2025. PalmX 2025: The First Shared Task on Benchmarking LLMs on Arabic and Islamic Culture. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou, China. Association for Computational Linguistics. Co-located with EMNLP 2025, November 5–9.
  4. Ohoud Alyemny, Hend Al-Khalifa, and Abdulrahman Mirza. 2023. A data-driven exploration of a new islamic fatwas dataset for arabic nlp tasks. Data, 8(10). ↗
  5. Wissam Antoun, Fady Baly, and Hazem M. Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. CoRR, abs/2003.00104. ↗
  6. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan Al-Rashed, Faisal A. Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, Ahmed Abdelali, Jeril Kuriakose, Abdalghani Abujabal, Nora Al-Twairesh, Areeb Alowisheq, and Haidar Khan. 2024. Allam: Large language models for arabic and english. ↗
  7. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. CoRR, abs/2005.14165. ↗
  8. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T. Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R. El-Beltagy, Wassim El-Hajj, Mustafa Jarrar, and Hamdy Mubarak. 2021. A panoramic survey of natural language processing in the arab world. Commun. ACM, 64(4):72–81. ↗
  9. Gael de Francony, Victor Guichard, Praveen Joshi, Haithem Afli, and Abdessalam Bouchekif. 2019. Hierarchical deep learning for Arabic dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 249–253, Florence, Italy. Association for Computational Linguistics.
  10. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms.
  11. Ali Farghaly and Khaled Shaalan. 2009. Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing, 8(4). ↗
  12. Arash Ghafouri, Hasan Naderi, Mohammad Aghajani asl, and Mahdi Firouzmandi. 2023. Islamicpcqa: A dataset for persian multi-hop complex question answering in islamic text resources. find
  13. Nizar Y Habash. 2010. Introduction to Arabic natural language processing. Morgan & Claypool Publishers. ↗
  14. Shehenaz Hossain, Fouad Shammary, Bahaulddin Shammary, and Haithem Afli. 2025. Enhancing dialectal Arabic intent detection through cross-dialect multilingual input augmentation. In Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4), pages 44–49, Abu Dhabi, UAE. Association for Computational Linguistics.
  15. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.
  16. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Song Dingjie, Zhihong Chen, Mosen Alharthi, Bang An, Juncai He, Ziche Liu, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2024a. AceGPT, localizing large language models in Arabic. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8139–8163, Mexico City, Mexico. Association for Computational Linguistics. ↗
  17. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Juncai He, Ziche Liu, Zhiyi Zhang, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2024b. Acegpt, localizing large language models in arabic. ↗
  18. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large language models are zero-shot reasoners.
  19. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2022. find
  20. Qur’an QA 2022: Overview of the first shared task on question answering over the holy qur’an. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 79–87, Marseille, France. European Language Resources Association.
  21. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2023. find
  22. Qur’an QA 2023 shared task: Overview of passage retrieval and reading comprehension tasks over the holy qur’an. In Proceedings of ArabicNLP 2023, pages 690–701, Singapore (Hybrid). Association for Computational Linguistics.
  23. Abdellah El Mekki, Houdaifa Atou, Omer Nacar, Shady Shehata, and Muhammad Abdul-Mageed. 2025. Nilechat: Towards linguistically diverse and culturally aware llms for local communities. ↗
  24. Marryam Mohammed, Sama Ali, Salma Khaled, Ayad Majeed, and Ensaf Mohamed. 2025. Aftina: enhancing stability and preventing hallucination in ai-based islamic fatwa generation using llms and rag. Neural Computing and Applications, pages 1–26. ↗
  25. Ali Mostafa and Omar Mohamed. 2022. GOF at qur’an QA 2022: Towards an efficient question answering for the holy qu’ran in the Arabic language using deep learning-based approach. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 104–111, Marseille, France. European Language Resources Association. Basel Mousi, Nadir Durrani, Fatema Ahmad, Md.
  26. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2024. Aradice: Benchmarks for dialectal and cultural capabilities in llms.
  27. Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16366–16393, Bangkok, Thailand. Association for Computational Linguistics.
  28. Faiza Qamar, Seemab Latif, and Rabia Latif. 2024. A benchmark dataset with larger context for non-factoid question answering over islamic text. ↗
  29. Zhaozhi Qian, Faroq Altam, Muhammad Alqurishi, and Riad Souissi. 2024. Cameleval: Advancing culturally aligned arabic language models and benchmarks. ↗
  30. Zakia Saadaoui, Ghassen Tlig, and Fethi Jarray. 2024. Llms based approach for quranic question answering. pages 112–118. find
  31. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, Alham Fikri Aji, Zhiqiang Shen, Zhengzhong Liu, Natalia Vassilieva, Joel Hestness, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Hector Xuguang Ren, Preslav Nakov, Timothy Baldwin, and Eric Xing. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. ↗
  32. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, Mus’ab Husaini, Soon-Gyo Jung, Ji Kim Lucas, Walid Magdy, Safa Messaoud, Abubakr Mohamed, Tasnim Mohiuddin, Basel Mousi, Hamdy Mubarak, Ahmad Musleh, Zan Naeem, Mourad Ouzzani, Dorde Popovic, Amin Sadeghi, Husrev Taha Sencar, Mohammed Shinoy, Omar Sinan, Yifan Zhang, Ahmed Ali, Yassine El Kheir, Xiaosong Ma, and Chaoyi Ruan. 2025. Fanar: An arabic-centric multimodal generative ai platform. ↗
  33. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ↗
  34. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11903. ↗
  35. Imed Zitouni. 2011. Introduction to arabic natural language processing n. y. habash. morgan & claypool (synthesis lectures on human language technologies, vol. 10), 2010, xvii+167pp; isbn 978-1-59829-795-9, ebook isbn 978-1-59829-796-6. Comput. Linguist., 37(3):623–625. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hossain-2025-adapt-mtu-hai-palmx-2025/