Elamur Research⌕
☰
home / papers / 2025 / ISL-NLP at PalmX 2025: Retrieval-Augmented Fine-Tuning for Arabic Cultural Question Answering

ISL-NLP at PalmX 2025: Retrieval-Augmented Fine-Tuning for Arabic Cultural Question Answering

Paper · 2025 · Mohamed Gomaa and Noureldin Elmadany

Summary

Many LLMs handle Arabic cultural knowledge poorly. This PalmX 2025 Subtask 1 entry tackles four-option multiple-choice questions in Modern Standard Arabic on the history, geography, arts and customs of Arab countries. The system has two stages.

  1. Evidence generation. The Gemini 2.5 Pro API writes a short piece of evidence for every training question. The evidence is at most 50 words, covers cultural, historical or geographical background, and is meant to separate the right option from the distractors without naming it.
  2. Fine-tuning. NileChat-3B is fine-tuned on (question, context, answer) triplets. Only the q_proj, v_proj and gate_proj layers are updated, which the authors say cuts trainable parameters by 68.2% compared with full fine-tuning.

The evidence is used only during training. The authors argue the model absorbs cultural cues it can then apply to bare questions at test time.

On the development set the system scores 73.60% accuracy, against 70.00% for NileChat-3B. On the blind test set it ranks 6th, with about 67.6–67.7%; the paper gives both figures. The work suggests that modest compute, two Kaggle T4 GPUs, plus LLM-generated hints can noticeably improve a small Arabic-centric model.

Key points

  • Data: PalmX 2025 Subtask 1 has 2,000 training, 500 development and 2,000 blind-test MCQs, each with options A–D.

  • Development-set results (precision / recall / F1 / accuracy):

    ModelPre.RecallF1Acc.
    Qwen1.5-1.8B63.2460.8859.1559.80
    Qwen2.5-1.5B63.73*63.8963.5963.60
    NileChat-3B71.7470.0069.9270.00
    Their system73.8173.8873.5473.60

    *The paper lists Qwen2.5-1.5B precision as 64.73.

  • Adaptation: only q_proj, v_proj and gate_proj are trained in NileChat-3B, giving a 68.2% reduction in trainable parameters.

  • Training setup: PEFT with the TRL 0.8.2 SFTrainer and Transformers ≥4.41.0. Three epochs, batch size 2 per device, 8 gradient-accumulation steps, learning rate 2×10⁻⁴, AdamW, FP16, two T4 GPUs.

  • Release: the weights are published as MohamedGomaa30/Ibn-Al-Nafs. The system placed 6th on the blind test.

Interesting ideas

  • I1. Supporting evidence appears only in training examples and is absent at test time, so the model is pushed to internalise cues rather than read them off the input
  • I2. The evidence prompt asks for facts that set the correct option apart from the distractors while forbidding the answer itself, which reduces label leakage
  • I3. Each hint is capped at 50 words, which keeps the added context focused and cheap
  • I4. Only a few chosen projection layers are updated, to adapt the model while limiting catastrophic forgetting of general skills
  • I5. The paper asserts that training with context helps on context-free questions but reports no with/without-context ablation, which shows why such a control matters

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): LLM teacher tiers could attach short justifications to silver examples during training only, and evaluation would then check whether the student holds up without them.
  • I2 (answer): when building hints or worked material for school-course tests, the teacher could be asked for discriminating facts but not the answer, keeping the test meaningful.
  • I3 (world): background facts pulled from WordNet/Wikidata into a text's world could be held to a small budget, so the reasoning trace stays readable and explainable.
  • I4 (tree): gparse could be tuned to a new treebank or UD dialect by updating a small layer subset, protecting accuracy on the original treebanks.
  • I5 (learn): any teacher-provided signal should come with a negative-control run without it, to confirm the gain really comes from that signal.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Arab Academy for Science, Technology, and Maritime TransportAlexandria · Egypt5 works · 50 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksQuestion answering · 441 worksInformation retrieval & search · 473 works

Cite

Cite as Mohamed Gomaa and Noureldin Elmadany. 2025. ISL-NLP at PalmX 2025: Retrieval-Augmented Fine-Tuning for Arabic Cultural Question Answering. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.109

@inproceedings{gomaa2025isl,
  title = {{ISL-NLP at PalmX 2025: Retrieval-Augmented Fine-Tuning for Arabic Cultural Question Answering}},
  author = {Mohamed Gomaa and Noureldin Elmadany},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.109},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.109}
}

References

16 references, 13 of them with a card of ours. ↗ leads to the work.

  1. Fanar Team Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur A. Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed G. Elfeky, Ahmed K. Elmagarmid, Mohamed Y. Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, Mus’ab Husaini, Soon-Gyo Jung, Ji Lucas, Walid Magdy, Safa Messaoud, Abubakr Mohamed, Tasnim Mohiuddin, Basel Mousi, Hamdy Mubarak, Ahmad Musleh, Zan Ahmad Naeem, Mourad Ouzzani, Dorde Popovic, Amin Sadeghi, Husrev Taha Sencar, Mohammed Shinoy, Omar Sinan, Yifan Zhang, Ahmed Ali, Yassine El Kheir, Xiaosong Ma, and Chaoyi Ruan. 2025. Fanar: An arabic-centric multimodal generative ai platform. ArXiv, abs/2501.13944. ↗
  2. Badr AlKhamissi, Muhammad ElNokrashy, Mai Alkhamissi, and Mona Diab. 2024. Investigating cultural alignment of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12404–12422, Bangkok, Thailand. Association for Computational Linguistics.
  3. Fakhraddin Alwajih, Abdellah El Mekki, Hamdy Mubarak, Majd Hawasly, Abubakr Mohamed, and Muhammad Abdul-Mageed. 2025. PalmX 2025: The First Shared Task on Benchmarking LLMs on Arabic and Islamic Culture. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou, China. Association for Computational Linguistics. Co-located with EMNLP 2025, November 5–9.
  4. Dalvi, Shammur A. Chowdhury, and Firoj Alam. 2024. Aradice: Benchmarks for dialectal and cultural capabilities in llms. ArXiv, abs/2409.11404. ↗
  5. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham Abdullah Alyahya, Sultan Alrashed, Faisal Abdulrahman Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Majed Alrubaian, Ali Alammari, Zaki Alawami, AbdulMohsen O. Al-Thubaity, Ahmed Abdelali, Jeril Kuriakose, Abdalghani Abujabal, Nora Al-Twairesh, Areeb Alowisheq, and Haidar Khan. 2024. Allam: Large language models for arabic and english. ArXiv, abs/2407.15390. ↗
  6. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. ArXiv, abs/2009.03300. ↗
  7. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Ziche Liu, Zhiyi Zhang, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2023. Acegpt, localizing large language models in arabic. In North American Chapter of the Association for Computational Linguistics. ↗
  8. Amr Keleg. 2025. LLM alignment for the Arabs: A homogenous culture or diverse ones. In Proceedings of the 3rd Workshop on Cross-Cultural Considerations in NLP (C3NLP 2025), pages 1–9, Albuquerque, New Mexico. Association for Computational Linguistics.
  9. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5622–5640, Bangkok, Thailand. Association for Computational Linguistics. ↗
  10. Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Pérez-Almendros, Abinew Ali Ayele, V’ictor Guti’errez-Basulto, Yazm’in Ib’anez-Garc’ia, Hwaran Lee, Shamsuddeen Hassan Muhammad, Kiwoong Park, Anar Rzayev, Nina White, Seid Muhie Yimam, Mohammad Taher Pilehvar, Nedjma Djouhra Ousidhoum, José Camacho-Collados, and Alice Oh. 2024. Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages. ArXiv, abs/2406.09948. ↗
  11. Michael J Ryan, William Held, and Diyi Yang. 2024. Unintended impacts of LLM alignment on global representation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16121–16140, Bangkok, Thailand. Association for Computational Linguistics.
  12. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Arun Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Alham Fikri Aji, Zhengzhong Liu, Andy Hock, Andrew Feldman, Jonathan Lee, A. Jackson, Preslav Nakov, Timothy Baldwin, and Eric P. Xing. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. ArXiv, abs/2308.16149. ↗
  13. Wenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai, Jen-tse Huang, Zhaopeng Tu, and Michael Lyu. 2024. Not all countries celebrate thanksgiving: On the cultural dominance in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6349–6384, Bangkok, Thailand. Association for Computational Linguistics. ↗
  14. Lingling Xu, Haoran Xie, S. Joe Qin, Xiaohui Tao, and Fu Lee Wang. 2023. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. ArXiv, abs/2312.12148. ↗
  15. Abdellah El Mekki, Houdaifa Atou, Omer Nacar, Shady Shehata, and Muhammad Abdul-Mageed. 2025. Nilechat: Towards linguistically diverse and culturally aware llms for local communities. ArXiv, abs/2505.18383. ↗
  16. Rada Mihalcea, Oana Ignat, Longju Bai, Angana Borah, Luis Chiruzzo, Zhijing Jin, Claude Kwizera, Joan Nwatu, Soujanya Poria, and Thamar Solorio. 2024. Why AI is WEIRD and should not be this way: Towards AI for everyone, with everyone, by everyone. arXiv preprint, arXiv:2410.16315. Basel Mousi, Nadir Durrani, Fatema Ahmad, Md Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gomaa-2025-isl-nlp-palmx-2025-retrieval/