Elamur Research⌕
☰
home / papers / 2025 / Cultura-Arabica: Probing and Enhancing Arabic Cultural Awareness in Large Language Models via LoRA

Cultura-Arabica: Probing and Enhancing Arabic Cultural Awareness in Large Language Models via LoRA

Paper · 2025 · Pulkit Chatwal and Santosh Kumar Mishra

Summary

The paper asks whether a small Arabic LLM can be adapted cheaply to answer questions that depend on cultural knowledge rather than on language competence alone. This kind of knowledge covers local customs and the norms governing social behaviour, along with history and geography, artistic traditions and phrases particular to regional dialects throughout the Arab countries. The authors entered the PalmX 2025 shared task, which consists of multiple-choice questions in Modern Standard Arabic. They fine-tuned NileChat-3B with LoRA, updating under 1% of the weights while the base stays frozen. Each item was recast as a fixed prompt with four lettered options, and the model was told to reply with a single letter. Training used the 2,000-question training split for three epochs on one A100 GPU. Afterwards the adapters were merged into the base checkpoint for deployment. On the 2,000-question blind test the system reached 71.65% accuracy and placed second, half a point below the winner. The result suggests that modest, targeted adaptation of a 3B model can rival heavier efforts on culturally grounded QA. This matters for languages and communities where compute and labelled data are limited.

Key points

  • Data: PalmX 2025 provides 2,000 training, 500 development and 2,000 blind-test MCQs, balanced across cultural domains and Arab countries.
  • Base model: NileChat-3B is a decoder-only model with 24 layers and a 50,000-token BPE vocabulary. Its EOS token is reused as the padding token.
  • LoRA setup:
    • rank 16, α = 32;
    • applied to q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj and down_proj.
  • Training:
    • Transformers Trainer, bfloat16 precision;
    • batch size 2 with 4-step gradient accumulation, giving an effective batch of 8;
    • constant learning rate of 2e-4, three epochs, causal LM objective (mlm=False).
  • Results: accuracy was 71.65%, ranking 2nd. The winner was HAI research group with 72.15%, and AYA_Team was 3rd with 71.45%, so the top three sit within one point. The merged model is released on Hugging Face as PalmQA-3B-Arabic (named NileChat-3B-Arabic-QA-Merged-v2 in the paper).

Interesting ideas

  • I1. Forcing the model to emit only an option letter narrows the output space and makes scoring trivial and stable, because training and evaluation then share one format.
  • I2. The task is framed as getting the right answer given both linguistic and cultural knowledge, which treats background knowledge as a separate factor from language competence.
  • I3. The adapters cover every projection, both attention and feed-forward, while the base stays frozen. This adapts behaviour broadly at low cost and avoids erasing what was learned in pretraining.
  • I4. After training, the adapters are folded back into the base weights, so deployment needs no extra loading step and keeps the original inference speed.
  • I5. The top three teams differ by under one point, which the authors read as evidence that minor fine-tuning choices can shift rankings.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-test readers could produce an explicit option label linked to the meaning-graph evidence that chose it. This would make multiple-choice scoring exact while keeping the explanation separate.
  • I2 (world): Questions could be tagged by whether they need only the parsed text or also compiled world knowledge (WordNet/Wikidata/ontology). Failures could then be traced to the parser or to missing knowledge.
  • I3 (tree): gparse could get a small adapter per UD dialect or treebank domain. The shared parser would stay intact, preventing regression on the main treebank.
  • I4 (learn): Domain-specific fine-tunes could be trained as adapters and merged into a single shipped checkpoint per release. Versioning and runtime stay simple.
  • I5 (learn): Gaps this small should not be trusted without repeated runs and negative controls. Evaluation reports should state the spread across seeds before declaring that one configuration beats another.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Rajiv Gandhi Institute of Petroleum TechnologyRaebareli · India3 works · 3 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksInterpretability · 130 works

Cite

Cite as Pulkit Chatwal and Santosh Kumar Mishra. 2025. Cultura-Arabica: Probing and Enhancing Arabic Cultural Awareness in Large Language Models via LoRA. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.115

@inproceedings{chatwal2025cultura,
  title = {{Cultura-Arabica: Probing and Enhancing Arabic Cultural Awareness in Large Language Models via LoRA}},
  author = {Pulkit Chatwal and Santosh Kumar Mishra},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.115},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.115}
}

References

21 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Hamdy Mubarak, Shammur Absar Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, et al. 2023. Larabench: Benchmarking arabic ai with large language models. arXiv preprint arXiv:2305.14982. ↗
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020. Arbert & marbert: Deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. ↗
  3. Ebtesam Almazrouei, Ruxandra Cojocaru, Michele Baldo, Quentin Malartic, Hamza Alobeidli, Daniele Mazzotta, Guilherme Penedo, Giulia Campesan, Mugariya Farooq, Maitha Alhammadi, et al. 2023. Alghafa evaluation benchmark for arabic language models. In Proceedings of ArabicNLP 2023, pages 244– 275.
  4. Fakhraddin Alwajih, Abdellah El Mekki, Hamdy Mubarak, Majd Hawasly, Abubakr Mohamed, and Muhammad Abdul-Mageed. 2025. PalmX 2025: The First Shared Task on Benchmarking LLMs on Arabic and Islamic Culture. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou, China. Association for Computational Linguistics. Co-located with EMNLP 2025, November 5–9.
  5. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  6. Gary Marcus and Ernest Davis. 2019. Rebooting AI: Building artificial intelligence we can trust. Vintage. find
  7. Abdellah El Mekki, Houdaifa Atou, Omer Nacar, Shady Shehata, and Muhammad Abdul-Mageed. 2025. Nilechat: Towards linguistically diverse and culturally aware llms for local communities. ↗
  8. Ahmad Mustapha, Hadi Al-Khansa, Hadi Al-Mubasher, Aya Mourad, Ranam Hamoud, Hasan El-Husseini, Marwah Al-Sakkaf, and Mariette Awad. 2024. Arastem: A native arabic multiple choice question benchmark for evaluating llms knowledge in stem subjects. arXiv preprint arXiv:2501.00559. ↗
  9. Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. 2023. Having beer after prayer? measuring cultural bias in large language models. arXiv preprint arXiv:2305.14456. ↗
  10. Zaid Alyafeai, Khalid Almubarak, Ahmed Ashraf, Deema Alnuhait, Saied Alshahrani, Gubran AQ Abdulrahman, Gamil Ahmed, Qais Gawah, Zead Saleh, Mustafa Ghaleb, et al. 2024. Cidar: Culturally relevant instruction dataset for arabic. arXiv preprint arXiv:2402.03177. ↗
  11. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  12. Prakash Aryan. 2024. Resource-aware arabic llm creation: Model adaptation, integration, and multi-domain testing. In International Conference on Advanced Network Technologies and Intelligent Computing, pages 415–434. Springer.
  13. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multi-task, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023. ↗
  14. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113.
  15. Siddhesh Pawar, Junyeong Park, Jiho Jin, Arnav Arora, Junho Myung, Srishti Yadav, Faiz Ghifari Haznitrama, Inhwa Song, Alice Oh, and Isabelle Augenstein. 2025. Survey of cultural awareness in language models: Text and beyond. Computational Linguistics, pages 1–96. ↗
  16. Laura Cabello Piqueras and Anders Søgaard. 2022. Are pretrained multilingual models equally fair across languages? arXiv preprint arXiv:2210.05457. ↗
  17. Srijith Radhakrishnan, Chao-Han Huck Yang, Sumeer Ahmad Khan, Narsis A Kiani, David Gomez-Cabrero, and Jesper N Tegner. 2023. A parameter-efficient learning approach to arabic dialect identification with pre-trained general-purpose speech model. arXiv preprint arXiv:2305.11244. ↗
  18. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, et al. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. ↗
  19. Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcec. 2024. Cultural bias and cultural alignment of large language models. PNAS nexus, 3(9):pgae346. ↗
  20. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. ↗
  21. Chengzhi Zhong, Fei Cheng, Qianying Liu, Junfeng Jiang, Zhen Wan, Chenhui Chu, Yugo Murawaki, and Sadao Kurohashi. 2024. Beyond english-centric llms: What language do multilingual language models think in? arXiv preprint arXiv:2408.10811. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chatwal-2025-cultura-arabica-probing-enhancing-arabic/