Elamur Research⌕
☰
home / papers / 2025 / CultranAI at PalmX 2025: Data Augmentation for Cultural Knowledge Representation

CultranAI at PalmX 2025: Data Augmentation for Cultural Knowledge Representation

Paper · 2025 · Hunzalah Hassan Bhatti, Youssef Ahmed, Md Arid Hasan et al.

Summary

The paper describes CultranAI, a submission to the PalmX 2025 shared task (General Culture Evaluation subtask). The task asks models under 13B parameters to answer four-option multiple-choice questions in MSA about Arab history, customs, geography, literature and food. The authors first compared several Arabic-centric and multilingual open LLMs zero-shot, and Fanar came out on top. They then enlarged the small official training set in two ways. First, they recast the community-written Palm QA data into MCQs, using GPT-4.1 to invent distractors. Second, they built PalmX-ext, about 22K new country-specific MCQs gathered with the NativQA framework and filtered by GPT-4.1. Fanar was adapted with LoRA on this combined data. Full-precision LoRA beat QLoRA only marginally, while augmentation gave a clear improvement on the dev set. The final system reached 84.1% on held-out data and placed 5th on the blind test with 70.5% accuracy. The work shows that cheap, LLM-assisted data expansion and lightweight adapters can noticeably improve cultural QA in a lower-resource language.

Key points

  • Data sizes: PalmX has 2,000 train, 500 dev and 2,000 test questions. Of the 1,926 public Palm test pairs, 950 were used for training and 950 for evaluation, split with stratification by country. PalmX-ext adds about 22,000 training MCQs.
  • Zero-shot accuracy on PalmX Dev:
    • Fanar: 72.40, the best.
    • ALLaM-7B-Instruct: 70.60.
    • Miraj Mini: 70.20.
    • Qwen2.5-7B-Instruct: 69.20.
    • tiny-random-LlamaForCausalLM: 23.40, which sits near chance.
  • The paper's naming is inconsistent: Fanar appears as Fanar-7B in the tables and as Fanar-1-9B-Instruct in the abstract.
  • PEFT and augmentation results on PalmX Dev:
    • LoRA scored 80.6 and QLoRA (4-bit) scored 80.0.
    • Training on PalmX alone gave 76.6, versus 80.6 with Palm and PalmX-ext added.
  • Final configuration: 3 epochs, learning rate 2e-4, LoRA rank 64, alpha 16, dropout 0.1, with PalmX train and dev included in training. It scored 84.1% on held-out data, but the paper labels this set inconsistently (Palm test in the results section, PalmX dev in the abstract). It scored 70.5% on the blind test (rank 5).
  • Quality checks and known issues:
    • A manual check of 50 PalmX-ext items gave roughly 7.4 out of 10 for accuracy and clarity.
    • The authors admit some Palm items exceeded the 512-token limit.
    • The machine-generated wrong options frequently came out briefer than the right answer.

Interesting ideas

  • I1. Include a model with random weights as a floor baseline, so every reported score can be read against chance level on that exact test set.
  • I2. When a small labelled resource must provide both training and evaluation data, split it with stratification by a key attribute (here, country) so both halves cover the same regions.
  • I3. Converting free-form QA to MCQ with machine-made distractors can leak the answer through surface cues; the authors noticed that the incorrect choices tended to be briefer than the right one.
  • I4. Seed data collection by first tagging each question with its country, then retrieving new location-specific material for that tag, which widens topical coverage in a targeted way.
  • I5. Layer quality filters: keep only answers from whitelisted source domains, then let an LLM drop irrelevant or wrong items, then hand-score a small sample.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Every evaluation run could include a "random reader" negative control, confirming that gains do not come from answer-format artifacts.
  • I2 (learn): Silver-data and treebank splits could be stratified by genre or source, so a parser is not judged on registers it never saw.
  • I3 (answer): When building school-test or fable QA items, audit the options for length and form bias, and check that the reader's answer survives when such cues are equalized.
  • I4 (world): Gaps in the knowledge graph could drive generation, first labelling a failing question with its domain or entity type, then gathering facts only for that slot.
  • I5 (learn): LLM-teacher outputs could pass through the same cascade (source trust, teacher critique, small human spot-check with a recorded score) before entering silver data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TorontoToronto · Canada96 works · 142,007 citations🎓Qatar UniversityDoha · Qatar48 works · 1,026 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksKnowledge graphs & ontologies · 192 works

Cite

Cite as Hunzalah Hassan Bhatti, Youssef Ahmed, Md Arid Hasan and Firoj Alam. 2025. CultranAI at PalmX 2025: Data Augmentation for Cultural Knowledge Representation. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.111

@inproceedings{bhatti2025cultranai,
  title = {{CultranAI at PalmX 2025: Data Augmentation for Cultural Knowledge Representation}},
  author = {Hunzalah Hassan Bhatti, Youssef Ahmed, Md Arid Hasan and Firoj Alam},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.111},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.111}
}

References

27 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Youssef Elshahawy, Ahmed Ali, Nadir Durrani, Natasa Milic-Frayling, and Firoj Alam. 2024. LAraBench: Benchmarking Arabic AI with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 487–520, St. Julian’s, Malta. Association for Computational Linguistics. ↗
  2. Rawan Al-Matham, Kareem Darwish, Raghad Al-Rasheed, Waad Alshammari, Muneera Alhoshan, Amal Almazrua, Asma Al Wazrah, Mais Alheraki, Firoj Alam, Preslav Nakov, Norah Alzahrani, Eman alBilali, Nizar Habash, Abdelrahman El-Sheikh, Muhammad Elmallah, Haonan Li, Hamdy Mubarak, Mohamed Anwar, Zaid Alyafeai, and 24 others. 2025. BALSAM: A platform for benchmarking arabic large language models. arXiv preprint arXiv:2507.22603. ↗
  3. Firoj Alam, Md Arid Hasan, Sahinur Rahman Laskar, Mucahid Kutlu, and Shammur Absar Chowdhury. 2025. NativQA Framework: Enabling llms with native, local, and everyday knowledge. arXiv preprint arXiv:2504.05995. ↗
  4. Fakhraddin Alwajih, Abdellah El Mekki, Samar Mohamed Magdy, AbdelRahim A. Elmadany, Omer Nacar, El Moatez Billah Nagoudi, Reem Abdel-Salam, Hanin Atwany, Youssef Nafea, Abdulfattah Mohammed Yahya, Rahaf Alhamouri, Hamzah A. Alsayadi, Hiba Zayed, Sara Shatnawi, Serry Sibaee, Yasir Ech-chammakhy, Walid Al-Dhabyani, Marwa Mohamed Ali, Imen Jarraya, and 25 others. 2025a. Palm: A culturally inclusive and linguistically diverse dataset for Arabic LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32871–32894, Vienna, Austria. Association for Computational Linguistics.
  5. Fakhraddin Alwajih, Abdellah El Mekki, Hamdy Mubarak, Majd Hawasly, Abubakr Mohamed, and Muhammad Abdul-Mageed. 2025b. PalmX 2025: The first shared task on benchmarking llms on arabic and islamic culture. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou, China. Association for Computational Linguistics. Co-located with EMNLP 2025, November 5–9.
  6. Lama Ayash, Hassan Alhuzali, Ashwag Alasmari, and Sultan Aloufi. 2025. Saudiculture: A benchmark for evaluating large language models’ cultural competence within saudi arabia. Journal of King Saud University Computer and Information Sciences, 37(6):123. ↗
  7. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham Abdullah Alyahya, Sultan AlRashed, Faisal Abdulrahman Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Saad Amin Hassan, Dr. Majed Alrubaian, Ali Alammari, Zaki Alawami, and 7 others. 2025. AL-Lam: Large language models for arabic and english. In The Thirteenth International Conference on Learning Representations. find
  8. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with GPT-4. Technical report, Microsoft Research. ↗
  9. Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. 2025. CulturalBench: A robust, diverse and challenging benchmark for measuring LMs’ cultural knowledge through human-AI red-teaming. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25663– 25701, Vienna, Austria. Association for Computational Linguistics.
  10. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470. ↗
  11. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567, Florence, Italy. Association for Computational Linguistics. ↗
  12. Md. Arid Hasan, Maram Hasanain, Fatema Ahmad, Sahinur Rahman Laskar, Sunaya Upadhyay, Vrunda N Sukhadia, Mucahid Kutlu, Shammur Absar Chowdhury, and Firoj Alam. 2025. NativQA: Multilingual culturally-aligned natural query for LLMs. In Findings of the Association for Computational Linguistics: ACL 2025, pages 14886–14909, Vienna, Austria. Association for Computational Linguistics. ↗
  13. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  14. Daniel Khashabi, Amos Ng, Tushar Khot, Ashish Sabharwal, Hannaneh Hajishirzi, and Chris Callison-Burch. 2021. GooAQ: Open question answering with diverse answer types. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 421–433, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  15. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466. ↗
  16. Cheng Li, Mengzhuo Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. 2024a. CultureLLM:: Incorporating cultural differences into large language models. Advances in Neural Information Processing Systems, 37:84799–84838.
  17. Cheng Li, Damien Teney, Linyi Yang, Qingsong Wen, Xing Xie, and Jindong Wang. 2024b. CulturePark: Boosting cross-cultural understanding in large language models. In Advances in Neural Information Processing Systems, volume 37, pages 65183–65216. find
  18. Abdellah El Mekki, Houdaifa Atou, Omer Nacar, Shady Shehata, and Muhammad Abdul-Mageed. 2025. Nilechat: Towards linguistically diverse and culturally aware llms for local communities. arXiv preprint arXiv:2505.18383. ↗
  19. Basel Mousi, Nadir Durrani, Fatema Ahmad, Md Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2025. AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4186–4218, Abu Dhabi, UAE. Association for Computational Linguistics. Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, and 1 others. 2024. BLEnD: A benchmark for llms on everyday knowledge in diverse cultures and languages. In Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS), Vancouver, Canada. ↗
  20. Siddhesh Pawar, Junyeong Park, Jiho Jin, Arnav Arora, Junho Myung, Srishti Yadav, Faiz Ghifari Haznitrama, Inhwa Song, Alice Oh, and Isabelle Augenstein. 2025. Survey of cultural awareness in language models: Text and beyond. Computational Linguistics, pages 1–96. ↗
  21. Rifki Putri, Faiz Haznitrama, Dea Adhista, and Alice Oh. 2024. Can llm generate culturally relevant commonsense qa data? case study in indonesian and sundanese. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20571–20590. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, and 1 others. 2023. Jais and Jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. ↗
  22. Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Sunny Yu, Raya Horesh, Rogério Abreu De Paula, and Diyi Yang. 2024. CultureBank: An online community-driven knowledge base towards culturally aware language technologies. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4996–5025, Miami, Florida, USA. Association for Computational Linguistics.
  23. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, and 23 others. 2025. Fanar: An arabic-centric multimodal generative ai platform.
  24. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. ↗
  25. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191. ↗
  26. Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966. ↗
  27. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics. - **Unsure:** It is difficult to determine whether the question refers to any specific culture. """ user_prompt = f"""

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bhatti-2025-cultranai-palmx-2025-data-augmentation/