📄 AYA at PalmX 2025: Modeling Cultural and Islamic Knowledge in LLMs

Summary

The PalmX 2025 shared task checks how well large language models know Arab general culture (Subtask 1) and Islamic topics (Subtask 2). Both are tested with Modern Standard Arabic multiple-choice questions. The authors first ran zero-shot tests on several instruction-tuned models, both Arabic-centric and multilingual: Qwen2.5-7B, Jais-13B-Chat, Llama-3.1-8B, NileChat-3B, ALLaM-7B, Gemma-7b-it, Fanar and others. They then fine-tuned the most promising ones with LoRA in three settings: task data alone, culture and Islamic data pooled together, and data enlarged with GPT-4.1 paraphrases.

Different models won different subtasks:

• Subtask 1: Fanar-1-9B-Instruct trained on the pooled data did best on the culture dev set.
• Subtask 2: ALLaM-7B-Instruct trained with paraphrase augmentation did best on the Islamic dev set.

On the hidden test sets, the team placed third in Subtask 1 with 71.45% and first in Subtask 2 with 84.22%. The work shows that light adapter tuning plus cheap rewording of questions can noticeably raise accuracy on culture-specific knowledge, even when training data is small.

Key points

• Data sizes: the culture subtask has 2,000 training, 500 dev and 2,000 test questions. The Islamic subtask has 600 training, 300 dev and 1,000 test questions. Augmentation doubled each training set, to 4,000 and 1,200.
• LoRA settings: rank 16, alpha 32, learning rate 2e-4, dropout 0.05, maximum sequence length 512, three epochs. Adapters were applied only to the query and value projections.
• Culture dev set: Fanar-1-9B rose from 72.4 zero-shot to 80.2 after fine-tuning, and to 80.8 with pooled culture and Islamic data.
• Islamic dev set: ALLaM-7B rose from 72.7 to 76.3 after fine-tuning, and to 77.3 with paraphrase augmentation.
• Uneven effects: Qwen2.5-7B barely changed across settings. Gemma-7b-it dropped sharply with pooled data, to 39.6 on culture and 25.3 on the Islamic set.
• Error analysis (Islamic test): class B received the most correct answers (483). The most frequent errors were A→B (32) and D→B (10), and class D was the weakest.

Interesting ideas

• I1. The paraphrases were deliberately made harder or slightly confusing while keeping the options and gold answer fixed, so each training example gains a meaning-preserving but tougher twin.
• I2. A broad zero-shot sweep over many candidate models came first, and only the strongest per domain were fine-tuned, saving compute and guiding model choice.
• I3. Pooling data from a neighbouring domain helped one model, did nothing for another, and badly hurt a third, so data mixing must be checked per model rather than assumed beneficial.
• I4. Breaking errors down by answer letter showed the model drifting toward option B, which reveals a position preference rather than a knowledge gap.
• I5. The final system used a different model for each subtask instead of forcing one model to cover both domains.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): an LLM teacher tier could reword school-test questions with the answer locked, and Mova should give the same answer to both versions; this works as an invariance check alongside negative controls.
• I2 (learn): before choosing which LLM serves as teacher for silver data in a given domain, run a cheap baseline sweep on a small dev slice and pick per domain.
• I3 (tree): when gparse is trained on mixed treebanks or converted UD dialects, compare each mix against single-source training on the target dev set before adopting it.
• I4 (answer): log the chosen-option distribution on multiple-choice school tests and compare it with the gold distribution to catch letter bias in the answering readers.
• I5 (answer): route questions to domain-specific readers, such as fables or maths word problems, and evaluate each route separately so a weak domain does not hide behind a strong one.