Elamur Research⌕
☰
home / papers / 2024 / CamelEval: Advancing Culturally Aligned Arabic Language Models and Benchmarks

CamelEval: Advancing Culturally Aligned Arabic Language Models and Benchmarks

Summary

Arabic makes up only a small share of the web text used to pre-train LLMs. As a result, many existing models answer Arabic speakers awkwardly, get regional facts wrong, or ignore local norms. The authors present Juhaina, an open Arabic-English model with 9.24B parameters built on top of Gemma 2. They post-trained it in two phases: supervised fine-tuning on annotator-written answers, then preference alignment from human rankings of paired outputs. The training data came from internal collections, scraped web pages and translated datasets. It was organised by subject taxonomy, heavily tagged, cleaned, decontaminated against benchmarks and balanced across categories. The authors also argue that the Open Arabic LLM Leaderboard (OALL) measures the wrong things, so they propose CamelEval, an LLM-as-a-judge benchmark derived from AlpacaEval. On CamelEval, Juhaina beats comparably sized and some much larger open models and comes close to GPT-4o on culturally grounded prompts. The work matters because it shows a recipe for low-resource, culture-aware post-training together with an evaluation that rewards useful free-form answers.

Key points

  • Juhaina has 9.24B parameters and an 8,192-token context window. It was post-trained from Gemma 2, which was pre-trained on up to 13 trillion mostly English tokens, and is released on Huggingface under the MIT license.
  • The final post-training data held roughly 600,000 SFT rows and 40,000 alignment rows, with an equal number of samples drawn per subject category. Arabic is only about 0.6% of Common Crawl documents.
  • The authors name three weaknesses of OALL:
    • it scores the likelihood (NLL) of a single "gold" answer, which penalises valid answers in other formats;
    • it runs only zero-shot;
    • its public test set invites contamination and leaderboard gaming.
  • CamelEval has two sets of 805 prompts each. The Translated Set contains AlpacaEval prompts, with Freedom Intelligence's machine translations fixed by native speakers. The Curated Set was generated by GPT-4 from textbook-quality material on culture, religion, history and other fields, then filtered for difficulty and checked by humans.
  • Win rates against Gemma2-9B-IT (curated / translated):
    • Juhaina-9B: 98.47 / 91.37
    • GPT-4o: 99.73 / 97.11
    • Qwen2-72B-Instruct: 95.80 / 91.95
    • Silma-9B-v1: 30.56 / 45.26, which is worse than the base model it was tuned from.

Interesting ideas

  • I1. The authors turned the problems they actually ran into into a reusable data-quality checklist: near-duplicates, scraping leftovers, translation artifacts, wrong-language rows, broken code or structured text, and a skewed task mix. This makes quality auditable rather than judged by row counts. — For Mova (learn): Silver treebanks agreed by witness parsers could be run through a similar checklist, flagging translation-like or malformed sentences before training so that bad data does not quietly degrade the parser.
  • I2. Every sample is tagged with subject, sub-category, task type and difficulty, and those tags then drive both inspection and resampling to equal counts per category. Tagging therefore serves as both a diagnostic tool and a sampling control. — For Mova (learn): Tagging silver and teacher-generated examples by construction type and difficulty would let training batches be balanced, and per-tag error rates would show which syntactic phenomena are under-served.
  • I3. Instead of measuring how likely a single reference answer is, the benchmark compares two systems' answers pairwise and reports a win rate against a fixed baseline. This avoids penalising correct answers phrased differently. — For Mova (answer): Free-form answers to fable and tale questions could be scored as wins over a fixed baseline reader, instead of string-matching one gold answer that the explainable pipeline may legitimately word differently.
  • I4. The curated test questions were deliberately built to require knowledge beyond the source passage, and only the hardest ones were kept after human review. This probes understanding rather than copying from the text. — For Mova (answer): QA sets on tales could include questions that need the compiled knowledge layer (ontology, verb classes), not just the text's own world, revealing whether background knowledge is actually being used.
  • I5. One fine-tuned model scored below its own starting checkpoint, which shows that every adaptation should be measured directly against the model it came from. — For Mova (learn): Each retraining, or each new self-induced repair rule, should be evaluated head-to-head against the previous version as a built-in negative control, so regressions are caught immediately.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

16 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 13 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2409.12623 · arXiv 2409.12623

@misc{qian2024cameleval,
  title = {{CamelEval: Advancing Culturally Aligned Arabic Language Models and Benchmarks}},
  author = {Qian, Zhaozhi and Altam, Faroq and Alqurishi, Muhammad and Souissi, Riad},
  year = {2024},
  doi = {10.48550/arxiv.2409.12623},
  eprint = {2409.12623},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.12623}
}

References

25 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A LMAZROUEI , E., C OJOCARU , R., B ALDO , M., M ALARTIC , Q., A LOBEIDLI , H., M AZ -ZOTTA , D., P ENEDO , G., C AMPESAN , G., F AROOQ , M., A LHAMMADI , M., L AUNAY , J., AND N OUNE , B. AlGhafa evaluation benchmark for Arabic language models. In Proceedings of ArabicNLP 2023 (Singapore (Hybrid), Dec. 2023), H. Sawaf, S. El-Beltagy, W. Zaghouani, W. Magdy, A. Abdelali, N. Tomeh, I. Abu Farha, N. Habash, S. Khalifa, A. Keleg, H. Haddad, I. Zitouni, K. Mrini, and R. Almatham, Eds., Association for Computational Linguistics, pp. 244–275. find
  2. A LYAFEAI , Z., A LMUBARAK , K., A SHRAF , A., A LNUHAIT , D., A LSHAHRANI , S., A B -DULRAHMAN , G. A., A HMED , G., G AWAH , Q., S ALEH , Z., G HALEB , M., ET AL . Cidar: Culturally relevant instruction dataset for arabic. arXiv preprint arXiv:2402.03177 (2024). ↗
  3. A RYABUMI , V., D ANG , J., T ALUPURU , D., D ASH , S., C AIRUZ , D., L IN , H., V ENKITESH , B., S MITH , M., M ARCHISIO , K., R UDER , S., ET AL . Aya 23: Open weight releases to further multilingual progress. arXiv preprint arXiv:2405.15032 (2024). ↗ our card
  4. B ROWN , T. B. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020). ↗
  5. C RAWL , C. Statistics of common crawl monthly archives. https://commoncrawl.github.io/cc-crawl-statistics/plots/languages.html. Accessed: 2024-09-11. ↗
  6. D UBEY , A., J AUHRI , A., P ANDEY , A., K ADIAN , A., A L -D AHLE , A., L ETMAN , A., M ATHUR , A., S CHELTEN , A., Y ANG , A., F AN , A., ET AL . The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024). ↗
  7. E LFILALI , A., A LOBEIDLI , H., F OURRIER , C., B OUSSAHA , B. E. A., C O -JOCARU , R., H ABIB , N., AND H ACID , H. Open arabic llm leaderboard. https://huggingface.co/spaces/OALL/Open-Arabic-LLM-Leaderboard, 2024. ↗
  8. E THNOLOGUE . Ethnologue arabic statistics. https://www.ethnologue.com/language/ara/. Accessed: 2024-09-11. ↗
  9. F REEDOM I NTELLIGENCE . Acva arabic cultural value alignment. https://huggingface.co/datasets/FreedomIntelligence/ACVA-Arabic-Cultural-Value-Alignment, 2024. ↗
  10. H ONG , J., L EE , N., AND T HORNE , J. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691 2, 4 (2024), 5. ↗
  11. H U , E. J., S HEN , Y., W ALLIS , P., A LLEN -Z HU , Z., L I , Y., W ANG , S., W ANG , L., AND C HEN , W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021). ↗
  12. H UANG , H., Y U , F., Z HU , J., S UN , X., C HENG , H., S ONG , D., C HEN , Z., A LHARTHI , A., A N , B., L IU , Z., Z HANG , Z., C HEN , J., L I , J., W ANG , B., Z HANG , L., S UN , R., W AN , X., L I , H., AND X U , J. Acegpt, localizing large language models in arabic, 2023. find our card
  13. K OTO , F., L I , H., S HATNAWI , S., D OUGHMAN , J., S ADALLAH , A. B., A LRAEESI , A., A LMUBARAK , K., A LYAFEAI , Z., S ENGUPTA , N., S HEHATA , S., ET AL . Arabicmmlu: Assessing massive multitask language understanding in arabic. arXiv preprint arXiv:2402.12840 (2024). ↗ our card
  14. L I , X., Z HANG , T., D UBOIS , Y., T AORI , R., G ULRAJANI , I., G UESTRIN , C., L IANG , P., AND H ASHIMOTO , T. B. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 5 2023. ↗
  15. M UENNIGHOFF , N., W ANG , T., S UTAWIKA , L., R OBERTS , A., B IDERMAN , S., S CAO , T. L., B ARI , M. S., S HEN , S., Y ONG , Z.-X., S CHOELKOPF , H., ET AL . Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786 (2022). ↗ our card
  16. N AOUS , T., R YAN , M. J., R ITTER , A., AND X U , W. Having beer after prayer? measuring cultural bias in large language models. arXiv preprint arXiv:2305.14456 (2023). ↗ our card
  17. P ENEDO , G., M ALARTIC , Q., H ESSLOW , D., C OJOCARU , R., C APPELLI , A., A LOBEI -DLI , H., P ANNIER , B., A LMAZROUEI , E., AND L AUNAY , J. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116 (2023). ↗
  18. R AFAILOV , R., S HARMA , A., M ITCHELL , E., M ANNING , C. D., E RMON , S., AND F INN , C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems 36 (2024). find
  19. S ENGUPTA , N., S AHU , S. K., J IA , B., K ATIPOMU , S., L I , H., K OTO , F., M ARSHALL , W., G OSAL , G., L IU , C., C HEN , Z., ET AL . Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149 (2023). ↗ our card
  20. S INGH , S., V ARGUS , F., D SOUZA , D., K ARLSSON , B. F., M AHENDIRAN , A., K O , W.-Y., S HANDILYA , H., P ATEL , J., M ATACIUNAS , D., OM AHONY , L., ET AL . Aya dataset: An open-access collection for multilingual instruction tuning. arXiv preprint arXiv:2402.06619 (2024). ↗ our card
  21. S TANOVSKY , G., S MITH , N. A., AND Z ETTLEMOYER , L. Evaluating gender bias in machine translation. arXiv preprint arXiv:1906.00591 (2019). ↗ our card
  22. T EAM , G., R IVIERE , M., P ATHAK , S., S ESSA , P. G., H ARDIN , C., B HUPATIRAJU , S., H USSENOT , L., M ESNARD , T., S HAHRIARI , B., R AMÉ , A., ET AL . Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118 (2024). ↗
  23. V ANMASSENHOVE , E., S HTERIONOV , D., AND G WILLIAM , M. Machine translationese: Effects of algorithmic bias on linguistic complexity in machine translation. arXiv preprint arXiv:2102.00287 (2021). ↗
  24. W ANG , J., R UBINSTEIN , B., AND C OHN , T. Measuring and mitigating name biases in neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2022), pp. 2576–2590. find
  25. W U , C., G AN , Y., G E , Y., L U , Z., W ANG , J., F ENG , Y., L UO , P., AND S HAN , Y. Llama pro: Progressive llama with block expansion. arXiv preprint arXiv:2401.02415 (2024). ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qian-2024-cameleval-advancing-culturally-aligned-arabic/