Elamur Research
☰
home / papers / 2025 / Fanar: An Arabic-Centric Multimodal Generative AI Platform

Fanar: An Arabic-Centric Multimodal Generative AI Platform

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2501.13944

Summary

Arabic is spoken by over 400 million people, yet it makes up only about 0.5% of web text. That scarcity, along with high compute costs and limited GPU access, makes strong Arabic LLMs hard to build. The QCRI team at Hamad Bin Khalifa University, funded by Qatar's Ministry of Communications and Information Technology as a sovereign-AI effort, responds with Fanar, an Arabic-first generative AI platform.

Its core is two LLMs trained on a curated, deduplicated corpus of nearly 1 trillion Arabic, English and code tokens:

  • Fanar Star (7B) was trained from scratch.
  • Fanar Prime (9B) was built by continued pre-training of Gemma-2 9B.

An orchestrator decides which model answers each prompt. Around the models, the platform adds:

  • a morphology-aware Arabic tokenizer;
  • retrieval modules for Islamic content, recent events and biographies;
  • a fact-attribution service;
  • dialect-aware bilingual speech recognition;
  • image generation adapted to the region.

The authors report that both models lead comparable-size models on established benchmarks. They also release new culturally focused evaluation sets. The work offers a complete, documented recipe for building an LLM stack in a low-resource language.

Key points

  • According to Common Crawl statistics cited in the paper, English accounts for about 46% of web text and Arabic for only about 0.5%.
  • The pre-training pipeline uses syntactic, semantic and model-based filtering, deduplication, and machine translation (English→MSA and MSA→dialect). A custom morphology-based tokenizer replaces plain BPE.
  • The in-house benchmark Almieyar covers language, overall knowledge and reasoning, culture, and multimodality.
  • The Arab Cultural MCQ benchmark has 1K questions. They were selected from 7.5K GPT-4o-generated candidates through expert review and redundancy removal with bge-multilingual-gemma2 clustering.
  • On the aiXamine security and safety suite, Fanar Star scored the highest average among the compared models: 82.49, versus 74.85 for LLaMA-3.2-3B-Instruct and 61.07 for gemma-2-9b-it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

122 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 117 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.13944 · arXiv 2501.13944

@misc{abbas2025fanar,
  title = {{Fanar: An Arabic-Centric Multimodal Generative AI Platform}},
  author = {Abbas, Fanar Team: Ummar and Ahmad, Mohammad Shahmeer and Alam, Firoj and Altinisik, Enes and Asgari, Ehsannedin and Boshmaf, Yazan and Boughorbel, Sabri and Chawla, Sanjay and Chowdhury, Shammur and Dalvi, Fahim and Darwish, Kareem and Durrani, Nadir and Elfeky, Mohamed and Elmagarmid, Ahmed and Eltabakh, Mohamed and Fatehkia, Masoomali and Fragkopoulos, Anastasios and Hasanain, Maram and Hawasly, Majd and Husaini, Mus'ab and Jung, Soon-Gyo and Lucas, Ji Kim and Magdy, Walid and Messaoud, Safa and Mohamed, Abubakr and Mohiuddin, Tasnim and Mousi, Basel and Mubarak, Hamdy and Musleh, Ahmad and Naeem, Zan and Ouzzani, Mourad and Popovic, Dorde and Sadeghi, Amin and Sencar, Husrev Taha and Shinoy, Mohammed and Sinan, Omar and Zhang, Yifan and Ali, Ahmed and Kheir, Yassine El and Ma, Xiaosong and Ruan, Chaoyi},
  year = {2025},
  doi = {10.48550/arxiv.2501.13944},
  eprint = {2501.13944},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.13944}
}

References

153 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelali, A., Darwish, K., Durrani, N. et al (2016). Farasa: A fast and furious segmenter for Arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16, San Diego, California. Association for Computational Linguistics. find our card
  2. Abdelali, A., Mubarak, H., Chowdhury, S.A. et al (2024). LAraBench: Benchmarking Arabic AI with Large Language Models. In Graham, Y. and Purver, M., editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 487–520, St. Julian’s, Malta. Association for Computational Linguistics. find our card
  3. Adel, H., Asgari, E. and Schütze, H. (2018). Overview of character-based models for natural language processing. In Gelbukh, A., editor, Computational Linguistics and Intelligent Text Processing: 18th International Conference, CICLing 2017, Budapest, Hungary, April 17–23, 2017, Revised Selected Papers, Part I 18, pages 3–16, Cham. Springer International Publishing. find
  4. Adler, B., Agarwal, N., Aithal, A. et al (2024). Nemotron-4 340b technical report. arXiv preprint arXiv:2406.11704. ↗
  5. Al-Wer, E. and de Jong, R. (2017). Dialects of Arabic. The handbook of dialectology, pages 523–534. find
  6. Alharbi, S., Alowisheq, A., Tüske, Z. et al (2024). SADA: Saudi audio dataset for Arabic. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10286–10290. IEEE. find
  7. Ali, A., Chowdhury, S., Hussein, A. et al (2021). Arabic Code-Switching Speech Recognition using Monolingual Data. In Proc. of the 22nd Annual Conference of the International Speech Communication Association (INTERSPEECH), pages 3475–3479, Brno, Czechia. find
  8. Ali, A., Shon, S., Samih, Y. et al (2019). The MGB-5 challenge: Recognition and dialect identification of dialectal Arabic speech. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1026–1033. IEEE. find
  9. Ali, A., Vogel, S. and Renals, S. (2017). Speech recognition challenge in the wild: Arabic MGB-3. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 316–322. IEEE. find
  10. AllenAI (2024). OLMo Family Model Card. https://huggingface.co/collections/allenai/ olmo-suite-65aeaae8fe5b6b2122b46778. Accessed on Dec 1, 2024. ↗
  11. Alnosairee, A. and Sartini, N.W. (2021). A sociolinguistics study in arabic dialects. PRASASTI: Journal of Linguistics, 6:1–17. find
  12. Alolaywi, Y. (2022). Derivation between english and arabic with reference to translation: a contrastive analysis study. Arab World English Journal for Translation and Literary Studies, 6:128–141. find
  13. Asgari, E., McHardy, A.C. and Mofrad, M.R. (2019). Probabilistic variable-length segmentation of protein sequences for discriminative motif discovery (DiMotif) and sequence embedding (ProtVecX). Scientific reports, 9(1):3577. find
  14. Asgari, E., Sabet, M.J., Dufter, P. et al (2020). Subword sampling for low resource word alignment. arXiv preprint arXiv:2012.11657. ↗
  15. Azar, M.G., Guo, Z.D., Piot, B. et al (2024). A general theoretical paradigm to understand learning from human preferences. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR. find our card
  16. Azerbayev, Z., Schoelkopf, H., Paster, K. et al (2024). Llemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations. find our card
  17. Bandarkar, L., Liang, D., Muller, B. et al (2024). The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 749–775, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics. find our card
  18. Barham, S., Weller, O., Yuan, M. et al (2023). Megawika: Millions of reports and their sources across 50 diverse languages. arXiv preprint arXiv:2307.07049. ↗
  19. Bari, M.S., Alnumay, Y., Alzahrani, N.A. et al (2024). ALLaM: Large Language Models for Arabic and English. arXiv preprint arXiv:2407.15390. ↗
  20. Baumgartner, J., Zannettou, S., Keegan, B. et al (2020). The pushshift Reddit dataset. arXiv preprint arXiv:2001.08435. ↗
  21. Bisk, Y., Zellers, R., Gao, J. et al (2020). PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432– 7439. find our card
  22. Blakeney, C., Paul, M., Larsen, B.W. et al (2024). Does your data spark joy? performance gains from domain upsampling at the end of training. In First Conference on Language Modeling. find
  23. Bohnet, B., Tran, V.Q., Verga, P. et al (2022). Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models. arXiv preprint arXiv:2212.08037. ↗ our card
  24. Boughorbel, S. and Hawasly, M. (2023). Analyzing Multilingual Competency of LLMs in Multi-Turn Instruction Following: A Case Study of Arabic. In Sawaf, H., El-Beltagy, S., Zaghouani, W. et al, editors, Proceedings of ArabicNLP 2023, pages 128–139, Singapore (Hybrid). Association for Computational Linguistics. find
  25. Broder, A. (1997). On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171), pages 21–29. IEEE. find
  26. Brown, T.B., Mann, B., Ryder, N. et al (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165. ↗
  27. Chen, J., Xiao, S., Zhang, P. et al (2024a). BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv preprint arXiv:2402.03216. ↗ our card
  28. Chen, Y., Niu, Z., Ma, Z. et al (2024b). F5-TTS: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint arXiv:2410.06885. ↗
  29. Chowdhury, S.A., Hussein, A., Abdelali, A. et al (2021). Towards One Model to Rule All: Multilingual Strategy for Dialectal Code-Switching Arabic ASR. In Proc. of the 22nd Annual Conference of the International Speech Communication Association (INTERSPEECH), pages 2466–2470, Brno, Czechia. find
  30. Chowdhury, S.A., Samih, Y., Eldesouki, M. et al (2020). Effects of dialectal code-switching on speech modules: A study using egyptian Arabic broadcast speech. In Proc. Interspeech 2020, pages 2382–2386, Shanghai, China. find
  31. Clark, J.H., Garrette, D., Turc, I. et al (2022). Canine: Pre-training an efficient tokenization-free encoder for language representation. Transactions of the Association for Computational Linguistics, 10:73–91. find
  32. Clark, P., Cowhey, I., Etzioni, O. et al (2018). Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. ↗
  33. Cloud, A. (2024). Qwen2.5 Family Model Card. https://huggingface.co/collections/Qwen/ qwen25-66e81a666513e518adb90d9e. Accessed on Dec 1, 2024. ↗
  34. Cohere (2024). Aya Expanse Family Model Card. https://huggingface.co/collections/ CohereForAI/c4ai-aya-expanse-671a83d6b2c07c692beab3c3. Accessed on Dec 1, 2024. ↗
  35. Cohere For AI (2024). c4ai-command-r-plus-08-2024. c4ai-command-r-plus-08-2024. https://huggingface.co/CohereForAI/ ↗
  36. Consortium, B. (2024). Balsam. https://benchmarks.ksaa.gov.sa/b/balsam. ↗
  37. Csaki, Z., Li, B., Li, J.L. et al (2024). SambaLingo: Teaching large language models new languages. In Sälevä, J. and Owodunni, A., editors, Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024), pages 1–21, Miami, Florida, USA. Association for Computational Linguistics. find
  38. Cui, G., Yuan, L., Ding, N. et al (2024). UltraFeedback: Boosting language models with scaled AI feedback. arXiv preprint arXiv:2310.01377. ↗
  39. Dagan, G., Synnaeve, G. and Rozière, B. (2024). Getting the most out of your tokenizer for pre-training and domain adaptation. arXiv preprint arXiv:2402.01035. ↗
  40. Deiseroth, B., Brack, M., Schramowski, P. et al (2024). T-FREE: Subword Tokenizer-Free generative LLMs via sparse representations for memory-efficient embeddings. In Al-Onaizan, Y., Bansal, M. and Chen, Y.N., editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21829–21851, Miami, Florida, USA. Association for Computational Linguistics. find
  41. Dodge, J., Sap, M., Marasović, A. et al (2021). Documenting large webtext corpora: A case study on the colossal clean crawled corpus. arXiv preprint arXiv:2104.08758. ↗
  42. Dubey, A., Jauhri, A., Pandey, A. et al (2024). arXiv:2407.21783. The llama 3 herd of models. arXiv preprint Durrani, N., Dalvi, F., Sajjad, H. et al (2019). One size does not fit all: Comparing NMT representations of different granularities. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1504–1516, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  43. Elfilali, A., Alobeidli, H., Fourrier, C. et al (2024). Open arabic llm leaderboard. https://huggingface.co/spaces/OALL/Open-Arabic-LLM-Leaderboard. ↗
  44. Elmagarmid, A.K., Ipeirotis, P.G. and Verykios, V.S. (2006). Duplicate record detection: A survey. IEEE Transactions on knowledge and data engineering, 19(1):1–16. find
  45. Eskimez, S.E., Wang, X., Thakker, M. et al (2024). E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts. arXiv preprint arXiv:2406.18009. ↗
  46. Ethayarajh, K., Xu, W., Muennighoff, N. et al (2024). Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306. ↗
  47. Farghaly, A. and Shaalan, K. (2009). Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing (TALIP), 8(4):1–22. find
  48. Ferguson, C.A. (1959). Diglossia. WORD, 15(2):325–340. find
  49. Fourrier, C., Habib, N., Wolf, T. et al (2023). LightEval: A lightweight framework for LLM evaluation. https://github.com/huggingface/lighteval. ↗
  50. FreedomIntelligence (2024). AceGPT-v2 Family Model Card. https://huggingface.co/collections/ FreedomIntelligence/acegpt-v2-670e00e5f9a7754078b8dcf2. Accessed on Dec 1, 2024. ↗
  51. Gao, L., Dai, Z., Pasupat, P. et al (2022). RARR: Researching and Revising What Language Models Say, Using Language Models. arXiv preprint arXiv:2210.08726. ↗ our card
  52. Gao, L., Tow, J., Abbasi, B. et al (2024). A framework for few-shot language model evaluation. https://zenodo.org/records/12608602. ↗
  53. Google (2024). Gemma 2 Family Model Card. https://huggingface.co/collections/google/ gemma-2-release-667d6600fd5220e7b967f315. Accessed on Dec 1, 2024. ↗
  54. Gregory, L., Thomure, H.T., Kazem, A. et al (2021). Advancing arabic language teaching and learning. World Bank, 10:35917. find
  55. Groeneveld, D., Beltagy, I., Walsh, P. et al (2024). Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838. ↗
  56. Gunter, T., Wang, Z., Wang, C. et al (2024). Apple intelligence foundation language models. arXiv preprint arXiv:2407.21075. ↗
  57. Habash, N.Y. (2010). Introduction to Arabic natural language processing. Morgan & Claypool Publishers. find
  58. Hassan, S., Mubarak, H., Abdelali, A. et al (2021). ASAD: Arabic social media analytics and unDerstanding. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 113–118, Online. Association for Computational Linguistics. find
  59. Heafield, K. (2011). Kenlm: Faster and smaller language model queries. In Proceedings of the sixth workshop on statistical machine translation, pages 187–197. find our card
  60. Healey, J.F. and Smith, G.R. (2012). A brief introduction to the Arabic alphabet, volume 3. Saqi. find
  61. Hendrycks, D., Burns, C., Basart, S. et al (2020). Measuring massive multitask language understanding. In International Conference on Learning Representations. find our card
  62. Hong, J., Lee, N. and Thorne, J. (2024). Orpo: Monolithic preference optimization without reference model. In Al-Onaizan, Y., Bansal, M. and Chen, Y.N., editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170–11189, Miami, Florida, USA. Association for Computational Linguistics. find
  63. Horesh, U. and Cotter, W.M. (2016). Current research on linguistic variation in the arabic-speaking world. Language and Linguistics Compass, 10:370–381. find
  64. Huang, H., Yu, F., Zhu, J. et al (2023). Acegpt, localizing large language models in arabic. arXiv preprint arXiv:2309.12053. ↗
  65. Inception (2024). Jais Family Model Card. https://huggingface.co/collections/inceptionai/ jais-family-66add8bb9c381f5492ddb6f4. Accessed on Dec 1, 2024. ↗
  66. Joty, S., Sajjad, H., Durrani, N. et al (2015). How to avoid unwanted pregnancies: Domain adaptation using neural network models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1259–1270, Lisbon, Portugal. Association for Computational Linguistics. find
  67. Khalil, I., Boshmaf, Y., Deniz, F. et al (2025). aiXamine: LLM Security and Privacy, Simplified. https://aixamine.qcri.org. Accessed: 2025-01-15. ↗
  68. Kheir, Y.E., Mubarak, H., Ali, A. et al (2024). Beyond orthography: Automatic recovery of short vowels and dialectal sounds in Arabic. In Ku, L.W., Martins, A. and Srikumar, V., editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13172–13184, Bangkok, Thailand. Association for Computational Linguistics. find our card
  69. Kocetkov, D., Li, R., allal, L.B. et al (2023). The stack: 3 TB of permissively licensed source code. Transactions on Machine Learning Research. find
  70. Koto, F., Li, H., Shatnawi, S. et al (2024). Arabicmmlu: Assessing massive multitask language understanding in arabic. arXiv preprint arXiv:2402.12840. ↗ our card
  71. Kudugunta, S., Caswell, I., Zhang, B. et al (2023). MADLAD-400: A multilingual and document-level large audited dataset. arXiv preprint arXiv:2309.04662. ↗ our card
  72. Lambert, N., Morrison, J., Pyatkin, V. et al (2024). T\” ULU 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124. ↗ our card
  73. Lee, K., Ippolito, D., Nystrom, A. et al (2022). Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499. ↗
  74. Lewis, P., Perez, E., Piktus, A. et al (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474. Curran Associates, Inc. find
  75. Li, D., Sun, Z., Hu, X. et al (2023a). A Survey of Large Language Models Attribution. arXiv preprint arXiv:2311.03731. ↗
  76. Li, J., Fang, A., Smyrnis, G. et al (2024). DataComp-LM: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11794. ↗ our card
  77. Li, X., Zhang, T., Dubois, Y. et al (2023b). AlpacaEval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval. ↗
  78. Liang, D., Gonen, H., Mao, Y. et al (2023). XLM-V: Overcoming the vocabulary bottleneck in multilingual masked language models. In Bouamor, H., Pino, J. and Bali, K., editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13142–13152, Singapore. Association for Computational Linguistics. find
  79. Lim, J.P. and Lauw, H. (2023). Disentangling transformer language models as superposed topic models. In Bouamor, H., Pino, J. and Bali, K., editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8646–8666, Singapore. Association for Computational Linguistics. find
  80. Liu, Y., Gu, J., Goyal, N. et al (2020). Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. find
  81. Logasa Bogen, P., Symons, C.T., McKenzie, A. et al (2013). Massively scalable near duplicate detection in streams of documents using MDSH . In 2013 IEEE International Conference on Big Data, pages 480–486, Los Alamitos, CA, USA. IEEE Computer Society. find
  82. Longpre, S., Mahari, R., Lee, A.N. et al (2024). Consent in crisis: The rapid decline of the ai data commons. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  83. Loshchilov, I. and Hutter, F. (2019). Decoupled weight decay regularization. In International Conference on Learning Representations. find
  84. Lozhkov, A., Ben Allal, L., von Werra, L. et al (2024). Fineweb-edu. https://huggingface.co/ datasets/HuggingFaceFW/fineweb-edu. ↗
  85. McOmber, M.L. (1995). Morpheme edges and arabic infixation. AMSTERDAM STUDIES IN THE THEORY AND HISTORY OF LINGUISTIC SCIENCE SERIES 4, pages 173–173. find
  86. Meng, Y., Xia, M. and Chen, D. (2024). Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734. ↗
  87. Meta (2024). Llama 3.1 Family Model Card. https://huggingface.co/collections/meta-llama/ llama-31-669fc079a0c406a149a5738f. Accessed on Dec 1, 2024. ↗
  88. Mousi, B., Durrani, N., Ahmad, F. et al (2024). AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs. In COLING 2025. arXiv:2409.11404. ↗
  89. Mubarak, H. and Darwish, K. (2014a). Automatic correction of Arabic text: A cascaded approach. In Habash, N. and Vogel, S., editors, Proceedings of the EMNLP 2014 workshop on Arabic natural language processing (ANLP), pages 132–136, Doha, Qatar. Association for Computational Linguistics. find
  90. Mubarak, H. and Darwish, K. (2014b). Using Twitter to collect a multi-dialectal corpus of Arabic. In Habash, N. and Vogel, S., editors, Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 1–7, Doha, Qatar. Association for Computational Linguistics. find
  91. Mubarak, H., Hussein, A., Chowdhury, S.A. et al (2021). QASR: QCRI Aljazeera Speech Resource. A Large Scale Annotated Arabic Speech Corpus. In Zong, C., Xia, F., Li, W. et al, editors, Proc. of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), pages 2274–2285, Online. Association for Computational Linguistics. find
  92. Muennighoff, N., Rush, A., Barak, B. et al (2023). Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358–50376. find
  93. Mustafa, M., Eldeen, A.S., Bani-Ahmad, S. et al (2017). A comparative survey on arabic stemming: approaches and challenges. Intelligent Information Management, 09:39–67. find
  94. Nagoudi, E.M.B., Elmadany, A. and Abdul-Mageed, M. (2022a). AraT5: Text-to-text transformers for Arabic language generation. arXiv preprint arXiv:2109.12068. ↗
  95. Nagoudi, E.M.B., Elmadany, A. and Abdul-Mageed, M. (2022b). AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics. find
  96. OpenAI (2024). Multilingual massive multitask language understanding. https://huggingface.co/ datasets/openai/MMMLU. ↗
  97. OpenAI, Achiam, J., Adler, S. et al (2024). GPT-4 technical report. arXiv preprint arXiv:2303.08774. ↗ our card
  98. Östling, R., Scherrer, Y., Tiedemann, J. et al (2017). The Helsinki neural machine translation system. In Proceedings of the Second Conference on Machine Translation, pages 338–347, Copenhagen, Denmark. Association for Computational Linguistics. find
  99. Otani, N., Ozaki, S., Zhao, X. et al (2020). Pre-tokenization of multi-word expressions in cross-lingual word embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4451–4464, Online. Association for Computational Linguistics. find
  100. Panayotov, V., Chen, G., Povey, D. et al (2015). Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE. find
  101. Paster, K., Santos, M.D., Azerbayev, Z. et al (2024). OpenWebMath: An open dataset of high-quality mathematical web text. In The Twelfth International Conference on Learning Representations. find our card
  102. Penedo, G., Malartic, Q., Hesslow, D. et al (2023). The RefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data only. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  103. Pernias, P., Rampas, D., Richter, M.L. et al (2023). Wuerstchen: An efficient architecture for large-scale text-to-image diffusion models. arXiv preprint arXiv:2306.00637. ↗
  104. Provilkov, I., Emelianenko, D. and Voita, E. (2020). BPE-dropout: Simple and effective subword regularization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1882–1892, Online. Association for Computational Linguistics. find our card
  105. PS, D., VG, J. and MS, A. (2024). Inference acceleration for large language models on CPUs. arXiv preprint arXiv:2406.07553. ↗
  106. Qwen Team (2024). Qwen2.5: A party of foundation models. https://qwenlm.github.io/blog/qwen2. 5/. ↗
  107. Rafailov, R., Sharma, A., Mitchell, E. et al (2024). Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36. find
  108. Raffel, C., Shazeer, N., Roberts, A. et al (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1). find
  109. Ram, O., Levine, Y., Dalmedigos, I. et al (2023). In-Context Retrieval-Augmented Language Models. arXiv:2302.00083 [cs]. ↗
  110. Rana, A. (2010). Common crawl–building an open web-scale crawl using hadoop. https://commoncrawl.org/blog/slideshare-building-a-scalable-web-crawler-with-hadoop. Accessed on Dec 1, 2024. ↗
  111. Rei, R., Stewart, C., Farinha, A.C. et al (2020). COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  112. Riviere, G.T.M., Pathak, S., Sessa, P.G. et al (2024). Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, abs/2408.00118. ↗
  113. Sajjad, H., Abdelali, A., Durrani, N. et al (2020). AraBench: Benchmarking dialectal Arabic-English machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5094–5107, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  114. Sajjad, H., Dalvi, F., Durrani, N. et al (2017a). Challenging language-dependent segmentation for Arabic: An application to machine translation and part-of-speech tagging. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 601–607, Vancouver, Canada. Association for Computational Linguistics. find our card
  115. Sajjad, H., Durrani, N., Dalvi, F. et al (2017b). Neural machine translation training in a multi-domain scenario. In Proceedings of the 14th International Conference on Spoken Language Translation, pages 66–73, Tokyo, Japan. International Workshop on Spoken Language Translation. find
  116. Sakaguchi, K., Bras, R.L., Bhagavatula, C. et al (2021). Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. find
  117. Schuhmann, C., Beaumont, R., Vencu, R. et al (2022). LAION-5B: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402. ↗ our card
  118. Sengupta, N., Sahu, S.K., Jia, B. et al (2023). Jais and Jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. ↗ our card
  119. Sennrich, R., Haddow, B. and Birch, A. (2016). Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Linguistics. find
  120. Shen, M. (2024). Rethinking data selection for supervised fine-tuning. arXiv preprint arXiv:2402.06094. ↗
  121. Shen, Z., Tao, T., Ma, L. et al (2024). Slimpajama-dc: Understanding data combinations for llm training. find
  122. Shibata, Y., Kida, T., Fukamachi, S. et al (1999). Byte pair encoding: A text compression scheme that accelerates pattern matching. Technical Report DOI-TR-161, Department of Informatics, Kyushu University. find
  123. Siuzdak, H. (2023). Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. arXiv preprint arXiv:2306.00814. ↗
  124. Slack, D., Wang, J., Semenenkov, D. et al (2023). A holistic approach for test and evaluation of large language models. https://static.scale.com/uploads/6019a18f03a4ae003acb1113/ test-and-evaluation.pdf. Accessed on Dec 1, 2024. ↗
  125. Smit, P., Virpioja, S., Grönroos, S.A. et al (2014). Morfessor 2.0: Toolkit for statistical morphological segmentation. In Wintner, S., Tadić, M. and Babych, B., editors, Proceedings of the Demonstrations at the 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 21–24, Gothenburg, Sweden. Aalto University, Association for Computational Linguistics. find
  126. Snell, C., Lee, J., Xu, K. et al (2024). Scaling llm test-time compute optimally can be more effective than scaling model parameters. find our card
  127. Soldaini, L., Kinney, R., Bhagia, A. et al (2024). Dolma: an open corpus of three trillion tokens for language model pretraining research. In Ku, L.W., Martins, A. and Srikumar, V., editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15725–15788, Bangkok, Thailand. Association for Computational Linguistics. find
  128. Soldaini, L. and Lo, K. (2023). peS2o (Pretraining Efficiently on S2ORC) Dataset. Technical report, Allen Institute for AI. ODC-By, https://github.com/allenai/pes2o. ↗
  129. Sun, Z., Wang, X., Tay, Y. et al (2022). Recitation-Augmented Language Models. arXiv preprint arXiv:2210.01296. ↗
  130. Team, G., Mesnard, T., Hardin, C. et al (2024). Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. ↗
  131. Team, N., Costa-jussà, M.R., Cross, J. et al (2022). No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. ↗
  132. Tokpanov, Y., Millidge, B., Glorioso, P. et al (2024). Zyda: A 1.3t dataset for open language modeling. arXiv preprint arXiv:2406.01981. ↗
  133. Touvron, H., Martin, L., Stone, K. et al (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. ↗
  134. Tversky, A. and Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases: Biases in judgments reveal some heuristics of thinking under uncertainty. science, 185(4157):1124–1131. find
  135. Vaswani, A., Shazeer, N., Parmar, N. et al (2017). Attention is all you need. In Guyon, I., Luxburg, U.V., Bengio, S. et al, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. find
  136. Versteegh, K. (2014). Arabic language. Edinburgh University Press. find
  137. Waldis, A., Perlitz, Y., Choshen, L. et al (2024). Holmes: Benchmark the linguistic competence of language models. arXiv preprint arXiv:2404.18923. ↗
  138. Wang, C., Cho, K. and Gu, J. (2020). Neural machine translation with byte-level subwords. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 9154–9160. find
  139. Wang, Y., Ivison, H., Dasigi, P. et al (2023a). How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36:74764–74786. find our card
  140. Wang, Z., Dong, Y., Delalleau, O. et al (2024). Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406.08673. ↗
  141. Wang, Z., Dong, Y., Zeng, J. et al (2023b). Helpsteer: Multi-attribute helpfulness dataset for steerlm. arXiv preprint arXiv:2311.09528. ↗
  142. Watson, J.C. (2002). The phonology and morphology of Arabic. Oxford University Press, USA. find
  143. Weber, M., Fu, D., Anthony, Q. et al (2024). RedPajama: an open dataset for training large language models. arXiv preprint arXiv:2411.12372. ↗ our card
  144. Woo, S., Debnath, S., Hu, R. et al (2023). Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16133–16142. find
  145. Xia, M., Malladi, S., Gururangan, S. et al (2024). Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333. ↗ our card
  146. Xu, S., Pang, L., Shen, H. et al (2023). Search-in-the-Chain: Towards Accurate, Credible and Traceable Content Generation for Complex Knowledge-Intensive Tasks. arXiv preprint arXiv:2304.14732. ↗
  147. Zellers, R., Holtzman, A., Bisk, Y. et al (2019). HellaSwag: Can a machine really finish your sentence? In Korhonen, A., Traum, D. and Màrquez, L., editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. find
  148. Zeng, W., Ren, X., Su, T. et al (2021). PanGu-α: Large-scale autoregressive pretrained Chinese language models with auto-parallel computation. arXiv preprint arXiv:2104.12369. ↗
  149. Zhang, C., D’Haro, L.F., Chen, Y. et al (2024). A comprehensive analysis of the effectiveness of large language models as automatic dialogue evaluators. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19515–19524. find
  150. Zhao, H., Andriushchenko, M., Croce, F. et al (2024). Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning. arXiv preprint arXiv:2402.04833. ↗
  151. Zheng, L., Chiang, W.L., Sheng, Y. et al (2023). Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  152. Zhou, J., Lu, T., Mishra, S. et al (2023). Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911. ↗ our card
  153. Zhu, B., Frick, E., Wu, T. et al (2024). Starling-7B: Improving helpfulness and harmlessness with RLAIF. In First Conference on Language Modeling. find