Elamur Research⌕
☰
home / papers / 2024 / ArabianGPT: Native Arabic GPT-based Large Language Model

ArabianGPT: Native Arabic GPT-based Large Language Model

Summary

The paper argues that Arabic lacks generative language models built mainly around Arabic itself. Many existing models are multilingual or bilingual, and their vocabularies carry many English tokens, which the authors see as a poor fit for Arabic's rich morphology. Their answer is ArabianGPT, part of a wider ArabianLLM effort. It keeps the GPT-2 architecture and trains models of two sizes, ArabianGPT-0.1B and ArabianGPT-0.3B, on Arabic-only corpora. Both use a dedicated tokenizer, AraNizer, designed around Arabic script and word structure. The smaller model learns from a large news collection, and the larger one from a mixed corpus drawn from three sources. The base models are then fine-tuned for sentiment analysis, summarization and question answering. The strongest result is in sentiment classification, where accuracy jumps from roughly half correct to nearly all correct. On standard benchmark suites the two sizes score almost the same on average. The work matters mainly as an openly described recipe for a language-specific LLM, from corpus to tokenizer to evaluation.

Key points

  • Dataset D1, used for the 0.1B model: 5.37 million Arabic newspaper articles (15.5 GB, about 1.8B tokens) from the early 2000s to 2014. The authors explicitly warn about this 2014 knowledge cutoff.
  • Dataset D2, used for ArabianGPT-0.3B: three combined sources totalling 23 GB and about 3.34B tokens, mixing formal, colloquial, historical and contemporary text.
  • AraNizer is a 64K-vocabulary Arabic tokenizer. The models are built on the GPT-2 architecture and were trained and fine-tuned on A100 GPUs.
  • Sentiment analysis with ArabianGPT-0.1B rose from 56% accuracy (base) to 95% after 3 epochs of fine-tuning.
  • Benchmarks after fine-tuning (ARC 25-shot, HellaSwag 10-shot, MMLU 5-shot, TruthfulQA 0-shot):
    • The average score is about 31 for both sizes (QA: 31.2 vs 31.4; summarization: 31.7 vs 31.8).
    • The 0.3B model is stronger on HellaSwag but weaker on TruthfulQA, which the authors read as a sign of undertraining for its size.

Interesting ideas

  • I1. Build the vocabulary from the target language alone, so that foreign subwords do not take up slots and fragment native word forms; this is useful wherever morphology is rich. — For Mova (tokens): Audit the en tokenizer and lemma inventory for junk or foreign pieces that split English words oddly, and measure how often segmentation breaks morpheme boundaries.
  • I2. State the time span of the training corpus as an explicit limitation, so users know which events and vocabulary the model cannot know. — For Mova (world): Stamp compiled knowledge (Wikidata facts, ontology entries) with a validity date, so the world of a text can flag claims that depend on post-cutoff facts.
  • I3. Run the same benchmark battery before and after each fine-tuning, so task-specific gains and silent regressions show up side by side. — For Mova (learn): After every retraining on silver data, rerun a fixed suite plus negative controls and report per-task deltas, not only the headline metric.
  • I4. When a bigger model fails to beat a smaller one on some tests, treat that as a sign that data or training steps did not scale with parameters, rather than as noise. — For Mova (learn): When gparse grows in size, check whether treebank and silver data volumes grew too, before concluding that the architecture change does not help.
  • I5. Show sample outputs next to scores, because near-identical averages can hide large qualitative differences, such as one model giving relevant answers and the other incoherent ones. — For Mova (answer): Attach a few explained example answers to every QA-on-fables or word-problem score, so reviewers can see whether gains are real reasoning or lucky guessing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

14 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 11 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.15313 · arXiv 2402.15313

@misc{koubaa2024arabiangpt,
  title = {{ArabianGPT: Native Arabic GPT-based Large Language Model}},
  author = {Koubaa, Anis and Ammar, Adel and Ghouti, Lahouari and Najar, Omar and Sibaee, Serry},
  year = {2024},
  doi = {10.48550/arxiv.2402.15313},
  eprint = {2402.15313},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.15313}
}

References

19 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. In Advances in neural information processing systems, volume 30, 2018. find
  2. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  3. O. ElJundi, W. Antoun, N. El Droubi, H. Hajj, W. El-Hajj, and K. Shaban. hulmona: The universal language model in arabic. In Proceedings of the fourth Arabic natural language processing workshop, pages 68–77, August 2019. find
  4. W. Antoun, F. Baly, and H. Hajj. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104, 2020. ↗ our card
  5. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, and D. Amodei. Language models are few-shot learners. In Advances in neural information processing systems, volume 33, pages 1877–1901, 2020. find
  6. N. Sengupta, S. K. Sahu, B. Jia, S. Katipomu, H. Li, F. Koto, and E. Xing. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149, 2023. ↗ our card
  7. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Ziche Liu, et al. Acegpt, localizing large language models in arabic. arXiv preprint arXiv:2309.12053, 2023. ↗ our card
  8. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. An empirical study of pre-trained transformers for arabic information extraction. arXiv preprint arXiv:2004.14519, 2020. ↗
  9. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. Arbert & marbert: deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785, 2020. ↗ our card
  10. Wissam Antoun, Fady Baly, and Hazem Hajj. Araelectra: Pre-training text discriminators for arabic language understanding. arXiv preprint arXiv:2012.15516, 2020. ↗
  11. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. The interplay of variant, size, and task type in arabic pre-trained language models. arXiv preprint arXiv:2103.06678, 2021. ↗ our card
  12. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. Arat5: Text-to-text transformers for arabic language generation. arXiv preprint arXiv:2109.12068, 2021. ↗ our card
  13. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. Pre-training bert on arabic tweets: Practical considerations. arXiv preprint arXiv:2102.10684, 2021. ↗ our card
  14. Abbas Ghaddar, Yimeng Wu, Sunyam Bagga, Ahmad Rashid, Khalil Bibi, Mehdi Rezagholizadeh, Chao Xing, Yasheng Wang, Xinyu Duan, Zhefeng Wang, et al. Revisiting pre-trained language models and their evaluation for arabic natural language processing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3135–3151, 2022. find
  15. O. Najar, S. Sibaee, L. Ghouti, and A. Koubaa. Aranizer 0.1.8. https://pypi.org/project/aranizer, 2023. ↗
  16. T. Kudo and J. Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018. ↗ our card
  17. Yusuxke Shibata, Takuya Kida, Shuichi Fukamachi, Masayuki Takeda, Ayumi Shinohara, Takeshi Shinohara, and Setsuo Arikawa. Byte pair encoding: A text compression scheme that accelerates pattern matching. 1999. find
  18. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, and I. Polosukhin. Attention is all you need. In Advances in neural information processing systems, volume 30, 2017. find
  19. CSEBUETNLP. XL-Sum Arabic Dataset. https://huggingface.co/datasets/csebuetnlp/xlsum/viewer/arabic, 2024. Accessed on: 25/01/2024. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/koubaa-2024-arabiangpt-native-arabic-gpt-based/