Elamur Research⌕
☰
home / papers / 2024 / SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corpora

SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corpora

Summary

Saudi dialect is hard to process. Its spelling is not standardized, it is full of slang and regional expressions, and it differs from Modern Standard Arabic (MSA). Earlier Arabic language models were trained on MSA or on mixed-dialect data, and existing Saudi corpora were too small or mostly MSA to support pretraining. Faisal Qarah addresses this by building two new Saudi-only corpora. The Saudi Forums Corpus (SFC) gathers 15.2 GB of text drawn from five Saudi forums, while the Saudi Tweets Mega Corpus (STMC) contains over 141 million tweets. On these he pretrains SaudiBERT from scratch, a BERT-style model limited to a single dialect. He fine-tunes and compares it with six multidialect Arabic models on 11 Saudi-dialect datasets, split into a sentiment group and a text-classification group. SaudiBERT has the best average macro F1 in both groups, and it is smaller and needed less pretraining than the strongest competitor. The work shows that a focused corpus for one dialect can beat broader models trained on far more data.

Key points

  • New corpora. STMC (141M+ Saudi tweets) and SFC (15.2 GB from five forums) are the largest Saudi-dialect corpora reported. STMC and an SFC-mini subset are released on Hugging Face.
  • Results. Average macro F1 is 86.15% on the six sentiment datasets and 87.86% on the five classification datasets. The runner-up, MARBERTv2, scores 85.18% and 87.44%.
  • Comparison set and tasks. The baselines are AraBERTv0.2-Twitter, QARiB, CAMeLBERT-DA, MARBERTv1, MARBERTv2 and AraRoBERTa-SA. Tasks include sarcasm/irony, gender, events, sentiment and dialect identification. Scores are reported on validation sets.
  • Efficiency.
    • SaudiBERT has 143M parameters and a 75k-wordpiece vocabulary.
    • MARBERTv2 has 163M parameters and a 100k-wordpiece vocabulary.
    • SaudiBERT was pretrained for 12 epochs, against MARBERTv2's 40 epochs on 160 GB.
  • Only classification loss. SaudiBERT trails MARBERTv2 only on SDC-EDC, a task with a large share of Egyptian-dialect text. This hints that the narrow model gives up some coverage outside its dialect.

Interesting ideas

  • I1. A model trained only on the target language variety, with a smaller vocabulary fitted to it, can beat larger general models while costing less to train. — For Mova (tokens): Building tokenizer and lemma resources from the actual register Mova reads, such as fables and school texts, could beat generic web-derived lexicons with fewer entries.
  • I2. The author checked what existing "dialect" corpora really contain and found that KSUSC is about 89% MSA, so its advertised size overstates the useful dialect data. — For Mova (learn): Before training on silver data, measure what share actually matches the target genre or construction, rather than trusting the dataset's label.
  • I3. A dialect-identification task was created cheaply by tagging two single-dialect corpora with their source and merging them. — For Mova (learn): Merging texts from known sources, such as tales and news, gives free labeled sets for register-sensitive checks of the tagger or parser.
  • I4. The one task the specialized model lost contained a lot of text outside its dialect, which exposes where narrow pretraining stops helping. — For Mova (tree): Include deliberately off-register text in parser evaluation as a control, so that gains from specializing gparse are not mistaken for general gains.
  • I5. Results are given per task and as unweighted group averages, alongside model size and pretraining epochs, so accuracy can be weighed against cost. — For Mova (learn): Report each component's accuracy next to its training cost and per-dataset breakdown, so cheaper explainable components can be fairly compared with heavier ones.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

10 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2405.06239 · arXiv 2405.06239

@misc{qarah2024saudibert,
  title = {{SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corpora}},
  author = {Qarah, Faisal},
  year = {2024},
  doi = {10.48550/arxiv.2405.06239},
  eprint = {2405.06239},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.06239}
}

References

48 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Saudi Vision 2030. https://www.vision2030.gov.sa/en/ ↗
  2. Prochazka, T.: Saudi Arabian Dialects. Routledge, ??? (2013) find
  3. Al-Twairesh, N., Al-Matham, R., Madi, N., Almugren, N., Al-Aljmi, A.-H., Alshalan, S., Alshalan, R., Alrumayyan, N., Al-Manea, S., Bawazeer, S., et al.: Suar: Towards building a corpus for the saudi dialect. Procedia computer science 142, 72–82 (2018) find
  4. Alruily, M.: Issues of dialectal saudi twitter corpus. Int. Arab J. Inf. Technol. 17(3), 367–374 (2020) find
  5. Mubarak, H., Darwish, K.: Using twitter to collect a multi-dialectal corpus of arabic. In: Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pp. 1–7 (2014) find
  6. Khalifa, S., Habash, N., Abdulrahim, D., Hassan, S.: A large scale corpus of gulf arabic. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pp. 4282–4289 (2016) find
  7. Tarmom, T., Teahan, W., Atwell, E., Alsalka, M.A.: Compression versus traditional machine learning classifiers to detect code-switching in varieties and dialects: Arabic as a case study. Natural Language Engineering 26(6), 663–676 (2020) find
  8. Pasha, A., Al-Badrashiny, M., Diab, M.T., El Kholy, A., Eskander, R., Habash, N., Pooleery, M., Rambow, O., Roth, R.: Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In: Lrec, vol. 14, pp. 1094–1101 (2014) find our card
  9. Addawood, A., Alzeer, D.: Rewayatech: Saudi web novels dataset (2020) find
  10. Elgibreen, H., Faisal, M., Al Sulaiman, M., Abdou, S., Mekhtiche, M.A., Moussa, A.M., Alohali, Y.A., Abdul, W., Muhammad, G., Rashwan, M., et al.: An incremental approach to corpus design and construction: application to a large contemporary saudi corpus. IEEE Access 9, 88405–88428 (2021) find
  11. Antoun, W., Baly, F., Hajj, H.: Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104 (2020) ↗ our card
  12. Safaya, A., Abdullatif, M., Yuret, D.: Kuisail at semeval-2020 task 12: Bert-cnn for offensive speech identification in social media. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, pp. 2054–2059 (2020) find our card
  13. Antoun, W., Baly, F., Hajj, H.: Aragpt2: Pre-trained transformer for arabic language generation. In: Proceedings of the Sixth Arabic Natural Language Processing Workshop, pp. 196–207 (2021) find
  14. Abdul-Mageed, M., Elmadany, A., Nagoudi, E.M.B.: Arbert & marbert: deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785 (2020) ↗ our card
  15. Chowdhury, S.A., Abdelali, A., Darwish, K., Soon-Gyo, J., Salminen, J., Jansen, B.J.: Improving arabic text categorization using transformer training diversification. In: Proceedings of the Fifth Arabic Natural Language Processing Workshop, pp. 226–236 (2020) find
  16. Parker, R., et al.: Arabic Gigaword Fifth Edition LDC2011T11. Linguistic Data Consortium, Philadelphia (2011) find
  17. El-Khair, I.A.: 1.5 billion words arabic corpus. arXiv preprint arXiv:1611.04033 (2016) ↗
  18. Lison, P., Tiedemann, J.: Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pp. 923–929 (2016) find our card
  19. Suárez, P.J.O., Sagot, B., Romary, L.: Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In: 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7) (2019). Leibniz-Institut für Deutsche Sprache find
  20. Inoue, G., Alhafni, B., Baimukan, N., Bouamor, H., Habash, N.: The interplay of variant, size, and task type in arabic pre-trained language models. arXiv preprint arXiv:2103.06678 (2021) ↗ our card
  21. Zeroual, I., Goldhahn, D., Eckart, T., Lakhouaja, A.: Osian: Open source international arabic news corpus-preparation and integration into the clarin-infrastructure. In: Proceedings of the Fourth Arabic Natural Language Processing Workshop, pp. 175–182 (2019) find
  22. Elmadany, A., Nagoudi, E.B., Mageed, M.A.: Orca: A challenging benchmark for arabic language understanding. In: Findings of the Association for Computational Linguistics: ACL 2023, pp. 9559–9586 (2023) find
  23. Elgezouli, M., Elmadani, K.N., Saeed, M.: Sudabert: A pre-trained encoder representation for sudanese arabic dialect. In: 2020 International Conference on Computer, Control, Electrical, and Electronics Engineering (ICCCEEE), pp. 1–4 (2021). IEEE find
  24. Haddad, H., Rouhou, A.C., Messaoudi, A., Korched, A., Fourati, C., Sellami, A., Ben HajHmida, M., Ghriss, F.: Tunbert: pretraining bert for tunisian dialect understanding. SN Computer Science 4(2), 194 (2023) find
  25. Abdaoui, A., Berrimi, M., Oussalah, M., Moussaoui, A.: Dziribert: a pre-trained language model for the algerian dialect. arXiv preprint arXiv:2109.12346 (2021) ↗ our card
  26. Gaanoun, K., Naira, A.M., Allak, A., Benelallam, I.: Darijabert: a step forward in nlp for the written moroccan dialect. International Journal of Data Science and Analytics, 1–13 (2024) find
  27. Moussaoui, O., El Younnoussi, Y.: Pre-training two bert-like models for moroccan dialect: Morroberta and morrbert. In: MENDEL, vol. 29, pp. 55–61 (2023) find
  28. AlYami, R., Al-Zaidy, R.: Weakly and semi-supervised learning for arabic text classification using monodialectal language models. In: Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pp. 260–272 (2022) find
  29. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017) find
  30. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018) ↗
  31. Richardson, L.: Beautiful soup documentation. April (2007) find
  32. Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al.: Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016) ↗
  33. Kudo, T., Richardson, J.: Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226 (2018) ↗ our card
  34. He, P., Liu, X., Gao, J., Chen, W.: Deberta: Decoding-enhanced bert with disentangled attention. In: International Conference on Learning Representations (2021). https://openreview.net/forum?id=XPZIaotutsD ↗ our card
  35. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: Albert: A lite bert for self-supervised learning of language representations. In: International Conference on Learning Representations (2019) find
  36. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., Stoyanov, V.: Unsupervised cross-lingual representation learning at scale. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020). Association for Computational Linguistics find
  37. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019) ↗
  38. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.: Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45 (2020) find our card
  39. Alqurashi, T.: Arabic sentiment analysis for twitter data: a systematic literature review. Engineering, Technology & Applied Science Research 13(2), 10292–10300 (2023) find
  40. Almutairi, S.M., Alotaibi, F.M.: A comparative analysis for arabic sentiment analysis models in e-marketing using deep learning techniques. Journal of Engineering and Applied Sciences 10(1) (2023) find
  41. Almuqren, L., Cristea, A.: Aracust: a saudi telecom tweets corpus for sentiment analysis. PeerJ Computer Science 7, 510 (2021) find
  42. Alyami, S.N., Olatunji, S.O.: Application of support vector machine for arabic sentiment classification using twitter-based dataset. Journal of Information & Knowledge Management 19(01), 2040018 (2020) find
  43. Alduraibi, M., Alrefaey, R., Alqahmi, R., Almatrafi, S., Alayed, A.: SaudiShopInsights Dataset: Saudi Customer Reviews in Clothes and Electronics. https://doi.org/10.21227/6e56-4e15 . https://dx.doi.org/10.21227/6e56-4e15 ↗
  44. Alqahtani, D., Alzahrani, L., Bahareth, M., Alshameri, N., Al-Khalifa, H., Aldhubayi, L.: Customer sentiments toward saudi banks during the covid-19 pandemic. In: Proceedings of the 5th International Conference on Natural Language and Speech Processing (ICNLSP 2022), pp. 251–257 (2022) find
  45. Bayazed, A., Torabah, O., AlSulami, R., Alahmadi, D., Babour, A., Saeedi, K.: Sdct: multi-dialects corpus classification for saudi tweets. International Journal of Advanced Computer Science and Applications 11(11) (2020) find
  46. AlMazrua, H., AlHazzani, N., AlDawod, A., AlAwlaqi, L., AlReshoudi, N., Al-Khalifa, H., AlDhubayi, L.: Sa ‘7r: A saudi dialect irony dataset. In: Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pp. 60–70 (2022) find
  47. Hamoui, B., Mars, M., Almotairi, K.: Flodusta: Saudi tweets dataset for flood, dust storm, and traffic accident events. In: Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 1391–1396 (2020) find
  48. Alanazi, S.A.: Toward identifying features for automatic gender detection: A corpus creation and analysis. IEEE Access 7, 111931–111943 (2019) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qarah-2024-saudibert-large-language-model-pretrained/