Elamur Research⌕
☰
home / papers / 2021 / Pre-Training BERT on Arabic Tweets: Practical Considerations

Pre-Training BERT on Arabic Tweets: Practical Considerations

Summary

The paper asks what actually matters when building a BERT model from scratch for Arabic social media. This text is largely dialectal, has no fixed spelling, and is full of hashtags, mentions and emojis. The authors gathered about 440M deduplicated Arabic tweets and added formal sources (Arabic Gigaword, Abu El-Khair Corpus, Open Subtitles). From this data they pre-trained five base-size BERT variants, called QARiB. The variants differ in corpus size (10–60GB), in the balance of tweets and MSA, and in whether words were split into morphemes with Farasa before BPE. All models used masked language modeling only and were trained on a Cloud TPU. Multiple checkpoints were fine-tuned on five tweet tasks and compared with mBERT, AraBERT and ArabicBERT. QARiB variants score best on three of the five tasks. The main lessons are:

  • gains from extra data level off quickly;
  • adding formal text helps even on tweet tasks;
  • Farasa segmentation gives clear gains on some tasks;
  • loss and step count are unreliable for choosing a checkpoint.

Key points

  • Tweet corpus: 1.4B tweets collected in 2012–2020 through the Twitter streaming API ("lang:ar"), reduced to over 440M unique tweets (2.7B words) after deduplication.
  • Five models: QARiB10 and QARiB25 (tweets only), QARiB25 mix and QARiB60 mix (balanced tweets + MSA), and QARiB25 mix far (Farasa-segmented). All use BPE instead of WordPiece and drop next-sentence prediction.
  • Evaluation tasks:
    • AJGT (sentiment)
    • SemEval 2018 Task 1 E-c (emotion)
    • ANERCORP plus tweet NER sets
    • OffensEval 2020
    • QADI (dialect identification, 18 countries)
  • Best results:
    • QARiB25 mix far reaches 69.1 F1 on NER, compared with 66.6 for AraBERTv1.
    • QARiB25 mix reaches 46.8 on Emotion.
    • QARiB25 and QARiB60 mix both reach 90.0 on Offensive.
    • AraBERTv1 stays best on AJGT (93.6), and ArabicBERT stays best on QADI (61.7).
    • mBERT trails on every task.
  • Compute: a single V100 was estimated to need over a year per model. A TPU v2 used through Colab, with 12-hour sessions, did the job in days. Data was preprocessed and sharded locally to keep cloud costs down.

Interesting ideas

  • I1. Normalized deduplication key. Duplicates were detected by comparing an aggressively normalized copy of each tweet, with no diacritics, URLs, mentions or numbers, unified letter variants, and limited character repetition. This removes near-duplicates such as retweets and spam variants, and the same trick fits any noisy user-generated corpus.
  • I2. Two-stage tokenization. Text is first split by a morphological segmenter, and BPE then runs on top of it to keep the vocabulary bounded. This could help other morphologically rich languages, Ukrainian included, where prefixes and suffixes inflate subword vocabularies.
  • I3. Checkpoint selection by task battery. Instead of trusting loss or training length, the authors evaluate saved checkpoints across several downstream tasks and pick the best one overall. Performance fluctuated as training went on, so this is cheap insurance for any pre-training project.
  • I4. Match the pre-training objective to text length. Next-sentence prediction was dropped because tweets are usually a single sentence. It is a reminder to question default objectives when the target domain consists of short texts.
  • I5. Mix formal text into informal corpora. Blending formal text with informal data outperformed informal-only training even on informal benchmarks. This suggests that genre diversity can matter more than strict domain matching when building corpora for dialects or social media.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

69 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 66 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2102.10684 · arXiv 2102.10684

@misc{abdelali2021pre,
  title = {{Pre-Training BERT on Arabic Tweets: Practical Considerations}},
  author = {Abdelali, Ahmed and Hassan, Sabit and Mubarak, Hamdy and Darwish, Kareem and Samih, Younes},
  year = {2021},
  doi = {10.48550/arxiv.2102.10684},
  eprint = {2102.10684},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2102.10684}
}

References

30 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016a. Farasa: A fast and furious segmenter for Arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16, San Diego, California. Association for Computational Linguistics. find our card
  2. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016b. Farasa: A fast and furious segmenter for arabic. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Demonstrations, pages 11–16. find our card
  3. Ahmed Abdelali, Hamdy Mubarak, Younes Samih, Sabit Hassan, and Kareem Darwish. 2020. Arabic dialect identification in the wild. arXiv preprint arXiv:2005.06557. ↗
  4. Khaled Mohammad Alomari, Hatem M. ElSherif, and Khaled Shaalan. 2017. Arabic tweets sentimental analysis using machine learning. In Advances in Artificial Intelligence: From Theory to Practice, pages 602–610, Cham. Springer International Publishing. find
  5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. ↗ our card
  6. Yassine Benajiba, Mona Diab, and Paolo Rosso. 2008. Arabic named entity recognition using optimized feature sets. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 284–293. find
  7. José Canete, Gabriel Chaperon, and Rodrigo Fuentes. 2019. Spanish pre-trained bert model and evaluation data. PML4DC at ICLR 2020. find
  8. Abdelghani Dahou, Shengwu Xiong, Junwei Zhou, and Mohamed Abd Elaziz. 2019. Multi-channel embedding convolutional neural network model for arabic sentiment classification. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 18(4). ↗
  9. Kareem Darwish. 2013. Named entity recognition using cross-lingual resources: Arabic as an example. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1558–1567. find
  10. Kareem Darwish and Wei Gao. 2014. Simple effective microblog named entity recognition: Arabic as an example. In LREC, pages 2513–2517. find
  11. Pieter Delobelle, Thomas Winters, and Bettina Berendt. 2020. Robbert: a dutch roberta-based language model. find our card
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North. find
  13. Marc Djandji, Fady Baly, Wissam Antoun, and Hazem Hajj. 2020. Multi-task learning using AraBert for offensive language detection. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 97–101, Marseille, France. European Language Resource Association. find
  14. Ibrahim Abu El-Khair. 2016. 1.5 billion words arabic corpus. arXiv preprint arXiv:1611.04033. ↗
  15. Pierre Lison and Jörg Tiedemann. 2016. Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France. European Language Resources Association (ELRA). find our card
  16. Louis Martin, B. Muller, Pedro Javier Ortiz Suárez, Y. Dupont, L. Romary, ’Eric de la Clergerie, Djamé Seddah, and Benoı̂t Sagot. 2020. Camembert: a tasty french language model. ArXiv, abs/1911.03894. ↗ our card
  17. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. SemEval-2018 task 1: Affect in tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 1–17, New Orleans, Louisiana. Association for Computational Linguistics. find our card
  18. Hamdy Mubarak, Ammar Rashed, Kareem Darwish, Younes Samih, and Ahmed Abdelali. 2020. Arabic offensive language on twitter: Analysis and experiments. ↗
  19. Pedro Javier Ortiz Suárez, Laurent Romary, and Benoı̂t Sagot. 2020. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703–1714, Online. Association for Computational Linguistics. find our card
  20. Robert Plutchik. 1991. The emotions. University Press of America. find
  21. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. find
  22. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. Kuisail at semeval-2020 task 12: Bert-cnn for offensive speech identification in social media. find
  23. Stefan Schweter. 2020. Berturk - bert models for turkish. find
  24. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Linguistics. find
  25. Bashar Talafha, Mohammad Ali, Muhy Eddin Za’ter, Haitham Seelawi, Ibraheem Tuffaha, Mostafa Samir, Wael Farhan, and Hussein T. Al-Natsheh. 2020. Multi-dialect arabic bert for country-level dialect identification. ↗
  26. Huishuang Tian, Kexin Yang, Dayiheng Liu, and Jiancheng Lv. 2020. Anchibert: A pre-trained model for ancient chineselanguage understanding and generation. arXiv preprint arXiv:2009.11473. ↗
  27. Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: Bert for finnish. ↗ our card
  28. Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. Bertje: A dutch bert model. ↗ our card
  29. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, ukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. find
  30. Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. 2020. SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020). In Proceedings of SemEval. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abdelali-2021-pre-training-bert-arabic-tweets/