Elamur Research⌕
☰
home / papers / 2021 / DziriBERT: a Pre-trained Language Model for the Algerian Dialect

DziriBERT: a Pre-trained Language Model for the Algerian Dialect

Summary

Algerian Arabic (Darja) dominates informal online writing in Algeria. It is often typed in Latin letters with digits standing in for Arabic sounds (Arabizi), and it mixes vocabulary from Arabic, Tamazight, French and other languages. Models built for standard Arabic or for many languages at once therefore handle it poorly. The authors gathered about 1.2 million tweets from large Algerian cities, finding them through common dialect keywords. After cleaning, roughly 150 MB of text remained. They used it to train DziriBERT, a BERT Base encoder with its own WordPiece vocabulary, pre-trained with masked language modeling only. They fine-tuned it on four public classification sets, Twifil and Narabizi, against multilingual and Arabic baselines. DziriBERT roughly ties MARBERT on Twifil, which is mostly in Arabic script. On Narabizi, which is entirely Latin-script, it wins clearly. MARBERT was trained on about a thousand times more text. The study shows that a small, carefully targeted corpus can beat huge generic pre-training for an underserved language variety, and that the resulting model is also more compact.

Key points

  • Pre-training corpus: 1.1M tweets after deduplication and removal of entries shorter than three tokens (about 20M tokens, ~150 MB). The data was split into 1M tweets for training and 100k for testing. Mentions, emails and links were replaced with placeholders.
  • Training setup:
    • BERT Base (12 layers, 12 heads, hidden size 768) with a 50k WordPiece vocabulary and no NSP objective.
    • Masking rate raised to 25% from the standard 15%, since tweets are short.
    • Batch size 64 on a single NVIDIA T4, for about 10 days (50 epochs, ~800k steps).
  • Evaluation datasets:
    • Twifil sentiment: 9,437 tweets, 3 classes.
    • Twifil emotion: 5,110 tweets, 10 Plutchik classes.
    • Narabizi sentiment: 1,279 sentences, 4 classes.
    • Narabizi topic: 1,279 sentences, 5 classes.
    • Baselines: mBERT, XLM-R, AraBERT, QARiB, CamelBERT-da, CamelBERT-mix and MARBERT.
  • Results (DziriBERT vs MARBERT):
    • Narabizi accuracy: 63.5 vs 58.0 on sentiment, 62.8 vs 49.0 on topic.
    • Narabizi topic macro-F1: 54.8 vs 31.0.
    • Twifil: essentially tied (sentiment 80.5 vs 80.6, emotion 70.4 vs 70.2).
  • Model size: 124M parameters and 498 MB on disk, compared with 278M parameters and 1,147 MB for XLM-R. The difference comes mostly from vocabulary size, since embeddings hold a large share of the weights.

Interesting ideas

  • I1. Seeding data collection with a handful of register-typical keywords, combined with a location filter, quickly yields a corpus for a variety that has no existing dataset; this can bootstrap any niche register cheaply. — For Mova (learn): Raw text for silver parsing in under-covered registers (fables, school word problems, informal English) could be harvested by seeding searches with genre-marker words, then parsed by the witness parsers.
  • I2. Raising the masking rate when training sequences are short keeps enough prediction targets per example, so pre-training stays efficient on tweet-length input. — For Mova (tree): If gparse's encoder is pre-trained or adapted on short sentences such as test items or dialogue lines, a higher mask rate is a cheap knob worth trying before collecting more text.
  • I3. A compact vocabulary trained on in-domain text cuts total parameters noticeably, because the embedding matrix is a major part of the model; smaller models are easier to ship and run. — For Mova (tokens): Fitting the subword vocabulary of the GPU parser to Mova's actual target text could shrink the model while keeping coverage of its domain words.
  • I4. Error analysis showed the benchmark itself contained off-target language, such as standard Arabic and other dialects, which explained part of a rival's strength; inspecting what the test data really contains changes how scores should be read. — For Mova (learn): Evaluation sets could be audited and split by text type, for example nonstandard spelling or archaic tale language, so a score gap is attributed to the right cause rather than to the parser in general.
  • I5. Every model was run with the same ten published random seeds, and the seeds were shipped in the evaluation script, which makes comparisons repeatable and less sensitive to lucky runs. — For Mova (learn): Training and evaluating parsers or repair rules over a fixed, versioned seed list would let negative-control comparisons be rerun exactly and reported as averages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

31 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 27 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2109.12346 · arXiv 2109.12346

@misc{abdaoui2021dziribert,
  title = {{DziriBERT: a Pre-trained Language Model for the Algerian Dialect}},
  author = {Abdaoui, Amine and Berrimi, Mohamed and Oussalah, Mourad and Moussaoui, Abdelouahab},
  year = {2021},
  doi = {10.48550/arxiv.2109.12346},
  eprint = {2109.12346},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2109.12346}
}

References

21 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Jun. 2019, pp. 4171–4186. [Online]. Available: https://www.aclweb.org/anthology/N19-1423 ↗
  2. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33, 2020, pp. 1877–1901. [Online]. Available: https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf ↗
  3. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in Advances in Neural Information Processing Systems, C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, Eds., vol. 26. Curran Associates, Inc., 2013. [Online]. Available: https://proceedings.neurips.cc/paper/2013/file/9aa42b31882ec039965f3c4923ce901b-Paper.pdf ↗
  4. J. Pennington, R. Socher, and C. Manning, “GloVe: Global vectors for word representation,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Oct. 2014, pp. 1532–1543. [Online]. Available: https://www.aclweb.org/anthology/D14-1162 ↗
  5. L. Martin, B. Muller, P. J. O. Suárez, Y. Dupont, L. Romary, É. V. de la Clergerie, D. Seddah, and B. Sagot, “Camembert: a tasty french language model,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. [Online]. Available: https://www.aclweb.org/anthology/2020.acl-main.645 ↗ our card
  6. M. Polignano, P. Basile, M. de Gemmis, G. Semeraro, and V. Basile, “Alberto: Italian bert language understanding model for nlp challenging tasks based on tweets,” in 6th Italian Conference on Computational Linguistics, CLiC-it 2019, vol. 2481, 11 2019. [Online]. Available: http://ceur-ws.org/Vol-2481/paper57.pdf ↗ our card
  7. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, É. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Unsupervised cross-lingual representation learning at scale,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 8440–8451. [Online]. Available: https://www.aclweb.org/anthology/2020.acl-main.747.pdf ↗
  8. J. Younes, E. Souissi, H. Achour, and A. Ferchichi, “Language resources for maghrebi arabic dialects’ nlp: a survey,” Language Resources and Evaluation, vol. 54, no. 4, pp. 1079–1142, 2020. [Online]. Available: https://link.springer.com/article/10.1007/s10579-020-09490-9 ↗
  9. W. Antoun, F. Baly, and H. Hajj, “AraBERT: Transformer-based model for Arabic language understanding,” in Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection. Marseille, France: European Language Resource Association, May 2020, pp. 9–15. [Online]. Available: https://www.aclweb.org/anthology/2020.osact-1.2 ↗ our card
  10. Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey et al., “Google’s neural machine translation system: Bridging the gap between human and machine translation,” arXiv preprint arXiv:1609.08144, 2016. [Online]. Available: https://arxiv.org/abs/1609.08144 ↗
  11. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019. [Online]. Available: https://arxiv.org/abs/1907.11692 ↗
  12. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A lite bert for self-supervised learning of language representations,” arXiv preprint arXiv:1909.11942, 2019. [Online]. Available: https://arxiv.org/abs/1909.11942 ↗
  13. I. Guellil, A. Adeel, F. Azouaou, M. Boubred, Y. Houichi, and A. A. Moumna, “Sexism detection: The first corpus in algerian dialect with a code-switching in arabic/ french and english,” 2021. [Online]. Available: https://arxiv.org/abs/2104.01443 ↗ our card
  14. L. Moudjari, K. Akli-Astouati, and F. Benamara, “An algerian corpus and an annotation platform for opinion and emotion analysis,” in Proceedings of The 12th Language Resources and Evaluation Conference, 2020, pp. 1202–1210. [Online]. Available: https://www.aclweb.org/anthology/2020.lrec-1.151/ ↗
  15. S. Touileb and J. Barnes, “The interplay between language similarity and script on a novel multi-layer algerian dialect corpus,” in Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 2021, pp. 3700–3712. [Online]. Available: https://aclanthology.org/2021.findings-acl.324/ ↗
  16. R. Plutchik, “Emotions: A general psychoevolutionary theory,” Approaches to emotion, vol. 1984, pp. 197–219, 1984. find
  17. D. Seddah, F. Essaidi, A. Fethi, M. Futeral, B. Muller, P. J. O. Suárez, B. Sagot, and A. Srivastava, “Building a user-generated content north-african arabizi treebank: Tackling hell,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 1139–1150. [Online]. Available: https://aclanthology.org/2020.acl-main.107/ ↗
  18. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush, “Transformers: State-of-the-art natural language processing,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Online, Oct. 2020, pp. 38–45. [Online]. Available: https://www.aclweb.org/anthology/2020.emnlp-demos.6 ↗ our card
  19. M. Abdul-Mageed, A. A. Elmadany, and E. M. B. Nagoudi, “ARBERT & MARBERT: deep bidirectional transformers for arabic,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, 2021, p. 7088–7105. [Online]. Available: https://aclanthology.org/2021.acl-long.551.pdf ↗ our card
  20. A. Abdaoui, C. Pradel, and G. Sigel, “Load what you need: Smaller versions of mutililingual BERT,” in Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing @EMNLP, 2020, pp. 119–123. [Online]. Available: https://www.aclweb.org/anthology/2020.sustainlp-1.16 ↗
  21. A. Riabi, B. Sagot, and D. Seddah, “Can character-based language models improve downstream task performance in low-resource and noisy language scenarios?” 2021. [Online]. Available: https://arxiv.org/abs/2110.13658 ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abdaoui-2021-dziribert-pre-trained-language-model/