Elamur Research⌕
☰
home / papers / 2020 / Arabic Dialect Identification Using BERT-Based Domain Adaptation

Arabic Dialect Identification Using BERT-Based Domain Adaptation

Paper · 2020 · Ahmad Beltagy, Abdelrahman Abouelenin and Omar ElSherief

Summary

The paper describes a system for the NADI 2020 Shared Task 1, which asks for the country-level dialect of an Arabic tweet across 21 classes. The labelled data is small and very unbalanced, and some tweets are noisy. The authors build on AraBERT and improve it in three steps. First, they clean the tweets by removing URLs, mentions, emojis, Latin characters, tashkil and letter elongations, and they split clitics with Farasa. Second, they upsample the rare classes. Third, and most important, they continue masked-language-model training of AraBERT on about 2M of the unlabelled tweets NADI supplied, which adapts the model to tweet text. A single linear head on the 768-dimensional output then predicts the class. Each step raises dev macro-F1, and domain adaptation gives the largest gain. The submitted model placed 4th, with 23.09% macro-F1 on test. A hierarchical variant that combines AraBERT with Naive Bayes did worse. The work shows that cheap in-domain pretraining on unlabelled text can matter more than clever architectures when labels are scarce.

Key points

  • NADI corpus: 21 country labels, with 21,000 train, 4,957 dev and 5,000 test tweets, plus 10M unlabelled tweet IDs. The classes are skewed: Egypt has 4,473 training examples and Sudan only 210.
  • Dev macro-F1 ladder:
    • plain AraBERT: 18.6%
    • with cleaning: 20.54%
    • with upsampling of minority classes to 750: 21.33%
    • with MLM domain adaptation: 24.43%
  • The domain adaptation used 2M tweets, because more data gave no gain. Training ran for 1 epoch (about 20 hours) with lr 2e-5 and 15% masking. Inputs were truncated or padded to 64 tokens.
  • The hybrid model routes six rare classes as one "Not majority" bucket to Naive Bayes. It reached 22.3% dev F1, below plain AraBERT.
  • The final test score was 23.09%, which ranked 4th. The conclusion states 22.03%, which looks like a typo.

Interesting ideas

  • I1. Domain-adaptive pretraining. Before training on labels, the authors continue masked-LM training of the pretrained encoder on unlabelled text from the target genre. This was the biggest single gain and is cheap wherever raw in-domain text is plentiful.
  • I2. Incremental ablation ladder. Each component is added one at a time and its macro-F1 gain is reported separately. This makes it clear which step earns its cost.
  • I3. Finding saturation points. Both the upsampling target (750 per class) and the amount of unlabelled data (2M) were set where further increases stopped helping. This avoids wasted compute.
  • I4. Separating uncorrectable label noise. The error analysis finds tweets whose label depends on the user, not the text, such as app-generated Quranic verses, prayers and retweets of other dialects. No model can fix these, so they should be identified rather than fitted.
  • I5. Clitic splitting before encoding. Splitting prefixes and suffixes with Farasa merges surface variants of the same core word, giving the model more consistent units.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): The gparse transformer encoder could get a short MLM pass on fables, tales and school-test texts before parser training, to fit it to the genres Mova actually reads.
  • I2 (learn): Evaluation reports for new training recipes could use the same cumulative table, so each added trick is credited with a measured delta.
  • I3 (learn): When growing silver data agreed by witness parsers, Mova could track the score curve and stop adding data at the plateau rather than guessing a size.
  • I4 (learn): Silver data could be filtered for quoted or boilerplate text whose label is not grounded in the sentence itself, and such items could serve as negative controls.
  • I5 (tokens): The English tokenizer can likewise split clitics such as "'s" and "n't" consistently, following UD conventions, so morphology and lemmas see stable stems.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Alexandria UniversityAlexandria · Egypt39 works · 287 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Ahmad Beltagy, Abdelrahman Abouelenin and Omar ElSherief. 2020. Arabic Dialect Identification Using BERT-Based Domain Adaptation. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{beltagy2020arabic,
  title = {{Arabic Dialect Identification Using BERT-Based Domain Adaptation}},
  author = {Ahmad Beltagy, Abdelrahman Abouelenin and Omar ElSherief},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.26/}
}

References

15 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Kareem Darwish, and Hamdy Durrani, Nadir andz Mubarak. 2016. Farasa: A fast and furious segmenter for Arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16, San Diego, California, June. Association for Computational Linguistics.
  2. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. NADI 2020: The First Nuanced Arabic Dialect Identification Shared Task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop (WANLP 2020), Barcelona, Spain. ↗
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. ↗
  4. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European Language Resources Association (ELRA). find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.
  6. Youssef Fares, Zeyad El-Zanaty, Kareem Abdel-Salam, Muhammed Ezzeldin, Aliaa Mohamed, Karim El-Awaad, and Marwan Torki. 2019. Arabic dialect identification with deep learning and hybrid frequency based features. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 224–228, Florence, Italy, August. Association for Computational Linguistics.
  7. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In ACL. find
  8. Mai Ibrahim, Marwan Torki, and Nagwa El-Makky. 2018. Imbalanced toxic comments classification using data augmentation and deep learning. In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 875–878, Dec. find
  9. Mai Ibrahim, Marwan Torki, and Nagwa El-Makky. 2020. Alexu-backtranslation-tl at semeval-2020 task [12]: Improving offensive language detection using data augmentation and transfer learning. In Proceedings of the International Workshop on Semantic Evaluation (SemEval). ↗
  10. Kamran Kowsari, Donald E. Brown, Mojtaba Heidarysafa, Kiana Jafari Meimandi, Matthew S. Gerber, and Laura E. Barnes. 2017. Hdltex: Hierarchical deep learning for text classification. 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 364–371.
  11. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  12. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained arabic dialect identification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1332–1344.
  13. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. ArXiv, abs/1706.03762. ↗
  14. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. ↗
  15. Taha Zerrouki. 2010. pyarabic, an arabic language library for python. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/beltagy-2020-arabic-dialect-identification-bert-based/