Elamur Research⌕
☰
home / papers / 2021 / Self-Training Pre-Trained Language Models for Zero- and Few-Shot Multi-Dialectal Arabic Sequence Labeling

Self-Training Pre-Trained Language Models for Zero- and Few-Shot Multi-Dialectal Arabic Sequence Labeling

Paper · 2021 · Muhammad Khalifa, Muhammad Abdul-Mageed and Khaled Shaalan

Summary

Labeled data for Arabic NLP is mostly in Modern Standard Arabic (MSA), while everyday online writing is in dialects. The authors ask how well a tagger trained only on MSA works on dialectal Arabic (DA), and how unlabeled dialect text can close the gap. They fine-tune XLM-R on MSA for NER and POS tagging and test it on Egyptian, Gulf and Levantine data. Without dialect data, NER drops by about 20 F1 points. They then self-train the model: it labels unlabeled DA sentences, keeps the ones it is confident about, adds them to the training pool, and repeats. In the zero-shot setting this recovers up to about 10 F1 points for NER and about 2 accuracy points for POS tagging. In few-shot settings, self-training still helps, even when all available gold dialect data is used. An ablation links the gains to the added dialect examples, and an error analysis shows that most fixes remove false positives. The method needs no language-specific design, so it can be reused for other language varieties. Code and models are public.

Key points

  • Backbone: XLM-R, chosen over mBERT partly because its Common Crawl pretraining data likely contains more dialect text than Arabic Wikipedia. Baselines are NERA, WC-BiLSTM, WC-CNN and mBERT. The authors report state-of-the-art results for both NER and POS.
  • NER data: ANERCorp (~150K tokens), ACE 2003 BNews (~15K), ACE 2003 Newswire (~27K) and the Twitter set of Darwish 2013 (~81K). All use the LOC/ORG/PER types.
  • POS data: the multi-dialectal Twitter set of Darwish et al. 2018. It has 350 tweets each for MSA, EGY, GLF and LEV, tagged with 21 tags, some specific to social media.
  • Two selection rules were compared:
    • a threshold τ on each sentence's lowest token softmax confidence;
    • a fixed-size rule that takes the top S sentences by that same minimum.
  • Error analysis: 59.7% of the improvements were corrected false positives. Typical cases are dialect words that look like MSA names or words that often appear near entities, such as interjections and idioms.

Interesting ideas

  • I1. A sentence is scored by its least confident token rather than an average, so one doubtful label is enough to keep it out of the pseudo-labeled pool.
  • I2. The paper tests both an absolute confidence threshold and a fixed top-S quota per round, since a threshold can admit too much or too little as the model changes.
  • I3. Self-training on unlabeled target-variety text still adds accuracy when all gold target data is already in use. Unlabeled in-domain text is therefore worth using even when some gold data exists.
  • I4. An ablation ties the gain to the added target-domain examples rather than to extra training rounds. This works as a control that the improvement has the claimed cause.
  • I5. The error analysis counts which error type self-training fixes, here mostly false positives from lookalikes of standard-variety entities. This shows where the method helps and where it fails.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When gathering silver trees, accept a sentence only if its weakest arc/label confidence (or weakest witness agreement) clears a bar, instead of averaging over the tree.
  • I2 (learn): Run both rules in silver-data self-training loops and compare, to control how fast unverified data enters training.
  • I3 (tree): Adapt gparse and the perceptron parser to informal or web English by self-training on raw in-domain text alongside the existing treebanks.
  • I4 (learn): Add a negative control where the self-training loop runs with in-domain pseudo-labels removed or replaced, to confirm the silver data itself causes the gains.
  • I5 (tokens): Break tagger and NER errors on informal English into false positives and false negatives, to see whether slang and interjections are being mistaken for names.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada104 works · 37,402 citations🎓Cairo UniversityGiza · Egypt43 works · 1,352 citations🎓British University in DubaiDubai · United Arab Emirates11 works · 589 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksSemitic languages · 3,066 worksPart-of-speech tagging · 126 works

Cite

Cite as Muhammad Khalifa, Muhammad Abdul-Mageed and Khaled Shaalan. 2021. Self-Training Pre-Trained Language Models for Zero- and Few-Shot Multi-Dialectal Arabic Sequence Labeling. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.eacl-main.65

@inproceedings{khalifa2021self,
  title = {{Self-Training Pre-Trained Language Models for Zero- and Few-Shot Multi-Dialectal Arabic Sequence Labeling}},
  author = {Muhammad Khalifa, Muhammad Abdul-Mageed and Khaled Shaalan},
  booktitle = {Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume},
  year = {2021},
  doi = {10.18653/v1/2021.eacl-main.65},
  url = {https://doi.org/10.18653/v1/2021.eacl-main.65}
}

References

52 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sherief Abdallah, Khaled Shaalan, and Muhammad Shoaib. 2012. Integrating rule-based system with classification for Arabic named entity recognition. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 311– 322. Springer. find
  2. Muhammad Abdul-Mageed, Hassan Alhuzali, and Mohamed Elaraby. 2018. You tweet what you speak: A city-level dataset of arabic dialects. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
  3. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020a. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. arXiv preprint arXiv:2101.01785. ↗
  4. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020b. Nadi 2020: The first nuanced arabic dialect identification shared task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop (WANLP 2020), pages 97–110, Barcelona, Spain. ↗
  5. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, and Lyle Ungar. 2020c. Micro-dialect identification in diaglossic and code-switched environments. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5855–5876. find
  6. Muhammad Abdul-Mageed, Chiyu Zhang, Azadeh Hashemi, and El Moatez Billah Nagoudi. 2020d. AraNet: A Deep Learning Toolkit for Arabic Social Media. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 16–23. ↗
  7. Mahmoud Al-Ayyoub, Aya Nuseir, Kholoud Alsmearat, Yaser Jararweh, and Brij Gupta. 2018. Deep learning for arabic nlp: A survey. Journal of computational science, 26:522–531. ↗
  8. Mohammad Al-Smadi, Saad Al-Zboon, Yaser Jararweh, and Patrick Juola. 2020. Transfer learning for Arabic named entity recognition with deep neural networks. IEEE Access, 8:37736–37745. ↗
  9. Randah Alharbi, Walid Magdy, Kareem Darwish, Ahmed AbdelAli, and Hamdy Mubarak. 2018. Part-of-speech tagging for Arabic gulf dialect using bilstm. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  10. Ali Alshehri, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2020. Understanding and detecting dangerous speech in social media. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for Arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  11. Yassine Benajiba, Paolo Rosso, and José Miguel Bened\’ıruiz. 2007. Anersys: An Arabic named entity recognition system based on maximum entropy. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 143–153. find
  12. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137–1155.
  13. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR shared task on Arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199–207.
  14. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. ↗
  15. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. ↗
  16. Brenda Danet and Susan C Herring. 2007. The multilingual Internet: Language, culture, and communication online. Oxford University Press. find
  17. Kareem Darwish. 2013. Named entity recognition using cross-lingual resources: Arabic as an example. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1558–1567.
  18. Kareem Darwish, Hamdy Mubarak, Ahmed Abdelali, Mohamed Eldesouki, Younes Samih, Randah Alharbi, Mohammed Attia, Walid Magdy, and Laura Kallmeyer. 2018. Multi-dialect arabic pos tagging: a crf approach. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
  19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  20. Xin Luna Dong and Gerard de Melo. 2019. A robust self-learning framework for cross-lingual text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6307–6311. find
  21. Ismail El Bazi and Nabil Laachfoubi. 2019. Arabic named entity recognition using deep learning approach. International Journal of Electrical & Computer Engineering (2088-8708), 9(3). ↗
  22. Mohamed Elaraby and Muhammad Abdul-Mageed. 2018. Deep models for Arabic dialect identification on benchmarked data. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pages 263–274.
  23. Mourad Gridach. 2016. Character-aware neural networks for Arabic named entity recognition for social media. In Proceedings of the 6th workshop on South and Southeast Asian natural language processing (WSSANLP2016), pages 23–32.
  24. Mohammad Sadegh Hajmohammadi, Roliana Ibrahim, Ali Selamat, and Hamido Fujita. 2015. Combination of active learning and self-training for cross-lingual sentiment classification with density analysis of unlabelled samples. Information sciences, 317:67–77. ↗
  25. Susan C Herring, Jannis Androutsopoulos, et al. 2015. Computer-mediated discourse 2.0. The handbook of discourse analysis, 2:127–151. ↗
  26. Muhammad Khalifa and Khaled Shaalan. 2019. Character convolutions for arabic named entity recognition with long short-term memory networks. Computer Speech & Language, 58:335–346. ↗
  27. Joo-Kyung Kim, Young-Bum Kim, Ruhi Sarikaya, and Eric Fosler-Lussier. 2017. Cross-lingual transfer learning for pos tagging without cross-lingual resources. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 2832–2838. ↗
  28. Zornitsa Kozareva, Boyan Bonev, and Andres Montoyo. 2005. Self-training and co-training applied to spanish named entity recognition. In Mexican International conference on Artificial Intelligence, pages 770–779. Springer. find
  29. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360. ↗
  30. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. ↗
  31. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  32. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. arXiv preprint arXiv:1603.01354. ↗
  33. Mohamed Maamouri, Ann Bies, Tim Buckwalter, and Wigdan Mekki. 2004. The penn Arabic treebank: Building a large-scale annotated Arabic corpus. In NEMLAR conference on Arabic language resources and tools, volume 27, pages 466–467. Cairo. find
  34. Mohamed Maamouri, Ann Bies, Seth Kulick, Michael Ciul, Nizar Habash, and Ramy Eskander. 2014. Developing an egyptian Arabic treebank: Impact of dialectal morphology on annotation and tool development. In LREC, pages 2348–2354. find
  35. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. ↗
  36. Alexis Mitchell, Stephanie Strassel, Mark Przybocki, J Davis, George Doddington, Ralph Grishman, Adam Meyers, Ada Brunstain, Lisa Ferro, and Beth Sundheim. 2003. Tides extraction (ace) 2003 multilingual training data. LDC2004T09, Philadelphia, Penn.: Linguistic Data Consortium. ↗
  37. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL Tools: An Open Source Python Toolkit for Arabic Natural Language Processing. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 7022–7032.
  38. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958.
  39. Kenji Sagae. 2010. Self-training without reranking for parser domain adaptation and its impact on semantic role labeling. In Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing, pages 37–44. find
  40. Khaled Shaalan. 2014. A survey of Arabic named entity recognition and classification. Computational Linguistics, 40(2):469–510. ↗
  41. Khaled Shaalan and Mai Oudah. 2014. A hybrid approach to Arabic named entity recognition. Journal of Information Science, 40(1):67–87. ↗
  42. Abdulhadi Shoufan and Sumaya Alameri. 2015. Natural language processing for dialectical arabic: A survey. In Proceedings of the second workshop on Arabic natural language processing, pages 36–48.
  43. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. Mass: Masked sequence to sequence pre-training for language generation. arXiv preprint arXiv:1905.02450. ↗
  44. Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, and Amelia Archer. 2019. Small and practical bert models for sequence labeling. arXiv preprint arXiv:1909.00100. ↗
  45. Vincent Van Asch and Walter Daelemans. 2016. Predicting the effectiveness of self-training: Application to sentiment classification. arXiv preprint arXiv:1601.03288. ↗
  46. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
  47. Wen Wang, Zhongqiang Huang, and Mary Harper. 2007. Semi-supervised learning for part-of-speech tagging of mandarin transcribed speech. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP’07, volume 4, pages IV–137. IEEE. find
  48. Jiateng Xie, Zhilin Yang, Graham Neubig, Noah A Smith, and Jaime Carbonell. 2018. Neural cross-lingual named entity recognition with minimal resources. arXiv preprint arXiv:1808.09861. ↗
  49. Wei Xu and Alex Rudnicky. 2000. Can artificial neural networks learn language models? In Sixth international conference on spoken language processing. find
  50. An Yang, Quan Wang, Jing Liu, Kai Liu, Yajuan Lyu, Hua Wu, Qiaoqiao She, and Sujian Li. 2019. Enhancing pre-trained language representations with rich knowledge for machine reading comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2346–2357. ↗
  51. Omar Zaidan and Chris Callison-Burch. 2011. The arabic online commentary dataset: an annotated dataset of informal arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 37–41.
  52. Omar F Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/khalifa-2021-self-training-pre-trained-language/