Elamur Research⌕
☰
home / papers / 2019 / No Army, No Navy: BERT Semi-Supervised Learning of Arabic Dialects

No Army, No Navy: BERT Semi-Supervised Learning of Arabic Dialects

Paper · 2019 · Chiyu Zhang and Muhammad Abdul-Mageed

Summary

This paper addresses MADAR shared task 2: identifying which of 21 Arab countries a Twitter user's dialect belongs to. Labels are given per user, but the models classify individual tweets, so the authors predict tweet-level dialects and then combine them into one decision per user. They compare a unidirectional GRU baseline with fine-tuned BERT-Base Multilingual Cased. They also enlarge the training data by merging in the MADAR task 1 corpus. They then apply self-training: the best BERT model labels about 10M unlabeled in-house tweets, and its most confident predictions are added back to the training set. Confidence is measured either across all classes or within each class. A user's label is the majority class among tweets whose softmax score passes a threshold tuned on DEV. Tweet-level accuracy stays below 50%, but voting raises user-level scores sharply. The BERT-5% system ranked first, with 71.70 macro F1 and 77.40% accuracy on TEST. The result shows that self-training plus confidence-filtered voting can make a multilingual pretrained model competitive on fine-grained dialect distinctions with little labeled data.

Key points

  • Data. After crawling (some tweets were no longer available), the splits held 2,036 / 281 / 466 users for TRAIN-A / DEV / TEST, or 193,086 / 26,588 / 43,909 tweets. TRAIN-B adds the 95,000 task 1 sentences, giving 288,086 training items. DEV and TEST always come only from task 2.
  • Tweet-level results on DEV.
    • GRU baseline: 46.81 accuracy, 29.84 macro F1. The table gives 29.84; the prose says 28.84.
    • BERT-A: 48.50 / 33.93.
    • BERT-B: 48.76 / 34.87.
  • Self-training on unlabeled-10M (9,981,965 tweets). Top 5%, 10% and 25% of predictions were selected, either globally or per class.
    • BERT-5% gave the best F1 (35.93).
    • BERT-10% gave the best accuracy (49.39).
    • Per-class selection did not consistently help; 25% Class SEMI reached 35.81 F1.
  • User-level results.
    • The tuned softmax thresholds fell around 0.89–0.92.
    • On DEV, BERT-25% Class SEMI scored best (80.78 / 79.25).
    • On TEST, BERT-5% won the task (77.40 / 71.70).
    • The 34 unreachable users were assigned the majority class, Saudi Arabia.
  • Setup.
    • BERT: 110M parameters, pretrained on 104 languages, max length 50, batch 32, learning rate 2e-5, 10 epochs.
    • GRU: 500 units, 10k-word vocabulary, dropout 0.5, Adam at 1e-3.

Interesting ideas

  • I1. Turn many weak item-level guesses into a strong group-level label by voting only over items whose confidence clears a dev-tuned cutoff; even a noisy classifier (<50% per tweet) can then yield ~77% per user.
  • I2. Select self-training data by top-N% confidence and compare a global cut against per-class quotas, since a global cut can let frequent classes crowd out rare ones.
  • I3. Merge a related but differently-sourced labeled corpus into a small in-domain training set while keeping dev and test strictly in-domain, so any gain is measured honestly on the target.
  • I4. Sweep the confidence threshold over its whole range and plot performance, instead of fixing an arbitrary cutoff, to see how sensitive the aggregate decision is.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Entity-level attributes such as an entity's type, gender or stance could be decided by a confidence-filtered vote over mention-level predictions. Each vote stays traceable to the specific mentions that cast it.
  • I2 (learn): When building silver data agreed by witness parsers, sentences could be ranked by agreement and kept under per-label quotas, such as per dependency relation or per VerbNet class. Both regimes could be checked against negative controls.
  • I3 (tree): Converted treebanks from other UD dialects could be added to gparse training while evaluating LAS only on the target English treebank. This would show whether the extra data really transfers.
  • I4 (answer): The readers could sweep their answer-or-abstain confidence threshold on school-test dev sets. The curve would show where abstaining starts to pay off.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada104 works · 37,402 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Chiyu Zhang and Muhammad Abdul-Mageed. 2019. No Army, No Navy: BERT Semi-Supervised Learning of Arabic Dialects. Proceedings of the Fourth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-4637

@inproceedings{zhang2019no,
  title = {{No Army, No Navy: BERT Semi-Supervised Learning of Arabic Dialects}},
  author = {Chiyu Zhang and Muhammad Abdul-Mageed},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Workshop},
  year = {2019},
  doi = {10.18653/v1/w19-4637},
  url = {https://doi.org/10.18653/v1/w19-4637}
}

References

31 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mona Diab, Nizar Habash, Owen Rambow, Mohamed Altantawy, and Yassine Benajiba. 2010. Colaba: Arabic dialect annotation and processing. In Lrec workshop on semitic language processing, pages 66–74. find
  2. Muhammad Abdul-Mageed. 2017a. Modeling arabic subjectivity and sentiment in lexical space. Information Processing & Management. find
  3. Muhammad Abdul-Mageed. 2017b. Not all segments are created equal: Syntactically motivated sentiment analysis in lexical space. In Proceedings of the third Arabic natural language processing workshop, pages 147–156. find
  4. Mohamed Elaraby and Muhammad Abdul-Mageed. 2018. Deep models for Arabic dialect identification on benchmarked data. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pages 263–274, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  5. Muhammad Abdul-Mageed, Hassan Alhuzali, and Mohamed Elaraby. 2018. You tweet what you speak: A city-level dataset of arabic dialects. In LREC, pages 3653–3659.
  6. Heba Elfardy and Mona T Diab. 2012. Simplified guidelines for the creation of large scale dialectal arabic annotations. In LREC, pages 371–378.
  7. Muhammad Abdul-Mageed, Mona Diab, and Sandra Kübler. 2014. Samar: Subjectivity and sentiment analysis for arabic social media. Computer Speech & Language, 28(1):20–37. find
  8. Heba Elfardy and Mona T Diab. 2013. Sentence level dialect identification in arabic. In ACL (2), pages 456–461.
  9. Muhammad Abdul-Mageed and Mona T Diab. 2012. Awatif: A multi-genre corpus for modern standard arabic subjectivity and sentiment analysis. In LREC, pages 3907–3914.
  10. Tommi Jauhiainen, Marco Lui, Marcos Zampieri, Timothy Baldwin, and Krister Lindén. 2018. Automatic language identification in texts: A survey. arXiv preprint arXiv:1804.08186. ↗
  11. Rania Al-Sabbagh and Roxana Girju. 2012. Yadac: Yet another dialectal arabic corpus. In LREC, pages 2882–2889.
  12. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  13. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018. The madar arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018). find
  14. Shervin Malmasi and Marcos Zampieri. 2016. Arabic dialect identification in speech transcripts. VarDial 3, page 106.
  15. Hamdy Mubarak and Kareem Darwish. 2014. Using twitter to collect a multi-dialectal corpus of arabic. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 1–7.
  16. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR Shared Task on Arabic Fine-Grained Dialect Identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop (WANLP19), Florence, Italy.
  17. Miha Pavlinek and Vili Podgorelec. 2017. Text classification method based on self-training and lda topic models. Expert Systems with Applications, 80:83– 93. find
  18. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. ↗
  19. Chatrine Qwaider, Motaz Saad, Stergios Chatzikyriakidis, and Simon Dobnik. 2018. Shami: A corpus of levantine arabic dialects. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018).
  20. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555. ↗
  21. Fatiha Sadat, Farnazeh Kazemi, and Atefeh Farzindar. 2014. Automatic identification of arabic language varieties and dialects in social media. Proceedings of SocialNLP, page 22.
  22. Ryan Cotterell and Chris Callison-Burch. 2014. A multi-dialect, multi-genre corpus of informal written arabic. In LREC, pages 241–245.
  23. Wael Salloum, Heba Elfardy, Linda Alamir-Salloum, Nizar Habash, and Mona Diab. 2014. Sentence level dialect identification for machine translation system selection. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 772–778. find
  24. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  25. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958.
  26. Isaac Triguero, Salvador Garcı́a, and Francisco Herrera. 2015. Self-labeled techniques for semi-supervised learning: taxonomy, software and empirical study. Knowledge and Information Systems, 42(2):245–284. find
  27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
  28. Wajdi Zaghouani and Anis Charfi. 2018. Arap-tweet: A large multi-dialect twitter corpus for gender, age and language variety identification. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018). find
  29. Omar F Zaidan and Chris Callison-Burch. 2011. The arabic online commentary dataset: an annotated dataset of informal arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 37–41. Association for Computational Linguistics.
  30. Omar F Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202.
  31. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Ahmed Ali, Suwon Shon, James Glass, Yves Scherrer, Tanja Samardžić, Nikola Ljubešić, Jörg Tiedemann, et al. 2018. Language identification and morphosyntactic tagging: The second vardial evaluation campaign. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2019-no-army-no-navy-bert/