Elamur Research⌕
☰
home / papers / 2022 / NLP DI at NADI Shared Task Subtask-1: Sub-word Level Convolutional Neural Models and Pre-trained Binary Classifiers for Dialect Identification

NLP DI at NADI Shared Task Subtask-1: Sub-word Level Convolutional Neural Models and Pre-trained Binary Classifiers for Dialect Identification

Paper · 2022 · Vani Kanjirangat, Tanja Samardzic, Ljiljana Dolamic et al.

Summary

The paper describes the NLP_DI team's entry to NADI 2022 Subtask 1, which asks systems to identify the country-level Arabic dialect of a short text. The organisers supplied training and development data covering 18 countries and two test sets. TEST-A contains all 18 dialects. TEST-B leaves out an unknown number of them. The team compared two families of models. The first is a CNN, adapted from Kim et al.'s neural language model encoder, trained from scratch on SentencePiece subwords (Unigram and BPE) with tuned vocabulary sizes. The second is fine-tuned Arabic BERT models (AraBERT and Multi-dialect-Arabic-BERT). For TEST-B they added two techniques. One is label smoothing. The other is a set of balanced one-vs-rest binary classifiers, whose average confidence on the test data was compared against development sets with dialects deliberately removed. This let them guess which dialects were absent and retrain without them. Multi-dialect-Arabic-BERT performed best overall. Detecting and dropping the absent classes gave the strongest TEST-B score. The work shows that estimating which labels are absent can help classification when the test label set is unknown.

Key points

  • Data: 20,398 training and 4,871 development instances over 18 country dialects. TEST-A has 4,871 instances with all dialects, and TEST-B has 1,474 instances with some dialects missing.
  • Subword vocabulary size was searched from character level up to 0.4·|V| (following Mielke et al.). The best sizes were 20,045 for Unigram and 7,045 for BPE.
  • Development macro F-scores were Unigram_CNN 17.06%, BPE_CNN 17.17%, AraBERT 21.38% and Multi-dialect-Arabic-BERT 26.59%. On TEST-A the best was Multi-dialect-Arabic-BERT at 26.12% F (42.07% accuracy), according to the results table.
  • On TEST-B the plain Multi-dialect-Arabic-BERT scored 13.47% F. Adding label smoothing (α = 0.1) raised this to 13.75%. Retraining without the five dialects judged missing (Algeria, Tunisia, Morocco, Iraq, Sudan) raised it to 16.44%.
  • The team's overall official F-score was 21.28%. The CNN models stayed clearly behind the pretrained ones on both test sets.

Interesting ideas

  • I1. Treat subword vocabulary size as a hyperparameter, sweeping it from character level up to a fraction of the word vocabulary and picking the size by downstream dev performance rather than by habit.
  • I2. Calibrate a test-time decision by first simulating that condition on dev data: remove known classes several times and record how per-class confidence behaves when a class is truly absent.
  • I3. Train one balanced one-vs-rest classifier per label and use its average positive probability over a whole batch as a corpus-level signal that the label occurs at all.
  • I4. Once a label is judged absent, retrain or restrict the model to the reduced label space rather than only filtering its predictions afterwards.
  • I5. Reuse a language-model encoder as a classifier by replacing its decoder with a dense and softmax head.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Size subword units for rare-word or noisy-text handling by measuring tagging or lemmatisation accuracy on dev, instead of fixing them a priori.
  • I2 (learn): This is a ready-made negative-control protocol, where we deliberately withhold phenomena from silver dev sets to learn what a classifier's "this is not present" signal looks like.
  • I3 (tree): Before parsing a new domain or a UD dialect conversion, we could estimate which relations or constructions are absent and restrict or reweight the parser's label set.
  • I4 (frames): If a text's genre clearly lacks certain speech acts or frame types (e.g., irony in maths problems), pruning them up front could cut spurious pragmatic readings.
  • I5 (tokens): A character or subword CNN encoder could be shared between the morphology tagger and auxiliary classifiers (register, dialect of English) without separate feature engineering.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of ZurichZurich · Switzerland78 works · 12,260 citations🔬Dalle Molle Institute for Artificial Intelligence ResearchLugano · Switzerland5 works · 18 citations🎓University of Applied Sciences and Arts of Southern SwitzerlandManno · Switzerland4 works · 17 citations🏛️Bundesamt für Rüstung armasuisseBern · Switzerland3 works · 16 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Vani Kanjirangat, Tanja Samardzic, Ljiljana Dolamic and Fabio Rinaldi. 2022. NLP DI at NADI Shared Task Subtask-1: Sub-word Level Convolutional Neural Models and Pre-trained Binary Classifiers for Dialect Identification. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.51

@inproceedings{kanjirangat2022nlp,
  title = {{NLP DI at NADI Shared Task Subtask-1: Sub-word Level Convolutional Neural Models and Pre-trained Binary Classifiers for Dialect Identification}},
  author = {Vani Kanjirangat, Tanja Samardzic, Ljiljana Dolamic and Fabio Rinaldi},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.51},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.51}
}

References

28 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. Nadi 2020: The first nuanced arabic dialect identification shared task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 97–110. ↗
  2. Muhammad Abdul-Mageed, Chiyu Zhang, Abdel-Rahim Elmadany, Houda Bouamor, and Nizar Habash. 2021. Nadi 2021: The second nuanced arabic dialect identification shared task. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 244–259. ↗
  3. Muhammad Abdul-Mageed, Chiyu Zhang, Abdel-Rahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. NADI 2022: The Third Nuanced Arabic Dialect Identification Shared Task. In Proceedings of the Seven Arabic Natural Language Processing Workshop (WANLP 2022).
  4. Ahmed Ali, Najim Dehak, Patrick Cardinal, Sameer Khurana, Sree Harsha Yella, James Glass, Peter Bell, and Steve Renals. 2016. Automatic dialect detection in arabic broadcast speech.
  5. Mohamed Ali. 2018. Character level convolutional neural network for arabic dialect identification. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pages 122–127.
  6. Badr AlKhamissi, Mohamed Gabr, Muhammad El-Nokrashy, and Khaled Essam. 2021. Adapting marbert for improved arabic dialect identification: Submission to the nadi 2021 shared task. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 260–264. ↗
  7. Nouf AlShenaifi and Aqil Azmi. 2020. Faheem at nadi shared task: Identifying the dialect of arabic tweet. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 282–287. ↗
  8. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. In Thirtieth AAAI conference on artificial intelligence.
  9. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  10. Shervin Malmasi, Marcos Zampieri, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, and Jörg Tiedemann. 2016. Discriminating between similar languages and arabic dialect identification: A report on the third dsl shared task. In Proceedings of the third workshop on NLP for similar languages, varieties and dialects (VarDial3), pages 1–14.
  11. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The madar shared task on arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199– 207.
  12. Andrei Butnaru and Radu Tudor Ionescu. 2018. Unibuckernel reloaded: First place in arabic dialect identification for the second year in a row. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pages 77–87. ↗
  13. Sabrina J Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. 2019. What kind of language is hard to language-model? arXiv preprint arXiv:1906.04726. ↗
  14. Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. 2019. When does label smoothing help? Advances in Neural Information Processing Systems, 32:4694– 4703. find
  15. Çağrı Çöltekin, Taraka Rama, and Verena Blaschke. 2018. Tübingen-oslo team at the vardial 2018 evaluation campaign: An analysis of n-gram features in language variety identification. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pages 55–65. find
  16. Bashar Talafha, Mohammad Ali, Muhy Eddin Za’ter, Haitham Seelawi, Ibraheem Tuffaha, Mostafa Samir, Wael Farhan, and Hussein Al-Natsheh. 2020. Multi-dialect arabic bert for country-level dialect identification. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 111–118. ↗
  17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. ↗
  18. Samia Touileb. 2020. Ltg-st at nadi shared task 1: Arabic dialect identification using a stacking classifier. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 313–319.
  19. Abdellah El Mekki, Ahmed Alami, Hamza Alami, Ahmed Khoumsi, and Ismail Berrada. 2020. Weighted combination of bert and n-gram features for nuanced arabic dialect identification. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 268–274.
  20. Abdellah El Mekki, Abdelkader El Mahdaouy, Kabil Essefar, Nabil El Mamoun, Ismail Berrada, and Ahmed Khoumsi. 2021. Bert-based multi-task model for country and province level msa and dialectal arabic identification. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 271– 275.
  21. Mohamed Elaraby and Muhammad Abdul-Mageed. 2018. Deep models for arabic dialect identification on benchmarked data. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pages 263–274.
  22. Salima Harrat, Karima Meftouh, Karima Abidi, and Kamel Smaïli. 2019. Automatic identification methods on a corpus of twenty five fine-grained arabic dialects. In International Conference on Arabic Language Processing, pages 79–92. Springer. find
  23. Mutaz Younes, Nour Al-Khdour, and AL-Smadi Mohammad. 2020. Team alexa at nadi shared task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 237–242.
  24. Omar Zaidan and Chris Callison-Burch. 2011. The arabic online commentary dataset: an annotated dataset of informal arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 37–41.
  25. Omar F Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗
  26. Marcos Zampieri, Shervin Malmasi, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, Jörg Tiedemann, Yves Scherrer, and Noëmi Aepli. 2017. Findings of the vardial evaluation campaign 2017. In Proceedings of the fourth workshop on NLP for similar languages, varieties and dialects.
  27. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Ahmed Ali, Suwon Shon, James Glass, Yves Scherrer, Tanja Samardžić, Nikola Ljubešić, Jörg Tiedemann, et al. 2018. Language identification and morphosyntactic tagging. the second vardial evaluation campaign.
  28. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. Osian: Open source international arabic news corpus-preparation and integration into the clarin-infrastructure. In Proceedings of the fourth arabic natural language processing workshop, pages 175–182. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kanjirangat-2022-nlp-di-nadi-shared-task/