Elamur Research⌕
☰
home / papers / 2020 / Semi-supervised Fine-grained Approach for Arabic dialect detection task

Semi-supervised Fine-grained Approach for Arabic dialect detection task

Paper · 2020 · Nitin Nikamanth Appiah Balaji and Bharathi B

Summary

This paper describes the TRY NLP team's entry to the NADI 2020 shared task on Arabic dialect identification from tweets. The task has two fine-grained levels: 21 country labels and 100 province labels. The training data is small and heavily imbalanced, so the authors first compared several text representations, each topped with an added classification layer: TF-IDF, AraVec, multilingual BERT and an Arabic FastText model. FastText worked best with only the labelled data. Its subword pieces cope well with the many unfamiliar tokens in tweets, and an Arabic-specific pretrained version exists, while BERT is only available as a general multilingual model. The authors then added self-training. They downloaded the roughly 10 million unlabelled tweets provided as IDs, labelled them with the FastText model, kept only the most confident predictions, and retrained, repeating this several times. On the dev set this raised country-level macro F1 from 0.1890 to 0.2232. The work shows that cheap pseudo-labelling of in-domain raw text can help a low-resource, many-class task more than switching to a larger pretrained model.

Key points

  • Data: 21,000 labelled training tweets, 4,957 dev and 5,000 test tweets. There were also 10 million unlabelled tweet IDs, and about 92% of those tweets could still be fetched through the Twitter API.
  • Preprocessing kept only Arabic text, removing links, hashtags, numbers, English and special characters.
  • Dev macro F1 at country level was TF-IDF 0.1522, AraVec 0.1159, multilingual BERT 0.1175, FastText 0.1890 and semi-supervised FastText 0.2232. At province level the same models scored 0.0442, 0.0213, 0.0275, 0.0437 and 0.0483.
  • The FastText setup used Common Crawl and Wikipedia Arabic vectors of dimension 300 (CBOW, character n-grams of length 5, window 5, 10 negatives), loaded through the Magnitude library.
  • On the test set, the submitted semi-supervised FastText model reached macro F1 0.2004 for country level and 0.0403 for province level. Confidence cut-offs of the top 1%, 5% and 10% were tried for selecting pseudo-labels.

Interesting ideas

  • I1. Self-training adds only a thin top slice of the model's most confident predictions in each round (1/5/10%), rather than all pseudo-labels, which limits how much error leaks back into training.
  • I2. With pseudo-labelling, country-level macro F1 rose while plain accuracy dipped slightly (0.3752 → 0.3674), so the gains landed on smaller classes that accuracy hides.
  • I3. A plain TF-IDF model fitted on the task's own corpus beat AraVec and multilingual BERT, and roughly matched FastText at province level, which makes it a strong cheap reference point.
  • I4. Character n-gram embeddings, plus a lookup that gives repeated unseen words a consistent vector, handle noisy social-media vocabulary that word-level models miss.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver parses agreed by witness parsers could be added in small, high-agreement slices per round, with retraining between rounds, instead of in one large batch.
  • I2 (learn): Evaluations should report per-class or macro scores next to overall accuracy, so that improvements or regressions on rare relations or labels stay visible.
  • I3 (learn): A simple in-domain baseline can act as a negative control, so a costly new component counts as a win only if it clearly beats this baseline.
  • I4 (tokens): Subword-based fallback features could stabilise tagging and lemma guesses for out-of-vocabulary or misspelled English words.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Nitin Nikamanth Appiah Balaji and Bharathi B. 2020. Semi-supervised Fine-grained Approach for Arabic dialect detection task. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{balaji2020semi,
  title = {{Semi-supervised Fine-grained Approach for Arabic dialect detection task}},
  author = {Nitin Nikamanth Appiah Balaji and Bharathi B},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.25/}
}

References

13 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. NADI 2020: The First Nuanced Arabic Dialect Identification Shared Task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop (WANLP 2020), Barcelona, Spain. ↗
  2. Kathrein Abu Kwaik and Motaz Saad. 2019. ArbDialectID at MADAR shared task 1: Language modelling and ensemble learning for fine grained Arabic dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 254–258, Florence, Italy, August. Association for Computational Linguistics.
  3. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European Language Resources Association (ELRA). find
  4. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR shared task on Arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199–207, Florence, Italy. Association for Computational Linguistics.
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  6. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018).
  7. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. ↗
  8. Pruthwik Mishra and Vandan Mujadia. 2019. Arabic dialect identification for travel and twitter text. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 234–238.
  9. Ajay Patel, Alexander Sands, Chris Callison-Burch, and Marianna Apidianaki. 2018. Magnitude: A fast, efficient universal vector embedding utility package. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 120–126. find
  10. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained arabic dialect identification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1332–1344.
  11. Younes Samih, Hamdy Mubarak, Ahmed Abdelali, Mohammed Attia, Mohamed Eldesouki, and Kareem Darwish. 2019. QC-GO submission for MADAR shared task: Arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 290–294, Florence, Italy, August. Association for Computational Linguistics.
  12. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
  13. Chiyu Zhang and Muhammad Abdul-Mageed. 2019. No army, no navy: Bert semi-supervised learning of arabic dialects. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 279–284.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/balaji-2020-semi-supervised-fine-grained-approach/