Elamur Research⌕
☰
home / papers / 2020 / Weighted combination of BERT and N-GRAM features for Nuanced Arabic Dialect Identification

Weighted combination of BERT and N-GRAM features for Nuanced Arabic Dialect Identification

Paper · 2020 · Abdellah El Mekki, Ahmed Alami, Hamza Alami et al.

Summary

The paper tackles fine-grained Arabic dialect identification on tweets for the NADI shared task. It has two subtasks: predicting one of 21 countries and predicting one of 100 provinces. The authors combine two very different classifiers. The first, M_NGRAM, is a linear model: TF-IDF features from word unigrams and character 3–5-grams, trained with SGD under the modified Huber loss. Its training data is enlarged with tweets pulled from a 10M unlabeled pool using country-specific keywords. The second, M_BERT, is a fine-tuned AraBERT model. Its input pipeline swaps emojis for [MASK] tokens and appends their Arabic glosses as a second segment. For countries, the two models' probabilities are merged by weighted soft voting, with the n-gram model given the larger share. For provinces, a hierarchical setup first predicts the country and then uses a per-country AraBERT classifier. The system placed 2nd for countries (test macro F1 25.99%) and 1st for provinces (6.39%). The authors argue that statistical surface features and contextual neural features complement each other, especially in noisy, low-resource, closely related language varieties.

Key points

  • NADI data: 21K training tweets, 4,957 dev, 5,000 test, plus 10M unlabeled tweets. The label distribution is heavily skewed: Egypt is about 21% of training data, while Djibouti and Bahrain are around 1% each.
  • On the dev set for countries, macro F1 was 25.02% for M_NGRAM, 22.42% for M_BERT, and 27.24% for the ensemble (accuracy 40.95%). The best voting weights were 0.7 for n-grams and 0.3 for BERT.
  • Keyword-harvested tweets (flag emojis, country and city names, local jargon) made the dataset about ten times larger. They were given a sample weight of 0.25, against 1 for the original labeled data.
  • M_BERT pipeline:
    • Farasa segmentation followed by the AraBERT WordPiece tokenizer.
    • Global max pooling over token vectors.
    • A softmax classifier.
    • AraBERT itself is a BERT-base-sized model pretrained on about 24GB of Arabic text.
  • Error analysis identified four main problems:
    • Class imbalance: Egypt reached about 68% F1, Djibouti about 8%, and Bahrain 0%.
    • Religious formulas shared across dialects.
    • Single dialect-marker words that mislead the model.
    • Topic bias, for example soccer tweets being labeled Djibouti.

Interesting ideas

  • I1. Tuning soft-voting weights on dev data showed that the weaker-looking surface model deserved more weight than the transformer, so complementary models should be weighted by evidence rather than equally.
  • I2. Heuristically harvested training examples were kept but given a quarter of the weight of gold examples, so they help without overwhelming the trusted labels.
  • I3. Non-lexical symbols (emojis) were turned into a mask placeholder plus a natural-language gloss in a separate segment, so the model sees content it can interpret.
  • I4. Character 3–5-grams were used to cope with non-standard spelling and out-of-vocabulary words.
  • I5. The finding that one class was learned mostly through its dominant topic shows that a classifier can pick up a shortcut instead of the intended signal, and this is worth testing for explicitly.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When several witness parsers vote on silver trees, their votes could be weighted by per-witness reliability measured on a held-out gold set, instead of using plain majority agreement.
  • I2 (learn): Silver or LLM-teacher annotations could enter perceptron and gparse training with a reduced sample weight relative to treebank gold, adjusted by their agreement level.
  • I3 (tokens): The tokenizer could map emojis and similar symbols to a placeholder token with a stable POS and attach their gloss as a lemma-like feature, rather than leaving them as unknown characters.
  • I4 (tokens): A character n-gram backoff could give the tagger and morphology analyzer an explainable guess for misspelled or unseen informal English words.
  • I5 (learn): Evaluation could add negative controls that hold the topic fixed while changing the structure (and the reverse), to catch decisions driven by topic rather than syntax or meaning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

💡 Concepts 1Key ideas and methods explained on our concept pages.N-gram · 26 works

Cite

Cite as Abdellah El Mekki, Ahmed Alami, Hamza Alami, Ahmed Khoumsi and Ismail Berrada. 2020. Weighted combination of BERT and N-GRAM features for Nuanced Arabic Dialect Identification. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mekki2020weighted,
  title = {{Weighted combination of BERT and N-GRAM features for Nuanced Arabic Dialect Identification}},
  author = {Abdellah {El Mekki}, Ahmed Alami, Hamza Alami, Ahmed Khoumsi and Ismail Berrada},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.27/}
}

References

20 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mourad Abbas, Mohamed Lichouri, and Abed Alhakim Freihat. 2019. find
  2. ST MADAR 2019 shared task: Arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 269–273, Florence, Italy, August. Association for Computational Linguistics.
  3. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for Arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16, San Diego, California, June. Association for Computational Linguistics.
  4. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. NADI 2020: The First Nuanced Arabic Dialect Identification Shared Task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop (WANLP 2020), Barcelona, Spain. ↗
  5. Mittal Anshul and Goel Arpit. 2012. Stock prediction using twitter sentiment analysis. find
  6. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  7. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR shared task on Arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199–207, Florence, Italy. Association for Computational Linguistics.
  8. Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555. ↗
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. ↗
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics.
  11. Nizar Habash, Abdelhadi Soudi, and Tim Buckwalter. 2007. On Arabic Transliteration. In A. van den Bosch and A. Soudi, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods, pages 15–22. Springer, Netherlands. ↗
  12. Nizar Habash, Mona Diab, and Owen Rambow. 2012. Conventional orthography for dialectal Arabic. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 711–718, Istanbul, Turkey, May. European Language Resources Association (ELRA).
  13. Alami Hamza, Ouatik El Alaoui Said, Benlahbib Abdessamad, and En-nahnahi Noureddine. 2020. Lisac fsdm-usmba team at semeval 2020 task 12: Overcoming arabert’s pretrain-finetune discrepancy for arabic offensive language identification.
  14. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  15. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2012, Kyoto, Japan, March 25-30, 2012, pages 5149–5152. IEEE. find
  16. Bashar Talafha, Wael Farhan, Ahmed Altakrouri, and Hussein Al-Natsheh. 2019. Mawdoo3 AI at MADAR shared task: Arabic tweet dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 239–243, Florence, Italy, August. Association for Computational Linguistics.
  17. Wikipedia. 2020. ISO 3166-1 alpha-2 — Wikipedia, the free encyclopedia. http://en.wikipedia.org/ w/index.php?title=ISO\%203166-1\%20alpha-2&oldid=972097545. [Online; accessed 13-August-2020]. ↗
  18. Omar Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Comput. Linguistics, 40(1):171–202. ↗
  19. Chiyu Zhang and Muhammad Abdul-Mageed. 2019. No army, no navy: BERT semi-supervised learning of Arabic dialects. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 279–284, Florence, Italy, August. Association for Computational Linguistics.
  20. Tong Zhang. 2004. Solving large scale linear prediction problems using stochastic gradient descent algorithms. In Proceedings of the Twenty-First International Conference on Machine Learning, ICML ’04, page 116, New York, NY, USA. Association for Computing Machinery. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mekki-2020-weighted-combination-bert-n-gram/