Elamur Research⌕
☰
home / papers / 2018 / Deep Models for Arabic Dialect Identification on Benchmarked Data

Deep Models for Arabic Dialect Identification on Benchmarked Data

Paper · 2018 · Mohamed Elaraby and Muhammad Abdul-Mageed

Summary

The authors look at automatic identification of Arabic varieties in user comments. Their starting point is the Arabic Online Commentary (AOC) corpus, a large collection of comments from news sites in which a crowd-labeled portion carries one of four tags: MSA, Egyptian (EGY), Gulf (GLF) or Levantine (LEV). Earlier work on AOC relied on cross-validation and came before neural methods. That made fair comparison hard and left open how well deep networks handle these closely related varieties.

The authors fix a random 80/10/10 train/dev/test split and offer it to other researchers. They then compare classical n-gram classifiers with six neural architectures, each initialized in three ways. The models are tested on three tasks: MSA vs. dialect, a 3-way choice among dialects, and the 4-way choice among all varieties.

Bidirectional recurrent models, especially an attention-based BiLSTM, give the best results when initialized with dialect-rich Twitter embeddings. A vocabulary-overlap analysis helps explain which varieties get confused with each other. The paper gives a reproducible reference point for neural Arabic dialect identification.

Key points

  • AOC holds about 3M comments, of which 108,173 have manual labels. MSA makes up about 58% of the labeled data.
  • Classical baselines: logistic regression, multinomial Naive Bayes and SVM, each run on 1–3-gram counts and on 1–3-gram TF-IDF features.
  • Neural models: CNN, LSTM, CLSTM, BiLSTM, BiGRU and Attention-BiLSTM.
  • Embedding initializations: random, word2vec trained on AOC, and Twitter-City CBOW vectors. The Twitter-City vectors are 300-dimensional and were trained on roughly 1B tweets from 29 cities in 10 Arab countries.
  • Test accuracy (Table 3):
    • Binary: BiGRU reaches 87.23 (87.65 on Dev).
    • 3-way: Attention-BiLSTM reaches 87.41. On this task Naive Bayes ties or beats it, with 87.81 on Test.
    • 4-way: Attention-BiLSTM reaches 82.45.
    • Majority-class baselines are 58.75 (binary and 3-way) and 46.49 (4-way).
  • Error patterns:
    • GLF and LEV share more vocabulary with MSA than EGY does.
    • In the 3-way task, LEV is mistaken for GLF 20% of the time.
    • In the 4-way task, GLF goes to MSA in 23% of cases and LEV goes to MSA in 20%.

Interesting ideas

  • I1. Replacing costly n-fold cross-validation with one fixed, published train/dev/test split makes slow-to-train neural models comparable across papers.
  • I2. A heat map of shared vocabulary, normalized by class size, predicted in advance which classes the model would later confuse. Cheap corpus statistics can thus explain errors before any training.
  • I3. Initializing from large, domain-matched pretrained embeddings beat both random vectors and vectors trained only on the small in-task corpus.
  • I4. Simple n-gram Naive Bayes stayed competitive and won on the smaller dialect-only task. Strong shallow baselines are needed to show that a deep model actually adds something.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Freeze and version the evaluation splits for each treebank and QA set, so that parser or reader changes are always measured on identical held-out data.
  • I2 (tree): Compute lexical and structural overlap between UD treebanks or English registers, and use it to predict and explain where the parser will mislabel text after dialect conversion.
  • I3 (tree): Initialize gparse from representations trained on large text of the target register rather than only on the treebank, and check the gain against a random-initialization control.
  • I4 (learn): Always report a shallow baseline next to each neural component as a negative control, especially on low-data slices where deep models may not pay off.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PittsburghPittsburgh · United States59 works · 22,324 citations🎓University of British ColumbiaVancouver · Canada98 works · 34,805 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Mohamed Elaraby and Muhammad Abdul-Mageed. 2018. Deep Models for Arabic Dialect Identification on Benchmarked Data. Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{elaraby2018deep,
  title = {{Deep Models for Arabic Dialect Identification on Benchmarked Data}},
  author = {Mohamed Elaraby and Muhammad Abdul-Mageed},
  booktitle = {Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018)},
  year = {2018},
  url = {https://aclanthology.org/W18-3930/}
}

References

53 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martı́n Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. Tensorflow: A system for large-scale machine learning. In OSDI, volume 16, pages 265–283. ↗
  2. Muhammad Abdul-Mageed, Anneke Buffone, Hao Peng, Johannes C Eichstaedt, and Lyle H Ungar. 2017. Recognizing pathogenic empathy in social media. In ICWSM, pages 448–451. find
  3. Muhammad Abdul-Mageed, Hassan Alhuzali, and Mohamed Elaraby. 2018. You tweet what you speak: A city-level dataset of arabic dialects. In LREC, pages 3653–3659.
  4. Muhammad Abdul-Mageed. 2015. Subjectivity and sentiment analysis of Arabic as a morophologically-rich language. Ph.D. thesis, Indiana University. find
  5. Muhammad Abdul-Mageed. 2017. Modeling arabic subjectivity and sentiment in lexical space. Information Processing & Management. ↗
  6. Steven Abney and Steven Bird. 2010. The human language project: building a universal corpus of the world’s languages. In Proceedings of the 48th annual meeting of the association for computational linguistics, pages 88–97. Association for Computational Linguistics. find
  7. Ahmed Ali, Najim Dehak, Patrick Cardinal, Sameer Khurana, Sree Harsha Yella, James Glass, Peter Bell, and Steve Renals. 2015. Automatic dialect detection in arabic broadcast speech. arXiv preprint arXiv:1509.06928. ↗
  8. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. ↗
  9. Yonatan Belinkov and James Glass. 2016. A character-level convolutional neural network for distinguishing similar languages and dialects. arXiv preprint arXiv:1609.07568. ↗
  10. François Chollet et al. 2015. Keras. find
  11. Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. 2015. Attention-based models for speech recognition. In Advances in neural information processing systems, pages 577–585. ↗
  12. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555. ↗
  13. Ryan Cotterell and Chris Callison-Burch. 2014. A multi-dialect, multi-genre corpus of informal written arabic. In LREC, pages 241–245.
  14. Kareem Darwish, Hassan Sajjad, and Hamdy Mubarak. 2014. Verifiably effective arabic dialect identification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1465–1468.
  15. Mona Diab, Nizar Habash, Owen Rambow, Mohamed Altantawy, and Yassine Benajiba. 2010. Colaba: Arabic dialect annotation and processing. In Lrec workshop on semitic language processing, pages 66–74. find
  16. Heba Elfardy and Mona Diab. 2013. Sentence level dialect identification in arabic. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 456–461.
  17. Heba Elfardy, Mohamed Al-Badrashiny, and Mona Diab. 2014. Aida: Identifying code switching in informal arabic text. In Proceedings of The First Workshop on Computational Approaches to Code Switching, pages 94–101.
  18. Nizar Habash, Mona T Diab, and Owen Rambow. 2012. Conventional orthography for dialectal arabic. In LREC, pages 711–718.
  19. Nizar Y Habash. 2010. Introduction to arabic natural language processing. Synthesis Lectures on Human Language Technologies, 3(1):1–187. ↗
  20. Bartosz Hawelka, Izabela Sitko, Euro Beinat, Stanislav Sobolevsky, Pavlos Kazakopoulos, and Carlo Ratti. 2014. Geo-located twitter as proxy for global mobility patterns. Cartography and Geographic Information Science, 41(3):260–271. ↗
  21. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. ↗
  22. Fei Huang. 2015. Improved arabic dialect classification with social media data. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2118–2126.
  23. Tommi Jauhiainen, Krister Lindén, and Heidi Jauhiainen. 2017. Evaluation of language identification methods using 285 languages. In Proceedings of the 21st Nordic Conference on Computational Linguistics, pages 183– 191. find
  24. Tommi Jauhiainen, Marco Lui, Marcos Zampieri, Timothy Baldwin, and Krister Lindén. 2018. Automatic language identification in texts: A survey. arXiv preprint arXiv:1804.08186. ↗
  25. Raja Jurdak, Kun Zhao, Jiajun Liu, Maurice AbouJaoude, Mark Cameron, and David Newth. 2015. Understanding human mobility from twitter. PloS one, 10(7):e0131469. ↗
  26. David Jurgens, Yulia Tsvetkov, and Dan Jurafsky. 2017. Incorporating dialectal variability for socially equitable language identification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 51–57. find
  27. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882. ↗
  28. Tom Kocmi and Ondřej Bojar. 2017. Lanidenn: Multilingual language identification on character window. arXiv preprint arXiv:1701.03338. ↗
  29. Wang Ling, Guang Xiang, Chris Dyer, Alan Black, and Isabel Trancoso. 2013. Microblogs as parallel corpora. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 176–186. find
  30. Thomas Louail, Maxime Lenormand, Oliva G Cantu Ros, Miguel Picornell, Ricardo Herranz, Enrique Frias-Martinez, José J Ramasco, and Marc Barthelemy. 2014. From mobile phone data to the spatial structure of cities. Scientific reports, 4:5276. ↗
  31. Marco Lui and Timothy Baldwin. 2012. langid. py: An off-the-shelf language identification tool. In Proceedings of the ACL 2012 system demonstrations, pages 25–30. Association for Computational Linguistics. — langid.py: An Off-the-shelf Language Identification Tool
  32. Jalal Mahmud, Jeffrey Nichols, and Clemens Drews. 2014. Home location identification of twitter users. ACM Transactions on Intelligent Systems and Technology (TIST), 5(3):47. ↗
  33. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. ↗
  34. Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807–814. find
  35. Maryam Najafian, Sameer Khurana, Suwon Shon, Ahmed Ali, and James Glass. 2018. Exploiting convolutional neural networks for phonotactic based dialect identification. ICASSP. find
  36. Quynh C Nguyen, Dapeng Li, Hsien-Wen Meng, Suraj Kath, Elaine Nsoesie, Feifei Li, and Ming Wen. 2016. Building a national neighborhood dataset from geotagged twitter data for indicators of happiness, diet, and physical activity. JMIR public health and surveillance, 2(2). ↗
  37. Quynh C Nguyen, Kimberly D Brunisholz, Weijun Yu, Matt McCullough, Heidi A Hanson, Michelle L Litchman, Feifei Li, Yuan Wan, James A VanDerslice, Ming Wen, et al. 2017. Twitter-derived neighborhood characteristics associated with obesity and diabetes. Scientific reports, 7(1):16425. ↗
  38. Leysia Palen and Amanda L Hughes. 2018. Social media in disaster communication. In Handbook of Disaster Research, pages 497–518. Springer. ↗
  39. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. Journal of machine learning research, 12(Oct):2825–2830. ↗
  40. Afshin Rahimi, Trevor Cohn, and Timothy Baldwin. 2017. A neural model for user geolocation and lexical dialectology. arXiv preprint arXiv:1704.04008. ↗
  41. Afshin Rahimi, Trevor Cohn, and Tim Baldwin. 2018. Semi-supervised user geolocation via graph convolutional networks. arXiv preprint arXiv:1804.08049. ↗
  42. Takeshi Sakaki, Makoto Okazaki, and Yutaka Matsuo. 2010. Earthquake shakes twitter users: real-time event detection by social sensors. In Proceedings of the 19th international conference on World wide web, pages 851–860. ACM. find
  43. Suwon Shon, Ahmed Ali, and James Glass. 2017. Mit-qcri arabic dialect identification system for the 2017 multi-genre broadcast challenge. arXiv preprint arXiv:1709.00387. ↗
  44. Suwon Shon, Ahmed Ali, and James Glass. 2018. Convolutional neural networks and language embeddings for end-to-end dialect recognition. arXiv preprint arXiv:1803.04567. ↗
  45. Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11–21. ↗
  46. Christoph Tillmann, Saab Mansour, and Yaser Al-Onaizan. 2014. Improved sentence-level arabic dialect classification. In Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects, pages 110–119.
  47. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000–6010.
  48. Yan Xu, Lili Mou, Ge Li, Yunchuan Chen, Hao Peng, and Zhi Jin. 2015. Classifying relations via long short term memory networks along shortest dependency paths. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1785–1794. ↗
  49. Antonio Jimeno Yepes, Andrew MacKinlay, and Bo Han. 2015. Investigating public health surveillance using twitter. Proceedings of BioNLP 15, pages 164–170. find
  50. Omar F Zaidan and Chris Callison-Burch. 2011. The arabic online commentary dataset: an annotated dataset of informal arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 37–41. Association for Computational Linguistics.
  51. Omar F Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗
  52. Chunting Zhou, Chonglin Sun, Zhiyuan Liu, and Francis Lau. 2015. A c-lstm neural network for text classification. arXiv preprint arXiv:1511.08630. ↗
  53. Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 207–212.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/elaraby-2018-deep-models-arabic-dialect-identification/