Elamur Research⌕
☰
home / papers / 2018 / Character Level Convolutional Neural Network for Arabic Dialect Identification

Character Level Convolutional Neural Network for Arabic Dialect Identification

Paper · 2018 · Mohamed Ali

Summary

The paper presents the system that team "safina" submitted for the VarDial 2018 shared task on identifying Arabic dialects (ADI). The task was to assign each utterance to one of five varieties: MSA, Egyptian, Gulf, Levantine, or North-African. The author used two of the provided inputs: ASR transcripts and precomputed 600-dimensional dialect embedding vectors. The model is a character-level CNN. Transcript characters are one-hot encoded and passed through convolutions of several widths with max-over-time pooling, then a softmax. The dialect embeddings feed a separate softmax, and the network's prediction is the mean of the two output distributions. Three variants differ only in what sits before the convolution: nothing, an embedding layer, or a GRU layer. The GRU variant scored best on the test set at about 57.6% F1 and placed second of six teams. Its margin over the plain one-hot version was small, though, and it took roughly ten times longer to train. The work shows that a compact character-based neural model combined with auxiliary embeddings is competitive for telling closely related dialects apart.

Key points

  • The data consisted of 14,591 training and 1,566 validation utterances, each with four feature sets: raw audio, ASR transcripts from a multi-dialect LVCSR system, 600-dim dialect embeddings produced by a Siamese network (Shon et al., 2018), and phoneme sequences from Czech, Hungarian, Russian and English recognizers. Only the transcripts and the dialect embeddings were used.
  • Settings chosen by cross-validation:
    • inputs truncated or padded to 256 characters;
    • embedding size 128, GRU size 128;
    • filter widths 2 through 8, with 256 feature maps each;
    • ReLU plus batch normalization after the convolutions;
    • dropout of 0.2.
    • Implementation in Keras on TensorFlow.
  • 5-fold cross-validation accuracy (train and validation merged):
    • TF-IDF Logistic Regression baseline: 0.5898;
    • one-hot CNN: 0.9214;
    • embedding CNN: 0.9262;
    • GRU CNN: 0.9264.
  • Test F1:
    • random baseline: 0.1995;
    • one-hot: 0.5711;
    • embedding: 0.5697;
    • GRU: 0.5759.
    • The GRU run ranked 2nd of 6.
    • The text calls the ranking metric weighted F1, while the table header says macro F1.
  • In the confusion matrix, Gulf is the hardest class and is often predicted as Levantine. Levantine in turn is frequently mislabeled as North-African.

Interesting ideas

  • I1. Late fusion by averaging probabilities: each input source gets its own softmax, and the final prediction is their mean, so every source's vote stays visible.
  • I2. Several convolution widths (2–8) followed by max-over-time pooling act as learned character n-gram detectors. Each pooled feature corresponds to one strongest-matching substring, which can be inspected.
  • I3. The author reports training cost next to accuracy and concludes that a tiny gain from the GRU did not justify about ten times the training time.
  • I4. The related work warns that single-source corpora are homogeneous and do not generalize to new topics. The paper's own numbers echo this, with roughly 0.92 cross-validation accuracy against roughly 0.58 test F1.
  • I5. A confusion matrix shows which neighbouring varieties blur together, which is more useful than a single score for similar-class problems.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Witness parsers or taggers could each emit a distribution that is combined by simple averaging, keeping a traceable record of which witness pushed a silver label in which direction.
  • I2 (tokens): A model of this kind could guess the POS or lemma of unknown words from affix-like substrings, and the winning substring would serve as an explanation for the guess.
  • I3 (learn): Evaluation reports could log compute time alongside every metric, so that heavier architectures such as gparse variants are adopted only when the gain pays for the cost.
  • I4 (learn): Held-out evaluation should draw on sources or genres absent from training (for example, other fables or other school tests), so that inflated in-distribution scores do not hide brittleness.
  • I5 (tree): During UD dialect conversion between treebank conventions, a confusion matrix over relation labels would show which conventions get mixed up and where conversion rules need work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Mohamed Ali. 2018. Character Level Convolutional Neural Network for Arabic Dialect Identification. Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ali2018character,
  title = {{Character Level Convolutional Neural Network for Arabic Dialect Identification}},
  author = {Mohamed Ali},
  booktitle = {Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018)},
  year = {2018},
  url = {https://aclanthology.org/W18-3913/}
}

References

24 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martı́n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. Tensor-Flow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org. find
  2. Ahmed Ali, Najim Dehak, Patrick Cardinal, Sameer Khurana, Sree Harsha Yella, James Glass, Peter Bell, and Steve Renals. 2016. Automatic Dialect Detection in Arabic Broadcast Speech. In Proceedings of INTER-SPEECH, pages 2934–2938.
  3. Yonatan Belinkov and James Glass. 2016. A Character-level Convolutional Neural Network for Distinguishing Similar Languages and Dialects. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages 145–152, Osaka, Japan. ↗
  4. Fadi Biadsy, Julia Hirschberg, and Nizar Habash. 2009. Spoken arabic dialect identification using phonotactic modeling. In Proceedings of the eacl 2009 workshop on computational approaches to semitic languages, pages 53–61. Association for Computational Linguistics.
  5. Çağri Çöltekin and Taraka Rama. 2016. Discriminating Similar Languages with Linear SVMs and Neural Networks. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages 15–24, Osaka, Japan. find
  6. François Chollet et al. 2015. Keras. https://keras.io. ↗
  7. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555. ↗
  8. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM. find
  9. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug):2493– 2537.
  10. Kareem Darwish, Hassan Sajjad, and Hamdy Mubarak. 2014. Verifiably effective arabic dialect identification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1465–1468.
  11. Heba Elfardy and Mona Diab. 2013. Sentence level dialect identification in arabic. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 456–461.
  12. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep sparse rectifier neural networks. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 315–323. find
  13. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167. ↗
  14. Radu Tudor Ionescu and Andrei Butnaru. 2017. Learning to identify arabic and german dialects using multiple kernels. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), pages 200–209, Valencia, Spain, April.
  15. Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 2013. 3d convolutional neural networks for human action recognition. IEEE transactions on pattern analysis and machine intelligence, 35(1):221–231. ↗
  16. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105.
  17. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  18. Pierre Sermanet, David Eigen, Xiang Zhang, Michaël Mathieu, Rob Fergus, and Yann LeCun. 2013. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229. ↗
  19. Suwon Shon, Ahmed Ali, and James Glass. 2018. Convolutional neural network and language embeddings for end-to-end dialect recognition. In Proc. Odyssey 2018 The Speaker and Language Recognition Workshop, pages 98–104. find
  20. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958. find
  21. Christoph Tillmann, Saab Mansour, and Yaser Al-Onaizan. 2014. Improved sentence-level arabic dialect classification. In Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects, pages 110–119.
  22. Omar F Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗
  23. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Ahmed Ali, Suwon Shon, James Glass, Yves Scherrer, Tanja Samardžić, Nikola Ljubešić, Jörg Tiedemann, Chris van der Lee, Stefan Grondelaers, Nelleke Oostdijk, Antal van den Bosch, Ritesh Kumar, Bornini Lahiri, and Mayank Jain. 2018. Language Identification and Morphosyntactic Tagging: The Second VarDial Evaluation Campaign. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), Santa Fe, USA.
  24. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pages 649–657. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ali-2018-character-level-convolutional-neural-network/