Elamur Research⌕
☰
home / papers / 2018 / Neural Network Architectures for Arabic Dialect Identification

Neural Network Architectures for Arabic Dialect Identification

Paper · 2018 · Elise Michon, Minh Quang Pham, Josep Crego et al.

Summary

SYSTRAN's first entry in the VarDial 2018 DSL shared task tackled the Arabic Dialect Identification (ADI) subtask. The goal was to assign each utterance to one of five classes: Egyptian, Gulf, Levantine, North African, or Modern Standard Arabic. The organisers supplied three kinds of input per utterance: ASR word transcripts in Buckwalter, phone transcripts from four phoneme recognisers, and precomputed acoustic embeddings. Earlier editions had found that neural models trailed SVM and KRR on this small dataset. The authors set out to show that neural networks could still be competitive.

They submitted three runs:

  • a Multi-Input CNN that learns a separate embedding for each text stream and then joins them with the acoustic vectors;
  • an end-to-end CNN-biLSTM working on mel-spectrograms they computed themselves;
  • a set of one-vs-rest CNN-biLSTMs.

The Multi-Input CNN was their best system, reaching macro F1 0.5289 and third place. Ablations showed that most of its power came from the acoustic features. The paper also documents several data problems: inconsistent labelling, duplicated audio, empty transcripts, and transcripts biased toward MSA. These findings make it a useful lesson in how much the input representation shapes dialect classification.

Key points

  • The data has 14,591 train, 1566 dev, 1492 MGB-3 test and 5345 Youtube test utterances. Phone transcripts came from Brno recognisers for Czech, English, Hungarian and Russian, and the acoustic embeddings were 600-dimensional.
  • Run 1, the Multi-Input CNN built in Keras/TensorFlow, scored F1 0.5289. It used char-level convolutions on words, phone-level convolutions on four phone streams, and the given acoustic embeddings, and ranked 3rd.
  • Run 2 was a CNN-biLSTM on log-amplitude mel-spectrograms (Librosa, 40 bands). It scored F1 0.3894 at submission, rising to 0.4235 later with smaller filters (864, 464). Run 3, the binary CNN-biLSTMs, scored 0.4339.
  • Masking, class-balanced batches and batch normalization all lowered scores, and 2D convolutions were costly and weaker. The CNN alone and the biLSTM alone each stayed at the random baseline (F1 0.1995).
  • The data audit found several problems:
    • labels sometimes followed the speaker and sometimes the spoken variety;
    • code-switching with MSA was frequent;
    • up to 6% of North African training files were duplicates;
    • about 5% of training transcripts were empty.

Interesting ideas

  • I1. Give each input view (characters, each phone recogniser's output, acoustic vectors) its own small encoder with its own dimensionality, and fuse them only near the classifier; this lets heterogeneous signals coexist without forcing a shared space.
  • I2. Ablate per input stream and compare each against the full combination; here, acoustic features alone nearly matched everything, revealing what the model actually used.
  • I3. Test each architectural piece alone against a chance baseline; finding that the CNN and the biLSTM were each at chance proved the gain came only from their combination.
  • I4. Compare per-class F1 inside one-vs-rest models with per-class F1 after they are combined; identical numbers showed the combination added nothing.
  • I5. Audit the data before modelling, looking for label criteria that drift (speaker identity vs. spoken variety), duplicates, empty inputs and mixed varieties inside one item.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): the tagger could keep separate char-level, affix and lexicon-lookup encoders whose outputs are concatenated late, so each view's contribution stays inspectable.
  • I2 (learn): routinely retrain or evaluate with one feature family removed, as a negative control that shows whether a component's claimed signal is real.
  • I3 (learn): when a pipeline stage is a composite (e.g. parser plus repair rules), report each part's score next to a random baseline to make the credit assignment explicit.
  • I4 (frames): if speech-act or irony detection is split into binary detectors, this check reveals whether the arbitration layer adds anything or just passes errors through.
  • I5 (learn): run the same audit on silver treebank data, flagging duplicated sentences and items where witness parsers disagree because the annotation criterion itself is inconsistent.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Centre National de la Recherche ScientifiqueParis · France168 works · 62,369 citations🎓Université Paris-SaclayGif-sur-Yvette · France45 works · 6,127 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksSpeech & spoken language · 480 works

Cite

Cite as Elise Michon, Minh Quang Pham, Josep Crego and Jean Senellart. 2018. Neural Network Architectures for Arabic Dialect Identification. Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{michon2018neural,
  title = {{Neural Network Architectures for Arabic Dialect Identification}},
  author = {Elise Michon, Minh Quang Pham, Josep Crego and Jean Senellart},
  booktitle = {Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018)},
  year = {2018},
  url = {https://aclanthology.org/W18-3914/}
}

References

24 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martı́n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. Tensor-Flow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org. find
  2. Ahmed Ali, Yifan Zhang, Patrick Cardinal, Najim Dahak, Stephan Vogel, and James Glass. 2014. A complete kaldi recipe for building arabic speech recognition systems. In Spoken Language Technology Workshop (SLT), 2014 IEEE, pages 525–529. IEEE. find
  3. Ahmed Ali, Najim Dehak, Patrick Cardinal, Sameer Khurana, Sree Harsha Yella, James Glass, Peter Bell, and Steve Renals. 2016. Automatic Dialect Detection in Arabic Broadcast Speech. In Proceedings of INTER-SPEECH, pages 2934–2938.
  4. Ahmed Ali, Stephan Vogel, and Steve Renals. 2017. Speech Recognition Challenge in the Wild: Arabic MGB-3. arXiv preprint arXiv:1709.07276. ↗
  5. Christian Bartz, Tom Herold, Haojin Yang, and Christoph Meinel. 2017. Language identification using deep convolutional recurrent neural networks. In International Conference on Neural Information Processing, pages 880–889. Springer. find
  6. Yonatan Belinkov and James Glass. 2016. A Character-level Convolutional Neural Network for Distinguishing Similar Languages and Dialects. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages 145–152, Osaka, Japan. ↗
  7. Çağri Çöltekin and Taraka Rama. 2016. Discriminating Similar Languages with Linear SVMs and Neural Networks. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages 15–24, Osaka, Japan. find
  8. Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1–27:27. ↗
  9. Keunwoo Choi, György Fazekas, Mark Sandler, and Kyunghyun Cho. 2017. Convolutional recurrent neural networks for music classification. In Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on, pages 2392–2396. IEEE. find
  10. François Chollet et al. 2015. Keras. https://keras.io. ↗
  11. Mohamed Eldesouki, Fahim Dalvi, Hassan Sajjad, and Kareem Darwish. 2016. QCRI @ DSL 2016: Spoken Arabic Dialect Identification Using Textual Features. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages 221–226, Osaka, Japan.
  12. Sriram Ganapathy, Kyu Han, Samuel Thomas, Mohamed Omar, Maarten Van Segbroeck, and Shrikanth S Narayanan. 2014. Robust language identification using convolutional neural network features. In Fifteenth annual conference of the international speech communication association. find
  13. Chinnappa Guggilla. 2016. Discrimination between Similar Languages, Varieties and Dialects using CNN- and LSTM-based Deep Neural Networks. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages 185–194, Osaka, Japan.
  14. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. In AAAI, pages 2741–2749.
  15. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882. ↗
  16. Bo Li, Tara N Sainath, Khe Chai Sim, Michiel Bacchiani, Eugene Weinstein, Patrick Nguyen, Zhifeng Chen, Yonghui Wu, and Kanishka Rao. 2017. Multi-dialect speech recognition with a single sequence-to-sequence model. arXiv preprint arXiv:1712.01541. ↗
  17. Shervin Malmasi, Marcos Zampieri, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, and Jörg Tiedemann. 2016. Discriminating between similar languages and arabic dialect identification: A report on the third dsl shared task. In Proceedings of the 3rd Workshop on Language Technology for Closely Related Languages, Varieties and Dialects (VarDial), Osaka, Japan.
  18. Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. 2015. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science conference, pages 18–25. find
  19. Fatiha Sadat, Farnazeh Kazemi, and Atefeh Farzindar. 2014. Automatic identification of arabic dialects in social media. In Proceedings of the first international workshop on Social media retrieval and analysis, pages 35–40. ACM. find
  20. Suwon Shon, Ahmed Ali, and James Glass. 2018. Convolutional Neural Networks and Language Embeddings for End-to-End Dialect Recognition. ArXiv e-prints arXiv:1803.04567. ↗
  21. Omar F Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗
  22. Marcos Zampieri, Shervin Malmasi, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, Jörg Tiedemann, Yves Scherrer, and Noëmi Aepli. 2017. Findings of the VarDial Evaluation Campaign 2017. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), Valencia, Spain.
  23. Ye Zhang, Stephen Roller, and Byron Wallace. 2016. Mgnc-cnn: A simple approach to exploiting multiple word embeddings for sentence classification. arXiv preprint arXiv:1603.00968. ↗
  24. Ayah Zirikly, Bart Desmet, and Mona Diab. 2016. The GW/LT3 VarDial 2016 Shared Task System for Dialects and Similar Languages Detection. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages 33–41, Osaka, Japan. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/michon-2018-neural-network-architectures-arabic-dialect/