Elamur Research⌕
☰
home / papers / 2016 / Discrimination between Similar Languages, Varieties and Dialects using CNN- and LSTM-based Deep Neural Networks

Discrimination between Similar Languages, Varieties and Dialects using CNN- and LSTM-based Deep Neural Networks

Paper · 2016 · Chinnappa Guggilla

Summary

The paper presents CGLI, a system entered in the Arabic dialect identification sub-task of the DSL 2016 shared task (closed track). The task is hard because all varieties use the same script and share much of their vocabulary. Each input is an ASR transcript, and the system assigns it to one of five classes: Egyptian, Gulf, Levantine, North-African or MSA. Instead of hand-built word, character or morphological features, the author trains two neural classifiers end to end on word embeddings. The first is a Kim (2014)-style CNN with convolution, max-over-time pooling and a softmax output. The second is an LSTM sequence model. The CNN, run with default hyper-parameters and no pre-trained embeddings, reached 43.29% weighted F1 on the test set. The official LSTM runs fell below the majority baseline because of a bug in how results were compiled. Once fixed, the LSTM scored about 42%. The work matters as an early test of feature-free deep models on closely related varieties. It also exposes their practical costs: long CPU training times and weak accuracy without tuning.

Key points

  • Five dialect labels (egy, glf, lav, msa, nor), closed training, ranked by weighted F1. The majority baseline was 0.2279.
  • CNN run3 results on the test set:
    • accuracy 0.4377
    • macro F1 0.4364
    • weighted F1 0.4329
  • Official LSTM runs and the post-fix result:
    • run1, random embeddings: weighted F1 0.1779
    • run2, pre-compiled embeddings: weighted F1 0.1944
    • after the bug fix, LSTM without pooling: about 0.423
    • after the bug fix, LSTM with average pooling: 0.4172
  • In the confusion matrix, MSA and Egyptian are recognised best. North-African and Levantine are confused most, mainly with Egyptian and Gulf. Gulf also overlaps with Egyptian and Levantine.
  • Compute cost was a major limitation:
    • 5-fold CNN cross-validation took 24 hours on a single i7 CPU with 8 GB RAM.
    • Training on sub-task 1 took 10 days, so no results were submitted for it.

Interesting ideas

  • I1. When an official score lands under the majority baseline, the author treats it as a sign of a pipeline fault rather than model weakness, and indeed finds a bug; a trivial baseline thus doubles as a sanity alarm.
  • I2. Removing pooling or switching to average pooling barely moves the overall F1, yet it moves accuracy between specific classes in opposite directions, so a single aggregate number hides real trade-offs.
  • I3. The confusion matrix is used to locate which pairs of varieties collapse into each other, and a targeted fix is proposed: build better representations only for the vocabulary shared by the most confused classes.
  • I4. The classifier relies on no language-specific analysis, so the same architecture could be pointed at any group of close varieties.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Every evaluation report could place a majority-class or trivial-rule control next to the real score and automatically flag any run that fails to beat it as a probable harness bug.
  • I2 (learn): Ablations of the parser or tagger should be reported per label (per deprel or per POS), so that a change which looks neutral overall but hurts one relation is not accepted silently.
  • I3 (tokens): For tag pairs the tagger often mixes up (e.g. ADP vs SCONJ, VBN vs VBD), Mova could collect the words that occur in both tags and give them dedicated disambiguation rules or extra silver data.
  • I4 (tree): A lightweight variety or register detector (British vs American, informal vs edited text) run before parsing could tell Mova which UD dialect-conversion rules or treebank-specific model to apply.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢AccentureZurich · Switzerland6 works · 27 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Chinnappa Guggilla. 2016. Discrimination between Similar Languages, Varieties and Dialects using CNN- and LSTM-based Deep Neural Networks. Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{guggilla2016discrimination,
  title = {{Discrimination between Similar Languages, Varieties and Dialects using CNN- and LSTM-based Deep Neural Networks}},
  author = {Chinnappa Guggilla},
  booktitle = {Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3)},
  year = {2016},
  url = {https://aclanthology.org/W16-4824/}
}

References

37 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fadi Biadsy, Julia Hirschberg, and Nizar Habash. 2009. Spoken arabic dialect identification using phonotactic modeling. In Proceedings of the eacl 2009 workshop on computational approaches to semitic languages, pages 53–61. Association for Computational Linguistics.
  2. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537.
  3. Kareem Darwish, Hassan Sajjad, and Hamdy Mubarak. 2014. Verifiably effective arabic dialect identification. In EMNLP, pages 1465–1468.
  4. Heba Elfardy and Mona T Diab. 2013. Sentence level dialect identification in arabic. In ACL (2), pages 456–461.
  5. Marc Franco-Salvador, Paolo Rosso, and Francisco Rangel. 2015. Distributed representations of words and documents for discriminating similar languages. In Proceedings of the Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects (LT4VarDial), pages 11–16, Hissar, Bulgaria. find
  6. Felix A. Gers, Jürgen Schmidhuber, and Fred Cummins. 2000. Learning to forget: Continual prediction with LSTM. Neural Computation, 12(10):2451–2471. ↗
  7. Felix Gers. 2001. Long Short-term Memory in Recurrent Neural Networks. Ph.D. thesis, Universität Hannover. ↗
  8. Cyril Goutte and Serge Leger. 2015. Experiments in discriminating similar languages. In Proceedings of the Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects (LT4VarDial), pages 78–84, Hissar, Bulgaria. find
  9. Cyril Goutte, Serge Léger, and Marine Carpuat. 2014. The nrc system for discriminating similar languages. In Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects (VarDial), pages 139–145, Dublin, Ireland. find
  10. Cyril Goutte, Serge Léger, Shervin Malmasi, and Marcos Zampieri. 2016. Discriminating Similar Languages: Evaluations and Explorations. find
  11. Alex Graves. 2013. Generating sequences with recurrent neural networks. CoRR, abs/1308.0850. ↗
  12. Nizar Habash, Owen Rambow, Mona Diab, and Reem Kanjawi-Faraj. 2008. Guidelines for annotation of arabic dialectness. In Proceedings of the LREC Workshop on HLT & NLP within the Arabic world, pages 49–53. find
  13. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735– 1780. ↗
  14. Rie Johnson and Tong Zhang. 2015. Semi-supervised convolutional neural networks for text categorization via region embedding. In Advances in neural information processing systems, pages 919–927. find
  15. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1746–1751.
  16. Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Recurrent convolutional neural networks for text classification. In AAAI, pages 2267–2273. find
  17. Yann LeCun and Yoshua Bengio. 1995. Convolutional networks for images, speech, and time series. In Michael A. Arbib, editor, The Handbook of Brain Theory and Neural Networks, pages 255–258. MIT Press, Cambridge, MA. find
  18. Yun Lei and John HL Hansen. 2011. Dialect classification via text-independent training and testing for arabic, spanish, and chinese. IEEE Transactions on Audio, Speech, and Language Processing, 19(1):85–96. ↗
  19. Shervin Malmasi and Mark Dras. 2014. Arabic native language identification. In Proceedings of the Arabic Natural Language Processing Workshop (EMNLP 2014), pages 180–186. Citeseer.
  20. Shervin Malmasi and Mark Dras. 2015. Language identification using classifier ensembles. In Proceedings of the Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects (LT4VarDial), pages 35–43, Hissar, Bulgaria. find
  21. Shervin Malmasi, Eshrag Refaee, and Mark Dras. 2015. Arabic Dialect Identification using a Parallel Multidialectal Corpus. In Proceedings of the 14th Conference of the Pacific Association for Computational Linguistics (PACLING 2015), pages 209–217, Bali, Indonesia, May. find
  22. Shervin Malmasi, Marcos Zampieri, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, and Jörg Tiedemann. 2016. Discriminating between similar languages and arabic dialect identification: A report on the third dsl shared task. In Proceedings of the 3rd Workshop on Language Technology for Closely Related Languages, Varieties and Dialects (VarDial), Osaka, Japan.
  23. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013a. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, pages 3111–3119. ↗
  24. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751.
  25. Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, and Rabab Ward. 2016. Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(4):694–707. ↗
  26. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning, volume 3, pages 1310–1318. ↗
  27. Radim ehek and Petr Sojka. 2010. Software framework for topic modelling with large corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks. find
  28. Francesco Visin, Kyle Kastner, Kyunghyun Cho, Matteo Matteucci, Aaron Courville, and Yoshua Bengio. 2015. Renet: A recurrent neural network based alternative to convolutional networks. arXiv preprint arXiv:1505.00393. ↗
  29. Omar F Zaidan and Chris Callison-Burch. 2011. The arabic online commentary dataset: an annotated dataset of informal arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 37–41. Association for Computational Linguistics.
  30. Omar F Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗
  31. Marcos Zampieri and Binyam Gebrekidan Gebre. 2012. Automatic identification of language varieties: The case of portuguese. In KONVENS2012-The 11th Conference on Natural Language Processing, pages 233–237. Österreichischen Gesellschaft für Artificial Intelligende (ÖGAI). find
  32. Marcos Zampieri, Liling Tan, Nikola Ljubešić, and Jörg Tiedemann. 2014. A report on the dsl shared task 2014. In Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects (VarDial), pages 58–67, Dublin, Ireland.
  33. Marcos Zampieri, Binyam Gebrekidan Gebre, Hernani Costa, and Josef van Genabith. 2015a. Comparing approaches to the identification of similar languages. In Proceedings of the Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects (LT4VarDial), pages 66–72, Hissar, Bulgaria. ↗
  34. Marcos Zampieri, Liling Tan, Nikola Ljubešić, Jörg Tiedemann, and Preslav Nakov. 2015b. Overview of the dsl shared task 2015. In Proceedings of the Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects (LT4VarDial), pages 1–9, Hissar, Bulgaria.
  35. Matthew D. Zeiler. 2012. ADADELTA: An adaptive learning rate method. CoRR, abs/1212.5701. ↗
  36. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, pages 649–657. ↗
  37. Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In The 54th Annual Meeting of the Association for Computational Linguistics, page 207.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/guggilla-2016-discrimination-between-similar-languages-varieties/