Elamur Research
☰
home / papers / 2016 / Discriminating between Similar Languages and Arabic Dialect Identification: A Report on the Third DSL Shared Task

Discriminating between Similar Languages and Arabic Dialect Identification: A Report on the Third DSL Shared Task

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W16-4801.pdf

Summary

The third DSL shared task, held at VarDial 2016 (co-located with COLING 2016), tested how well automatic systems can tell apart closely related languages, language varieties and dialects. It had two subtasks.

  • Subtask 1 used a new release of the DSL corpus collection, DSLCC v3.0. It contains short news texts in twelve languages and varieties, including Hexagonal and Canadian French for the first time. Systems were tested on in-domain news and on out-of-domain Twitter data.
  • Subtask 2 introduced Arabic dialect identification on speech transcripts from the Ali et al. (2016) corpus.

Both subtasks had a closed track (organizer data only) and an open track (any external data). Participation reached a record level for the series.

Across both subtasks, the strongest systems relied on character n-grams of high order combined with conventional classifiers such as SVM and logistic regression. Ensembles and hierarchical classifiers also did well. Most deep learning entries fell behind these simpler methods. The tweet test sets proved harder than news, and Arabic dialects remained difficult, with top F1 scores near 0.5.

The report gives a common benchmark and practical guidance for language identification among close varieties.

Key points

  • 37 teams registered, 24 submitted results and 20 wrote system papers, up from 8 submissions in 2014 and 10 in 2015.
  • DSLCC v3.0 provides 20,000 instances per variety: 18,000 for training and 2,000 for development. Test set A has 1,000 news instances per variety. Out-of-domain sets B1 and B2 each cover 100 Twitter users per variety. Mexican Spanish was also added, while easy pairs such as Czech–Slovak and Bulgarian–Macedonian were dropped.
  • Besides MSA, the Arabic data includes four regional varieties: Gulf (GLF), Egyptian (EGY), North African (NOR) and Levantine (LAV).
  • Arabic closed-track results were led by MAZA, an ensemble of word and character n-grams (F1 0.513). The best open-track result came from ASIREM (F1 0.527), which added 18,000 social media documents collected by native speakers.
  • Character n-grams beat word n-grams, especially orders 5–7. Deep models were costly to train, with one team reporting ten days of training.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Computing & GPUs

Cited by

122 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 118 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{malmasi2016discriminating,
  title = {{Discriminating between Similar Languages and Arabic Dialect Identification: A Report on the Third DSL Shared Task}},
  author = {Malmasi, Shervin and Zampieri, Marcos and Ljubešić, Nikola and Nakov, Preslav and Ali, Ahmed and Tiedemann, Jörg},
  booktitle = {VarDial 2016},
  year = {2016},
  url = {https://aclanthology.org/W16-4801/}
}

References

61 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Judit Ács, László Grad-Gyenge, and Thiago Bruno Rodrigues de Rezende Oliveira. 2015. A Two-level Classifier for Discriminating Similar Languages. In Proceedings of the LT4VarDial Workshop. find
  2. Wafia Adouane, Nasredine Semmar, Richard Johansson, and Victoria Bobicev. 2016. ASIREM Participation at the Discriminating Similar Languages Shared Task 2016. In Proceedings of the VarDial Workshop. find
  3. Ahmed Ali, Najim Dehak, Patrick Cardinal, Sameer Khurana, Sree Harsha Yella, James Glass, Peter Bell, and Steve Renals. 2016. Automatic Dialect Detection in Arabic Broadcast Speech. In Proceedings of Interspeech. find
  4. Areej Alshutayri, Eric Atwell, Abdulrahman Alosaimy, James Dickins, Michale Ingleby, and Janet Watson. 2016. Arabic Language WEKA-Based Dialect Classifier for Arabic Automatic Speech Recognition Transcripts. In Proceedings of the VarDial Workshop. find
  5. Timothy Baldwin and Marco Lui. 2010. Multilingual Language Identification: ALTW 2010 Shared Task Data. In Proceedings of ALTA. find
  6. Adrien Barbaresi. 2016. An Unsupervised Morphological Criterion for Discriminating Similar Languages. In Proceedings of the VarDial Workshop. find
  7. Yonatan Belinkov and James Glass. 2016. A Character-level Convolutional Neural Network for Distinguishing Similar Languages and Dialects. In Proceedings of the VarDial Workshop. find
  8. Johannes Bjerva. 2016. Byte-based Language Identification with Deep Convolutional Networks. In Proceedings of the VarDial Workshop. find
  9. Victoria Bobicev. 2015. LT4VarDial Workshop. Discriminating between Similar Languages Using PPM. In Proceedings of the Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. 2016. Findings of the 2016 Conference on Machine Translation. In Proceedings of WMT. find
  10. Çağrı Çöltekin and Taraka Rama. 2016. Discriminating Similar Languages with Linear SVMs and Neural Networks. In Proceedings of the VarDial Workshop. find
  11. Andre Cianflone and Leila Kosseim. 2016. N-gram and Neural Language Models for Discriminating Similar Languages. In Proceedings of the VarDial Workshop. find
  12. Alina Maria Ciobanu, Sergiu Nisioi, and Liviu P. Dinu. 2016. Vanilla Classifiers for Distinguishing between Similar Languages. In Proceedings of the VarDial Workshop. find
  13. Kareem Darwish, Hassan Sajjad, and Hamdy Mubarak. 2014. Verifiably Effective Arabic Dialect Identification. In Proceedings of EMNLP. find
  14. Mohamed Eldesouki, Fahim Dalvi, Hassan Sajjad, and Kareem Darwish. 2016. QCRI @ DSL 2016: Spoken Arabic Dialect Identification Using Textual Features. In Proceedings of the VarDial Workshop. find
  15. Heba Elfardy and Mona Diab. 2014. Sentence Level Dialect Identification in Arabic. In Proceedings of ACL. find
  16. Raül Fabra-Boluda, Francisco Rangel, and Paolo Rosso. 2015. NLEL UPV Autoritas participation at Discrimination between Similar Languages (DSL) 2015 shared task. In Proceedings of the LT4VarDial Workshop. find
  17. Hector-Hugo Franco-Penya and Liliana Mamani Sanchez. 2016. Tuning Bayes Baseline for Dialect Detection. In Proceedings of the VarDial Workshop. find
  18. Marc Franco-Salvador, Paolo Rosso, and Francisco Rangel. 2015. Distributed Representations of Words and Documents for Discriminating Similar Languages. In Proceedings of the LT4VarDial Workshop. find
  19. Pablo Gamallo, Iñaki Alegria, and José Ramom Pichel. 2016. Comparing two Basic Methods for Discriminating Between Similar Languages and Varieties. In Proceedings of the VarDial Workshop. find
  20. Cyril Goutte and Serge Léger. 2015. Experiments in Discriminating Similar Languages. In Proceedings of the LT4VarDial Workshop. find
  21. Cyril Goutte and Serge Léger. 2016. Advances in Ngram-based Discrimination of Similar Languages. In Proceedings of the VarDial Workshop. find
  22. Cyril Goutte, Serge Léger, and Marine Carpuat. 2014. The NRC System for Discriminating Similar Languages. In Proceedings of the VarDial Workshop. find
  23. Cyril Goutte, Serge Léger, Shervin Malmasi, and Marcos Zampieri. 2016. Discriminating Similar Languages: Evaluations and Explorations. In Proceedings of LREC. find
  24. Cyril Grouin, Dominic Forest, Lyne Da Sylva, Patrick Paroubek, and Pierre Zweigenbaum. 2010. Présentation et Résultats du Défi Fouille de Texte DEFT2010 Où et Quand un Article de Presse a-t-il Été Écrit? In Proceedings of DEFT. find
  25. Chinnappa Guggilla. 2016. Discrimination between Similar Languages, Varieties and Dialects using CNN and LSTM-based Deep Neural Networks. In Proceedings of the VarDial Workshop. find
  26. Bo Han, Afshin Rahimi, Leon Derczynski, and Timothy Baldwin. 2016. Twitter Geolocation Prediction Shared Task of the 2016 Workshop on Noisy User-generated Text. In Proceedings of the W-NUT Workshop. find
  27. Abualsoud Hanani, Aziz Qaroush, and Stephen Taylor. 2016. Classifying ASR Transcriptions According to Arabic Dialect. In Proceedings of the VarDial Workshop. find
  28. Ondr̆ej Herman, Vit Suchomel, Vı́t Baisa, and Pavel Pavel Rychlý. 2016. DSL Shared task 2016: Perfect Is The Enemy of Good Language Discrimination Through Expectation–Maximization and Chunk-based Language Model. In Proceedings of the VarDial Workshop. find
  29. Chu-ren Huang and Lung-hao Lee. 2008. Contrastive Approach towards Text Source Classification based on Top-Bag-of-Word Similarity. In Proceedings of PACLIC. find
  30. Radu Tudor Ionescu and Marius Popescu. 2016. UnibucKernel: An Approach for Arabic Dialect Identification based on Multiple String Kernels. In Proceedings of the VarDial Workshop. find
  31. Radu Tudor Ionescu, Marius Popescu, and Aoife Cahill. 2016. String Kernels for Native Language Identification: Insights from Behind the Curtains. Computational Linguistics, 43(3):491–525. find
  32. Tommi Jauhiainen, Heidi Jauhiainen, and Krister Lindén. 2015. Discriminating Similar Languages with Token-based Backoff. In Proceedings of the LT4VarDial Workshop. find
  33. Tommi Jauhiainen, Krister Lindén, and Heidi Jauhiainen. 2016. HeLI, a Word-Based Backoff Method for Language Identification. In Proceedings of the VarDial Workshop. find
  34. Ben King, Dragomir Radev, and Steven Abney. 2014. Experiments in Sentence Language Identification with Groups of Similar Languages. In Proceedings of the VarDial Workshop. find
  35. Nikola Ljubešić and Denis Kranjčić. 2015. Discriminating Between Closely Related Languages on Twitter. Informatica, 39(1). find
  36. Nikola Ljubešić, Nives Mikelic, and Damir Boras. 2007. Language identification: How to distinguish similar languages? In Proceedings of ITI. find
  37. Nikola Ljubešić, Darja Fišer, and Tomaž Erjavec. 2014. TweetCaT: a Tool for Building Twitter Corpora of Smaller Languages. In Proceedings of LREC. find
  38. Nikola Ljubešić, Tanja Samardžić, and Curdin Derungs. 2016. TweetGeo – A Tool for Collecting, Processing and Analysing Geo-encoded Data. In Proceedings of COLING. find
  39. Marco Lui and Paul Cook. 2013. Classifying English Documents by National Dialect. In Proceedings of ALTA. find
  40. Marco Lui, Ned Letcher, Oliver Adams, Long Duong, Paul Cook, and Timothy Baldwin. 2014. Exploring Methods and Resources for Discriminating Similar Languages. In Proceedings of VarDial. find
  41. Shervin Malmasi and Mark Dras. 2015a. Automatic Language Identification for Persian and Dari Texts. In Proceedings of PACLING. find
  42. Shervin Malmasi and Mark Dras. 2015b. Language Identification using Classifier Ensembles. In Proceedings of the VarDial Workshop. find
  43. Shervin Malmasi and Marcos Zampieri. 2016. Arabic Dialect Identification in Speech Transcripts. In Proceedings of the VarDial Workshop. find
  44. Paul McNamee. 2016. Language and Dialect Discrimination Using Compression-Inspired Language Models. In Proceedings of the VarDial Workshop. find
  45. Jordi Porta and José-Luis Sancho. 2014. Using Maximum Entropy Models to Discriminate between Similar Languages and Varieties. In Proceedings of the VarDial Workshop. find
  46. Matthew Purver. 2014. A Simple Baseline for Discriminating Similar Language. In Proceedings of VarDial Workshop. find
  47. Bali Ranaivo-Malançon. 2006. Automatic identification of close languages - case study: Malay and Indonesian. ECTI Transactions on Computer and Information Technology, 2:126–134. find
  48. Abdulhadi Shoufan and Sumaya Al-Ameri. 2015. Natural Language Processing for Dialectical Arabic: A Survey. In Proceedings of the Arabic NLP Workshop. find
  49. Thamar Solorio, Elizabeth Blair, Suraj Maharjan, Steven Bethard, Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Alison Chang, and Pascale Fung. 2014. Overview for the First Shared Task on Language Identification in Code-Switched Data. In Proceedings of the CodeSwitch Workshop. find
  50. Liling Tan, Marcos Zampieri, Nikola Ljubešić, and Jörg Tiedemann. 2014. Merging Comparable Data Sources for the Discrimination of Similar Languages: The DSL Corpus Collection. In Proceedings of the BUCC Workshop. find
  51. Christoph Tillmann, Saab Mansour, and Yaser Al-Onaizan. 2014. Improved Sentence-Level Arabic Dialect Classification. In Proceedings of the VarDial Workshop. find
  52. Samantha Wray and Ahmed Ali. 2015. Crowdsource a little to label a lot: labeling a speech corpus of dialectal Arabic. In Proceedings of Interspeech. find
  53. Omar F. Zaidan and Chris Callison-Burch. 2011. Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of ACL-HLT. find
  54. Omar F Zaidan and Chris Callison-Burch. 2014. Arabic Dialect Identification. Computational Linguistics, 40(1):171–202. find our card
  55. Marcos Zampieri and Binyam Gebrekidan Gebre. 2012. Automatic Identification of Language Varieties: The Case of Portuguese. In Proceedings of KONVENS. find
  56. Marcos Zampieri, Liling Tan, Nikola Ljubešić, and Jörg Tiedemann. 2014. A Report on the DSL Shared Task 2014. In Proceedings of the VarDial Workshop. find
  57. Marcos Zampieri, Binyam Gebrekidan Gebre, Hernani Costa, and Josef van Genabith. 2015a. Comparing Approaches to the Identification of Similar Languages. In Proceedings of the VarDial Workshop. find
  58. Marcos Zampieri, Liling Tan, Nikola Ljubešić, Jörg Tiedemann, and Preslav Nakov. 2015b. Overview of the DSL Shared Task 2015. In Proceedings of the LT4VarDial Workshop. find
  59. Ayah Zirikly, Bart Desmet, and Mona Diab. 2016. The GW/LT3 VarDial 2016 Shared Task System for Dialects and Similar Languages Detection. In Proceedings of the VarDial Workshop. find
  60. Arkaitz Zubiaga, Inaki San Vicente, Pablo Gamallo, José Ramom Pichel, Inaki Alegria, Nora Aranberri, Aitzol Ezeiza, and Vıctor Fresno. 2014. Overview of TweetLID: Tweet Language Identification at SEPLN 2014. In Proceedings of the TweetLID Workshop. find
  61. Arkaitz Zubiaga, Iñaki San Vicente, Pablo Gamallo, José Ramom Pichel, Iñaki Alegria, Nora Aranberri, Aitzol Ezeiza, and Vı́ctor Fresno. 2015. TweetLID: A Benchmark for Tweet Language Identification. Language Resources and Evaluation, pages 1–38. find