Elamur Research⌕
☰
home / papers / 2018 / Language Identification and Morphosyntactic Tagging: The Second VarDial Evaluation Campaign

Language Identification and Morphosyntactic Tagging: The Second VarDial Evaluation Campaign

Summary

This report describes the second VarDial Evaluation Campaign, held at the VarDial 2018 workshop alongside COLING 2018. The campaign provides shared benchmarks for NLP on closely related languages, language varieties and dialects. It had five shared tasks.

Two tasks returned from earlier years:

  • Arabic Dialect Identification (ADI), which now included phonetic features alongside the lexical and acoustic inputs.
  • Swiss German Dialect Identification (GDI), which added a "surprise" dialect that had no training data.

Three tasks were new:

  • Morphosyntactic Tagging of Tweets (MTT), focused on tweets written in South-Slavic languages.
  • Telling Dutch apart from Flemish in subtitle text (DFS).
  • Indo-Aryan Language Identification (ILI).

The organizers supplied the datasets, ran the evaluations and summarized each team's approach. A record 54 teams registered, 24 submitted runs, and 22 wrote system description papers. The approaches varied widely, from character n-gram models and SVMs to neural networks. In ILI, the SUKI system took first place. The campaign matters because it released new resources for under-studied varieties, including the first public corpus for several Indo-Aryan languages. It also offers standardized comparisons of methods on fine-grained language discrimination.

Key points

  • ADI: the organizers reused the 2017 training and development data. The test set secretly combined in-domain broadcast speech with out-of-domain YouTube clips of 5–30 seconds, and the two parts were scored separately.
  • GDI: the task covered Basel, Bern, Lucerne and Zurich dialects. It offered a four-way track and a five-way track that included the surprise dialect.
  • MTT and DFS: MTT asked teams to tag Slovene, Croatian and Serbian tweets with 900+ morphosyntactic labels. DFS provided over 50,000 subtitle phrases labelled as Netherlandic or Flemish Dutch.
  • ILI data: the corpus has about 15,000 sentences per language for Hindi, Braj Bhasha, Awadhi, Bhojpuri and Magahi, mostly from literature. The texts were digitized with Google OCR and then corrected by native speakers.
  • ILI results: there were eight submissions. SUKI won with a macro F1 of 0.958, using HeLI with adaptive character n-gram models (n = 1–6). The lowest score was 0.819 (LaMa).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

84 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 82 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zampieri2018language,
  title = {{Language Identification and Morphosyntactic Tagging: The Second VarDial Evaluation Campaign}},
  author = {Zampieri, Marcos and Malmasi, Shervin and Nakov, Preslav and Ali, Ahmed and Shon, Suwon and Glass, James and Scherrer, Yves and Samardžić, Tanja and Ljubešić, Nikola and Tiedemann, Jörg and Lee, Chris van der and Grondelaers, Stefan and Oostdijk, Nelleke and Speelman, Dirk and Bosch, Antal van den and Kumar, Ritesh and Lahiri, Bornini and Jain, Mayank},
  booktitle = {Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018)},
  year = {2018},
  url = {https://aclanthology.org/W18-3901/}
}

References

59 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Ali, Stephan Vogel, and Steve Renals. 2017. Speech recognition challenge in the wild: Arabic MGB-3. In Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). find our card
  2. Mohamed Ali. 2018a. Character level convolutional neural network for Arabic dialect identification. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  3. Mohamed Ali. 2018b. Character level convolutional neural network for German dialect identification. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  4. Mohamed Ali. 2018c. Character level convolutional neural network for Indo-Aryan language identification. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  5. Adrien Barbaresi. 2017. Discriminating between similar languages using weighted subword features. In Proceedings of the VarDial Workshop (VarDial). find
  6. Adrien Barbaresi. 2018. Computationally efficient discrimination between language varieties with large feature vectors and regularized classifiers. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  7. Fernando Benites, Ralf Grubenmann, Pius von Däniken, Dirk von Grünigen, Jan Deriu, and Mark Cieliebak. 2018. Twist Bytes – German dialect identification with data mining optimization. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  8. Yves Bestgen. 2017. Improving the character ngram model for the DSL task with BM25 weighting and less frequently used feature sets. In Proceedings of the VarDial Workshop (VarDial). find
  9. Andrei M. Butnaru and Radu Ionescu. 2018. UnibucKernel Reloaded: First place in Arabic dialect identification for the second year in a row. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  10. Çağrı Çöltekin and Taraka Rama. 2016. Discriminating similar languages with linear SVMs and neural networks. In Proceedings of the VarDial Workshop (VarDial). find
  11. Çağrı Çöltekin and Taraka Rama. 2017. Tübingen system in VarDial 2017 shared task: Experiments with language identification and cross-lingual parsing. In Proceedings of the VarDial Workshop (VarDial). find
  12. Çağrı Çöltekin, Taraka Rama, and Verena Blaschke. 2018. Tübingen-Oslo team at the VarDial 2018 evaluation campaign: An analysis of n-gram features in language variety identification. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  13. Narayan Choudhary and Girish Nath Jha. 2011. Creating multilingual parallel corpora in Indian languages. In Proceedings of LTC. find
  14. Alina Maria Ciobanu, Shervin Malmasi, and Liviu P. Dinu. 2018a. German dialect identification using classifier ensembles. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  15. Alina Maria Ciobanu, Marcos Zampieri, Shervin Malmasi, Santanu Pal, and Liviu P. Dinu. 2018b. Discriminating between Indo-Aryan languages using SVM ensembles. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  16. Eugen Dieth. 1986. Schwyzertütschi Dialäktschrift. 2 edition. find
  17. Steven Du and Yuan Yuan Wang. 2018. STEVENDU2018’s system in VarDial 2018: Discriminating between Dutch and Flemish in subtitles. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  18. Tomaž Erjavec, Nikola Ljubešić, and Nataša Logar. 2015. The slWaC corpus of the Slovene web. Informatica, 39(1):35–42. find
  19. Tomaž Erjavec, Darja Fišer, Jaka Čibej, Špela Arhar Holdt, Nikola Ljubešić, and Katja Zupan. 2017. CMC training corpus Janes-Tag 2.0. Slovenian language resource repository CLARIN.SI. find
  20. Divyanshu Gupta, Gourav Dhakad, Jayprakash Gupta, and Anil Kumar Singh. 2018. IIT (BHU) System for Indo-Aryan language identification (ILI) at VarDial 2018. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  21. Radu Tudor Ionescu and Andrei Butnaru. 2017. Learning to identify Arabic and German dialects using multiple kernels. In Proceedings of the VarDial Workshop (VarDial). find
  22. Radu Tudor Ionescu and Marius Popescu. 2016. UnibucKernel: An approach for Arabic dialect identification based on multiple string kernels. In Proceedings of the VarDial Workshop (VarDial). find
  23. Tommi Jauhiainen, Heidi Jauhiainen, and Krister Lindén. 2018a. HeLI-based experiments in discriminating between Dutch and Flemish subtitles. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  24. Tommi Jauhiainen, Heidi Jauhiainen, and Krister Lindén. 2018b. HeLI-based experiments in Swiss German dialect identification. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  25. Tommi Jauhiainen, Heidi Jauhiainen, and Krister Lindén. 2018c. Iterative language model adaptation for Indo-Aryan language identification. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  26. Tommi Jauhiainen, Marco Lui, Marcos Zampieri, Timothy Baldwin, and Krister Lindén. 2018d. Automatic language identification in texts: A survey. arXiv preprint arXiv:1804.08186. find our card
  27. Sameer Khurana and Ahmed Ali. 2016. QCRI advanced transcription system (QATS) for the Arabic Multi-Dialect Broadcast Media Recognition: MGB-2 Challenge. In Proceedings of SLT. find
  28. Tim Kreutz and Walter Daelemans. 2018. Exploring classifier combinations for language variety identification. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  29. Martin Kroon, Maria Medvedeva, and Barbara Plank. 2018. When simple n-gram models outperform syntactic approaches: Discriminating between Dutch and Flemish. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  30. Ritesh Kumar, Bornini Lahiri, Deepak Alok, Atul Kr. Ojha, Mayank Jain, Abdul Basit, and Yogesh Dawar. 2018. Automatic identification of closely-related Indian languages: Resources and experiments. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC). find
  31. Ritesh Kumar. 2012. Challenges in the development of annotated corpora of computer-mediated communication in Indian languages: A case of Hindi. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC). find
  32. Ritesh Kumar. 2014a. Developing politeness annotated corpus of Hindi blogs. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC). find
  33. Ritesh Kumar. 2014b. Politeness in Online Hindi Texts: Pragmatic and Computational Aspects. Ph.D. thesis, Jawaharlal Nehru University. find
  34. Nikola Ljubešić and Filip Klubička. 2014. {bs,hr,sr}wac - web corpora of Bosnian, Croatian and Serbian. In Proceedings of the 9th Web as Corpus Workshop (WaC-9), pages 29–35. find
  35. Nikola Ljubešić, Tomaž Erjavec, Maja Miličević, and Tanja Samardžić. 2017a. Croatian Twitter training corpus ReLDI-NormTagNER-hr 2.0. Slovenian language resource repository CLARIN.SI. find
  36. Nikola Ljubešić, Tomaž Erjavec, Maja Miličević, and Tanja Samardžić. 2017b. Serbian Twitter training corpus ReLDI-NormTagNER-sr 2.0. Slovenian language resource repository CLARIN.SI. find
  37. Nikola Ljubešić. 2018. Comparing CRF and LSTM performance on the task of morphosyntactic tagging of non-standard varieties of South Slavic languages. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  38. Nikola Ljubešić and Tomaž Erjavec. 2016. Corpus vs. lexicon supervision in morphosyntactic tagging: the case of Slovene. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC). find
  39. Nikola Ljubešić, Filip Klubička, Željko Agić, and Ivo-Pavao Jazbec. 2016. New inflectional lexicons and training corpora for improved morphosyntactic annotation of Croatian and Serbian. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC). find
  40. Shervin Malmasi and Marcos Zampieri. 2016. Arabic dialect identification in speech transcripts. In Proceedings of the VarDial Workshop (VarDial). find
  41. Shervin Malmasi and Marcos Zampieri. 2017a. Arabic dialect identification using iVectors and ASR transcripts. In Proceedings of the VarDial Workshop (VarDial). find
  42. Shervin Malmasi and Marcos Zampieri. 2017b. German dialect identification in interview transcriptions. In Proceedings of the VarDial Workshop (VarDial). find
  43. Shervin Malmasi, Marcos Zampieri, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, and Jörg Tiedemann. 2016. Discriminating between similar languages and Arabic dialect identification: A report on the third DSL shared task. find our card
  44. Pavel Matejka, Petr Schwarz, Jan Cernockỳ, and Pavel Chytil. 2005. Phonotactic language identification using high quality phoneme recognition. In Proc. Interspeech. find
  45. Sara Meftah and Nasredine Semmar. 2018. Using neural transfer learning for morpho-syntactic tagging of South-Slavic languages tweets. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  46. Elise Michon, Minh Quang Pham, Josep Crego, and Jean Senellart. 2018. Neural network architectures for Arabic dialect identification. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  47. Maryam Najafian, Sameer Khurana, Suwon Shon, Ahmed Ali, and James Glass. 2018. Exploiting convolutional neural networks for phonotactic based dialect identification. In IEEE ICASSP, pages 5174–5178. find
  48. Rabee Naser and Abualsoud Hanani. 2018. Birzeit Arabic dialect identication system for the 2018 VarDial challenge. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  49. Tanja Samardžić, Yves Scherrer, and Elvira Glaser. 2016. ArchiMob – a corpus of spoken Swiss German. In Proceedings of LREC. find
  50. Yves Scherrer and Philipp Stoeckle. 2016. A quantitative approach to Swiss German – dialectometric analyses and comparisons of linguistic levels. Dialectologia et Geolinguistica, 24(1):92–125. find
  51. Suwon Shon, Ahmed Ali, and James Glass. 2017. MIT-QCRI Arabic dialect identification system for the 2017 multi-genre broadcast challenge. In IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pages 374–380. find
  52. Suwon Shon, Ahmed Ali, and James Glass. 2018. Convolutional neural network and language embeddings for end-to-end dialect recognition. In Proceedings of the Speaker and Language Recognition Workshop (Odyssey). find
  53. Miikka Silfverberg and Senka Drobac. 2018. Sub-label dependencies for neural morphological tagging – the joint submission of University of Colorado and University of Helsinki for VarDial 2018. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  54. Liling Tan, Marcos Zampieri, Nikola Ljubešic, and Jörg Tiedemann. 2014. Merging comparable data sources for the discrimination of similar languages: The DSL corpus collection. In Proceedings of the BUCC Workshop. find
  55. Chris van der Lee and Antal van den Bosch. 2017. Exploring lexical and syntactic features for language variety identification. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  56. Hans van Halteren and Nelleke Oostdijk. 2018. Identification of differences between Dutch language varieties with the VarDial2018 Dutch-Flemish subtitle data. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  57. Marcos Zampieri, Liling Tan, Nikola Ljubešić, and Jörg Tiedemann. 2014. A report on the DSL shared task 2014. In Proceedings of the VarDial Workshop (VarDial). find our card
  58. Marcos Zampieri, Liling Tan, Nikola Ljubešić, Jörg Tiedemann, and Preslav Nakov. 2015. Overview of the DSL shared task 2015. In Proceedings of the LT4VarDial Workshop (LT4VarDial). find our card
  59. Marcos Zampieri, Shervin Malmasi, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, Jörg Tiedemann, Yves Scherrer, and Noëmi Aepli. 2017. Findings of the VarDial evaluation campaign 2017. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/zampieri-2018-language-identification-morphosyntactic-tagging-second/