Elamur Research
☰
home / papers / 2017 / Findings of the VarDial Evaluation Campaign 2017

Findings of the VarDial Evaluation Campaign 2017

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W17-1201.pdf

Summary

This paper reports on the VarDial 2017 evaluation campaign, held with the fourth VarDial workshop at EACL 2017. It grew out of the earlier DSL shared tasks and broadened them into four separate challenges:

  • DSL: telling apart closely related languages and national varieties in short news excerpts.
  • ADI: identifying Arabic dialects from speech transcripts plus acoustic features.
  • GDI: classifying Swiss German dialect transcripts, with no audio provided.
  • CLP: dependency parsing of a target language that has no treebank, using treebanks of a closely related source language.

For DSL, the organizers built DSLCC v4.0 following the method of Tan et al. Teams could compete in a closed track (organizer data only) or an open track (any data), with up to three runs in each.

Overall, 26 teams registered, 19 submitted runs, and 15 wrote system papers. All CLP entries relied on annotation projection rather than model transfer. CUNI won CLP clearly on LAS, and for Slovak the two best systems even beat the fully supervised upper bound.

The campaign matters because it offers shared datasets and a common evaluation setup for low-resource, closely related language varieties. This makes it directly relevant to Slavic and other neighbouring-language NLP work.

Key points

  • DSLCC v4.0 covers 14 languages and varieties, including Bosnian/Croatian/Serbian, Malay/Indonesian, Persian/Dari, two French, two Portuguese, and three Spanish varieties.
  • Each variety has 22,000 excerpts: 18k for training, 2k for development, and 2k for testing. This gives about 8.6M training tokens and roughly 0.55M test tokens.
  • ADI covers Egyptian, Gulf, Levantine, North-African, and MSA. GDI covers the Basel, Bern, Lucerne, and Zurich dialect areas.
  • CLP language pairs were Croatian←Slovenian, Slovak←Czech, and Norwegian←Danish/Swedish. Only three teams took part: CUNI, Helsinki-CLP, and tubasfs.
  • CUNI's LAS scores were 60.70 for Croatian, 70.21 for Norwegian, and 78.12 for Slovak. For Croatian and Norwegian, cross-lingual parsing still trails supervised parsing by about 8 LAS points.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Computing & GPUs

Topics: Treebanks

Cited by

116 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 114 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zampieri2017findings,
  title = {{Findings of the VarDial Evaluation Campaign 2017}},
  author = {Zampieri, Marcos and Malmasi, Shervin and Ljubešić, Nikola and Nakov, Preslav and Ali, Ahmed and Tiedemann, Jörg and Scherrer, Yves and Aepli, Noëmi},
  booktitle = {VarDial 2017},
  year = {2017},
  url = {https://aclanthology.org/W17-1201/}
}

References

62 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marcelo Criscuolo and Sandra Aluisio. 2017. Discriminating between similar languages with word-level convolutional neural networks. In Proceedings of the VarDial Workshop. find
  2. Kheireddine Abainia, Siham Ouamour, and Halim Sayoud. 2016. Effective Language Identification of Forum Texts Based on Statistical Approaches. Information Processing & Management, 52(4):491–512. find
  3. Eugen Dieth. 1986. Schwyzertütschi Dialäktschrift. Sauerländer, Aarau, 2 edition. find
  4. Željko Agić, Danijela Merkler, and Daša Berović. 2012. Slovene-Croatian treebank transfer using bilingual lexicon improves Croatian dependency parsing. In Proceedings of IS-LTC. find
  5. Pablo Gamallo, Jose Ramon Pichel, and Iñaki Alegria. 2017. A method based on perplexity for similar languages discrimination. In Proceedings of the Var-Dial Workshop. find
  6. Ahmed Ali, Yifan Zhang, Patrick Cardinal, Najim Da-hak, Stephan Vogel, and Jim Glass. 2014a. A complete Kaldi recipe for building Arabic speech recognition systems. In Proceedings of SLT. find
  7. Binyam Gebrekidan Gebre, Marcos Zampieri, Peter Wittenburg, and Tom Heskes. 2013. Improving native language identification with TF-IDF weighting. In Proceedings of the BEA workshop. find
  8. Ahmed Ali, Yifan Zhang, and Stephan Vogel. 2014b. QCRI Advanced Transcription System (QATS). In Proceedings of SLT. find
  9. Helena Gómez-Adorno, Ilia Markov, Jorge Baptista, Grigori Sidorov, and David Pinto. 2017. Discriminating between similar languages using a combination of typed and untyped character n-grams and words. In Proceedings of the VarDial Workshop. find
  10. Ahmed Ali, Najim Dehak, Patrick Cardinal, Sameer Khurana, Sree Harsha Yella, James Glass, Peter Bell, and Steve Renals. 2016. Automatic dialect detection in Arabic broadcast speech. In Proceedings of INTERSPEECH. find
  11. Cyril Goutte and Serge Léger. 2015. Experiments in discriminating similar languages. In Proceedings of the LT4VarDial Workshop. find
  12. Mohamad Hasan Bahari, Najim Dehak, Lukas Burget, Ahmed Ali, Jim Glass, et al. 2014. Non-negative factor analysis for GMM weight adaptation. IEEE Transactions on Audio Speech and Language Processing. find
  13. Cyril Goutte, Serge Léger, and Marine Carpuat. 2013. Feature space selection and combination for native language identification. In Proceedings of the BEA Workshop. find
  14. Cyril Goutte, Serge Léger, and Marine Carpuat. 2014. The NRC system for discriminating similar languages. In Proceedings of the VarDial Workshop. find
  15. Marco Lui and Paul Cook. 2013. Classifying English documents by national dialect. In Proceedings of ALTA. find
  16. Cyril Goutte, Serge Léger, Shervin Malmasi, and Marcos Zampieri. 2016. Discriminating similar languages: Evaluations and explorations. In Proceedings of LREC. find
  17. Marco Lui. 2014. Generalized language identification. Ph.D. thesis, University of Melbourne. find
  18. Shervin Malmasi and Mark Dras. 2015a. Automatic language identification for Persian and Dari texts. In Proceedings of PACLING. find
  19. Bo Han, Afshin Rahimi, Leon Derczynski, and Timothy Baldwin. 2016. Twitter geolocation prediction shared task of the 2016 workshop on noisy user-generated text. In Proceedings of the W-NUT Workshop. find
  20. Shervin Malmasi and Mark Dras. 2015b. Language identification using classifier ensembles. In Proceedings of the VarDial Workshop. find
  21. Shervin Malmasi and Marcos Zampieri. 2017a. Arabic dialect identification using iVectors and ASR transcripts. In Proceedings of the VarDial Workshop. find
  22. Abualsoud Hanani, Aziz Qaroush, and Stephen Taylor. 2017. Identifying dialects with textual and acoustic cues. In Proceedings of the VarDial Workshop. find
  23. Shervin Malmasi and Marcos Zampieri. 2017b. German dialect identification in interview transcriptions. In Proceedings of the VarDial Workshop. find
  24. Nora Hollenstein and Noëmi Aepli. 2015. A resource for natural language processing of Swiss German dialects. In Proceedings of GSCL. find
  25. Shervin Malmasi, Marcos Zampieri, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, and Jörg Tiedemann. 2016. Discriminating between similar languages and Arabic dialect identification: A report on the third DSL shared task. In Proceedings of the Var-Dial Workshop. find our card
  26. Rebecca Hwa, Philip Resnik, Amy Weinberg, Clara Cabezas, and Okan Kolak. 2005. Bootstrapping parsers via syntactic projection across parallel texts. Natural Language Engineering, 11(3):311–325. find
  27. Radu Tudor Ionescu and Andrei Butnaru. 2017. Learning to identify Arabic and German dialects using multiple kernels. In Proceedings of the VarDial Workshop. find
  28. Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, Claudia Bedini, Núria Bertomeu Castelló, and Jungmee Lee. 2013. Universal dependency annotation for multilingual parsing. In Proceedings of AC… find
  29. Scott Jarvis, Yves Bestgen, and Steve Pepper. 2013. Maximizing classification accuracy in native language identification. In Proceedings of the BEA Workshop. find
  30. Maria Medvedeva, Martin Kroon, and Barbara Plank. 2017. When sparse traditional models outperform dense neural networks: the curious case of discriminating between similar languages. In Proceedings of the VarDial Workshop. find
  31. Tommi Jauhiainen, Heidi Jauhiainen, and Krister Lindén. 2015. Discriminating Similar Languages with Token-based Backoff. In Proceedings of the LT4VarDial Workshop. find
  32. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, et al. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of LREC. find our card
  33. Tommi Jauhiainen, Krister Lindén, and Heidi Jauhiainen. 2016. HeLI, a word-based backoff method for language identification. In Proceedings of the VarDial Workshop. find
  34. Tommi Jauhiainen, Krister Lindén, and Heidi Jauhiainen. 2017. Evaluating HeLI with non-linear mappings. In Proceedings of the VarDial Workshop. find
  35. Will Radford and Matthias Gallé. 2016. Discriminating between similar languages in Twitter using label propagation. arXiv preprint arXiv:1607.05408. — [arXiv:1607.05408](https://arxiv.org/abs/1607.05408) find
  36. Sameer Khurana and Ahmed Ali. 2016. QCRI advanced transcription system (QATS) for the Arabic Multi-Dialect Broadcast Media Recognition: MGB-2 Challenge. In Proceedings of SLT. find
  37. Stephen E. Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at TREC-3. In Proceedings of TREC. find our card
  38. Adrian Leemann, Marie-José Kolly, Ross Purves, David Britain, and Elvira Glaser. 2016. Crowdsourcing language change with smartphone applications. PLOS ONE, 11(1):1–25. find
  39. Rudolf Rosa, Daniel Zeman, David Marec̆ek, and Zdenĕk Z̆abokrtský. 2017. Slavic forest, Norwegian wood. In Proceedings of the VarDial Workshop. find
  40. Tanja Samardžić, Yves Scherrer, and Elvira Glaser. 2016. ArchiMob – a corpus of spoken Swiss German. In Proceedings of LREC. find
  41. Nikola Ljubešić and Denis Kranjčić. 2015. Discriminating between closely related languages on Twitter. Informatica, 39(1). find
  42. Samantha Wray and Ahmed Ali. 2015. Crowdsource a little to label a lot: Labeling a speech corpus of dialectal Arabic. In Proceedings of INTERSPEECH. find
  43. Upendra Sapkota, Steven Bethard, Manuel Montes-y Gómez, and Thamar Solorio. 2015. Not all character n-grams are created equal: A study in authorship attribution. In Proceedings of NAACL. find
  44. Fan Xu, Mingwen Wang, and Maoxi Li. 2016. Sentence-level dialects identification in the Greater China region. International Journal on Natural Language Computing (IJNLC), 5(6). find
  45. Carolina Scarton, Liling Tan, and Lucia Specia. 2015. USHEF and USAAR-USHEF participation in the WMT15 quality estimation shared task. In Proceedings of WMT. find
  46. Marcos Zampieri, Binyam Gebrekidan Gebre, and Sascha Diwersy. N-gram language models and POS distribution for the identification of Spanish varieties. find
  47. Yves Scherrer and Owen Rambow. 2010. Word-based dialect identification with georeferenced rules. In Proceedings of EMNLP. find
  48. Marcos Zampieri, Liling Tan, Nikola Ljubešić, and Jörg Tiedemann. 2014. A report on the DSL shared task 2014. In Proceedings of the VarDial Workshop. find
  49. Yves Scherrer and Philipp Stoeckle. 2016. A quantitative approach to Swiss German – dialectometric analyses and comparisons of linguistic levels. Di-alectologia et Geolinguistica, 24(1):92–125. find
  50. Marcos Zampieri, Binyam Gebrekidan Gebre, Hernani Costa, and Josef van Genabith. 2015a. Comparing approaches to the identification of similar languages. In Proceedings of the VarDial Workshop. find
  51. Helmut Schmid. 1994. Probabilistic part-of-speech tagging using decision trees. In Proceedings of NeMLaP. find
  52. Marcos Zampieri, Liling Tan, Nikola Ljubešić, Jörg Tiedemann, and Preslav Nakov. 2015b. Overview of the DSL shared task 2015. In Proceedings of the LT4VarDial Workshop. find
  53. Milan Straka, Jan Hajič, and Straková. 2016. UD-Pipe: Trainable pipeline for processing CoNLL-U files performing tokenization, morphological analysis, POS tagging and parsing. In Proceedings of LREC. find
  54. Arkaitz Zubiaga, Inaki San Vicente, Pablo Gamallo, José Ramom Pichel, Inaki Alegria, Nora Aranberri, Aitzol Ezeiza, and Vıctor Fresno. 2014. Overview of TweetLID: Tweet language identification at SE-PLN 2014. In Proceedings of the TweetLID Workshop. find
  55. Oscar Täckström, Ryan McDonald, and Jakob Uszkoreit. 2012. Cross-lingual word clusters for direct transfer of linguistic structure. In Proceedings of NAACL. find our card
  56. Liling Tan, Marcos Zampieri, Nikola Ljubešić, and Jörg Tiedemann. 2014. Merging comparable data sources for the discrimination of similar languages: The DSL corpus collection. In Proceedings of the BUCC Workshop. find
  57. Jörg Tiedemann, Željko Agić, and Joakim Nivre. 2014. Treebank translation for cross-lingual parser induction. In Proceedings of CoNLL. find
  58. Jörg Tiedemann. 2012. Parallel Data, Tools and Interfaces in OPUS. In Proceedings of LREC. find
  59. Jörg Tiedemann. 2014. Rediscovering annotation projection for cross-lingual parser induction. In Proceedings of COLING. find
  60. Jörg Tiedemann. 2015. Cross-lingual dependency parsing with universal dependencies and predicted PoS labels. In Proceedings of Depling. find
  61. Jörg Tiedemann. 2017. Cross-lingual dependency parsing for closely related languages - Helsinki’s submission to VarDial 2017. In Proceedings of the VarDial Workshop. find
  62. Lan Wang, Masahiro Tanaka, and Hayato Yamana. 2016. What is your Mother Tongue?: Improving Chinese native language identification by cleaning noisy data and adopting BM25. In Proceedings of ICBDA. find