Elamur Research⌕
☰
home / papers / 2014 / A Report on the DSL Shared Task 2014

A Report on the DSL Shared Task 2014

Summary

Language identification tools now cover very many languages but still struggle to tell apart closely related pairs like Croatian versus Serbian, or Malay versus Indonesian. They also usually treat national varieties of pluricentric languages as a single class. The DSL shared task, held at the VarDial workshop at COLING 2014, was the first evaluation campaign devoted to this problem. The organizers built the DSL corpus collection from news-domain sources. It covers 13 languages and varieties in six groups, with 18,000 training, 2,000 development and 1,000 test sentences per class, each of at least 20 tokens. Teams could submit closed runs, trained only on DSL data, or open runs using any data. Of 22 registered teams, 8 submitted final results. The best closed system reached 95.71% accuracy on the 11 classes of groups A–E. English was set aside because many of its labels reflected the website a text was taken from rather than its real origin. The strongest systems first predicted the language group and then the specific class, using word and character features.

Key points

  • Six groups: A (bs, hr, sr), B (id, my), C (cz, sk), D (pt-BR, pt-PT), E (es-AR, es-ES), F (en-GB, en-US).
  • Data came from journalistic corpora, chosen because they reflect standard language and limit bias from source, register and topic. The collection is distributed as tab-separated text (sentence, group, language code).
  • Group F was unreliable because of republished and cross-cited articles. Baseline F-measure was only 0.59 for F, compared with 0.84 for E, the next-lowest group.
  • Each team could submit up to three closed and three open runs. Only UMich and UniMelb-NLP submitted open runs.
  • Top systems used Linear SVM, MaxEnt, Naive Bayes and langid.py with word and character n-grams. QMUL reported that word unigrams worked well with a Linear SVM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Computing & GPUs

Cited by

88 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 82 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zampieri2014report,
  title = {{A Report on the DSL Shared Task 2014}},
  author = {Zampieri, Marcos and Tan, Liling and Ljubešić, Nikola and Tiedemann, Jörg},
  booktitle = {VarDial 2014},
  year = {2014},
  doi = {10.3115/v1/w14-5307},
  url = {https://doi.org/10.3115/v1/w14-5307}
}

References

28 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stefanie Anstein. 2013. Computational approaches to the comparison of regional variety corpora : prototyping a semi-automatic system for German. Ph.D. thesis, University of Stuttgart. find
  2. Timothy Baldwin and Marco Lui. 2010. Multilingual language identification: ALTW 2010 shared task data. In Proceedings of Australasian Language Technology Association Workshop, pages 4–7. find
  3. Daniel Blanchard, Joel Tetreault, Derrick Higgins, Aoife Cahill, and Martin Chodorow. 2013. TOEFL11: A Corpus of Non-Native English. Technical report, Educational Testing Service. find
  4. Ralf Brown. 2013. Selecting and weighting n-grams to identify 1100 languages. In Proceedings of the 16th International Conference on Text Speech and Dialogue (TSD2013), Lecture Notes in Artificial Intelligence (LNAI 8082), pages 519–526, Pilsen, Czech Republic. Springer. find
  5. William Cavnar and John Trenkle. 1994. N-gram-based text catogorization. 3rd Symposium on Document Analysis and Information Retrieval (SDAIR-94). find
  6. Jack Chambers and Peter Trudgill. 1998. Dialectology (2nd Edition). Cambridge University Press. find
  7. Michael Clyne. 1992. Pluricentric Languages: Different Norms in Different Nations. CRC Press. find
  8. Sascha Diwersy, Stefan Evert, and Stella Neumann. 2014. A semi-supervised multivariate approach to the study of language variation. Linguistic Variation in Text and Speech, within and across Languages. find
  9. Cyril Goutte, Serge Léger, and Marine Carpuat. 2014. The NRC system for discriminating similar languages. In Proceedings of the 1st Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects (VarDial), Dublin, Ireland. find
  10. Cyril Grouin, Dominic Forest, Lyne Da Sylva, Patrick Paroubek, and Pierre Zweigenbaum. 2010. Présentation et résultats du défi fouille de texte DEFT2010 où et quand un article de presse a-t-il été écrit? Actes du sixième DÉfi Fouille de Textes. find
  11. Chu-ren Huang and Lung-hao Lee. 2008. Contrastive approach towards text source classification based on top-bag-of-word similarity. In Proceedings of PACLIC 2008, pages 404–410. find
  12. Ben King, Dragomir Radev, and Steven Abney. 2014. Experiments in sentence language identification with groups of similar languages. In Proceedings of the 1st Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects (VarDial), Dublin, Ireland. find
  13. Nikola Ljubešić, Nives Mikelic, and Damir Boras. 2007. Language identification: How to distinguish similar languages? In Proceedings of the 29th International Conference on Information Technology Interfaces. find
  14. Marco Lui and Timothy Baldwin. 2012. langid.py: An off-the-shelf language identification tool. In Proceedings of the 50th Meeting of the ACL. find our card
  15. Marco Lui and Paul Cook. 2013. Classifying English documents by national dialect. In Proceedings of Australasian Language Tchnology Workshop, pages 5–15. find
  16. Marco Lui, Ned Letcher, Oliver Adams, Long Duong, Paul Cook, and Timothy Baldwin. 2014. Exploring methods and resources for discriminating similar languages. In Proceedings of the 1st Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects (VarDial), Dublin, Ireland. find
  17. Yves Piersman, Dirk Geeraerts, and Dirk Speelman. 2010. The automatic identification of lexical variation between language varieties. Natural Language Engineering, 16:469–491. find
  18. Jordi Porta and José-Luis Sancho. 2014. Using maximum entropy models to discriminate between similar languages and varieties. In Proceedings of the 1st Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects (VarDial), Dublin, Ireland. find
  19. Matthew Purver. 2014. A simple baseline for discriminating similar language. In Proceedings of the 1st Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects (VarDial), Dublin, Ireland. find
  20. Bali Ranaivo-Malançon. 2006. Automatic identification of close languages - case study: Malay and Indonesian. ECTI Transactions on Computer and Information Technology, 2:126–134. find
  21. Augusto Soares da Silva. 2010. Measuring and parameterizing lexical convergence and divergence between European and Brazilian Portuguese: endo/exogeneousness and foreign and normative influence. Advances in Cognitive Sociolinguistics. find
  22. Liling Tan, Marcos Zampieri, Nikola Ljubešić, and Jörg Tiedemann. 2014. Merging comparable data sources for the discrimination of similar languages: The DSL corpus collection. In Proceedings of The Workshop on Building and Using Comparable Corpora (BUCC), Reykjavik, Iceland. find
  23. Joel Tetreault, Daniel Blanchard, and Aoife Cahill. 2013. A report on the first native language identification shared task. In Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, Atlanta, GA, USA, June. Association for Computational Linguistics. find
  24. Jörg Tiedemann and Nikola Ljubešić. 2012. Efficient discrimination between closely related languages. In Proceedings of COLING 2012, pages 2619–2634, Mumbai, India. find
  25. Omar F Zaidan and Chris Callison-Burch. 2013. Arabic dialect identification. Computational Linguistics. find our card
  26. Marcos Zampieri and Binyam Gebrekidan Gebre. 2012. Automatic identification of language varieties: The case of Portuguese. In Proceedings of KONVENS2012, pages 233–237, Vienna, Austria. find
  27. Marcos Zampieri, Binyam Gebrekidan Gebre, and Sascha Diwersy. 2013. N-gram language models and POS distribution for the identification of Spanish varieties. In Proceedings of TALN2013, pages 580–587, Sable d’Olonne, France. find
  28. Marcos Zampieri. 2013. Using bag-of-words to distinguish similar languages: How efficient are they? In Proceedings of the 14th IEEE International Symposium on Computational Intelligence and Informatics (CINTI2013), pages 37–41, Budapest, Hungary. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/zampieri-2014-report-dsl-shared-task/