Elamur Research⌕
☰
home / papers / 2015 / Overview of the DSL Shared Task 2015

Overview of the DSL Shared Task 2015

Summary

This report describes the second Discriminating between Similar Languages (DSL) shared task, held at the LT4VarDial'2015 workshop (RANLP'2015). Language identification works well for unrelated languages, but it still fails on closely related languages and national varieties. The organisers released DSLCC v2.0, a corpus of journalistic text excerpts. It covers six groups of similar languages and adds an Others class of languages that never appear in training, which makes the setting more realistic.

Two test sets were used. Set A kept the original texts. Set B replaced named entities with #NE# placeholders, so that systems could not rely on topic cues such as place or politician names. Teams could submit closed runs (trained only on DSLCC v2.0) or open runs (any data).

Ten teams submitted results. The best system scored above 95% on set A. Hiding entities lowered accuracy by only about 2 points. This suggests that related varieties have learnable linguistic differences beyond topical vocabulary.

Key points

  • DSLCC v2.0 contains 308,000 examples in 14 classes (22,000 each):
    • bs/hr/sr
    • id/my
    • cz/sk
    • pt-BR/pt-PT
    • es-AR/es-ES
    • bg/mk
    • Others (xx), a mix of Catalan, Russian, Slovene and Tagalog
  • 24 teams registered and 10 submitted. An additional Unshared Task track received no papers.
  • The closed-track winner was MAC, an ensemble of SVM classifiers, with 95.54% on set A and 94.01% on set B. The open-track winner was NRC, a two-stage group-then-language classifier, with 95.65% on A and 93.41% on B. NRC's gain came partly from adding DSLCC v1.0 as training data.
  • Bulgarian–Macedonian, Czech–Slovak and Indonesian–Malay were nearly solved on set A. Indonesian–Malay became hard on set B, probably because country-specific named entities had been helping on set A. Bosnian–Croatian–Serbian and the Spanish and Portuguese varieties remained the hardest groups.
  • Plans for a third edition include adding dialects (e.g., Arabic) and studying how text length affects accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Computing & GPUs

Cited by

78 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 74 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zampieri2015overview,
  title = {{Overview of the DSL Shared Task 2015}},
  author = {Zampieri, Marcos and Tan, Liling and Ljubešić, Nikola and Tiedemann, Jörg and Nakov, Preslav},
  booktitle = {LT4VarDial 2015},
  year = {2015},
  url = {https://aclanthology.org/W15-5401/}
}

References

40 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Judit Ács, László Grad-Gyenge, and Thiago Bruno Rodrigues de Rezende Oliveira. 2015. A two-level classifier for discriminating similar languages. In Proceedings of the Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects, LT4VarDial ’15, pages 73–77, Hissar,… find
  2. Timothy Baldwin and Marco Lui. 2010. Multilingual language identification: ALTW 2010 shared task data. In Proceedings of Australasian Language Technology Association Workshop, ALTA ’10, pages 4–7, Melbourne, Australia. find
  3. Chris Biemann, Gerhard Heyer, Uwe Quasthoff, and Matthias Richter. 2007. The Leipzig corpora collection-monolingual corpora of standard size. In Proceedings of Corpus Linguistics, Birmingham, UK. find
  4. Victoria Bobicev. 2015. Discriminating between similar languages using PPM. In Proceedings of the Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects, LT4VarDial ’15, pages 59–65, Hissar, Bulgaria. find
  5. Ralf Brown. 2014. Non-linear mapping for improved identification of 1300+ languages. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP ’14, pages 627– 632, Doha, Qatar. find
  6. Raül Fabra-Boluda, Francisco Rangel, and Paolo Rosso. 2015. NLEL UPV autoritas participation at Discrimination between Similar Languages (DSL) 2015 shared task. In Proceedings of the Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects, LT4VarDial ’15, pages 5… find
  7. Marc Franco-Salvador, Paolo Rosso, and Francisco Rangel. 2015. Distributed representations of words and documents for discriminating similar languages. In Proceedings of the Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects, LT4VarDial ’15, pages 11–16, Hiss… find
  8. Binyam Gebrekidan Gebre, Marcos Zampieri, Peter Wittenburg, and Tom Heskes. 2013. Improving native language identification with TF-IDF weighting. In Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, BEA8, pages 216–223, Atlanta, GA, USA. find
  9. Cyril Goutte and Serge Léger. 2015. Experiments in discriminating similar languages. In Proceedings of the Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects, LT4VarDial ’15, pages 78–84, Hissar, Bulgaria. find
  10. Cyril Goutte, Serge Léger, and Marine Carpuat. 2014. The NRC system for discriminating similar languages. In Proc. of the Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects, VarDial ’14, pages 139–145, Dublin, Ireland. find
  11. Cyril Grouin, Dominic Forest, Lyne Da Sylva, Patrick Paroubek, and Pierre Zweigenbaum. 2010. Présentation et résultats du défi fouille de texte DEFT2010 où et quand un article de presse a-t-il été écrit? In Actes du sixième Défi Fouille de Textes, DEFT ’10, Montreal, Canada. find
  12. Chu-ren Huang and Lung-hao Lee. 2008. Contrastive approach towards text source classification based on top-bag-of-word similarity. In Proceedings of 22nd Pacific Asia Conference on Language, Information and Computation, PACLIC ’08, pages 404– 410, Cebu City, Philippines. find
  13. Tommi Jauhiainen, Heidi Jauhiainen, and Krister Lindén. 2015a. Discriminating similar languages with token-based backoff. In Proceedings of the Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects, LT4VarDial ’15, pages 44–51, Hissar, Bulgaria. find
  14. Tommi Jauhiainen, Krister Lindén, and Heidi Jauhiainen. 2015b. Language set identification in noisy synthetic multilingual documents. In Proceedings of the 16th International Conference on Computational Linguistics and Intelligent Text Processing, CICLING ’15, pages 633–643, Cairo, Egypt. find
  15. Ben King, Dragomir Radev, and Steven Abney. 2014. Experiments in sentence language identification with groups of similar languages. In Proceedings of the 1st Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects, VarDial ’14, pages 146–154, Dublin, Ireland. find
  16. Nikola Ljubešić and Denis Kranjčić. 2015. Discriminating between closely related languages on Twitter. Informatica, 39(1). find
  17. Marco Lui and Timothy Baldwin. 2011. Cross-domain feature selection for language identification. In Proceedings of 5th International Joint Conference on Natural Language Processing, IJCNLP ’11, pages 553–561, Chiang Mai, Thailand. find
  18. Marco Lui and Timothy Baldwin. 2012. Langid.py: An off-the-shelf language identification tool. In Proceedings of the ACL 2012 System Demonstrations, ACL ’12, pages 25–30, Jeju Island, Korea. find our card
  19. Marco Lui and Paul Cook. 2013. Classifying English documents by national dialect. In Proceedings of Australasian Language Technology Workshop, ALTA ’13, pages 5–15, Brisbane, Australia. find
  20. Marco Lui, Ned Letcher, Oliver Adams, Long Duong, Paul Cook, and Timothy Baldwin. 2014. Exploring methods and resources for discriminating similar languages. In Proceedings of the 1st Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects, VarDial ’14, pages 129–138, Dublin, Ire… find
  21. Shervin Malmasi and Mark Dras. 2015a. Automatic language identification for Persian and Dari texts. In Proceedings of the 14th Conference of the Pacific Association for Computational Linguistics, PA-CLING ’15, pages 59–64, Bali, Indonesia. find
  22. Shervin Malmasi and Mark Dras. 2015b. Language identification using classifier ensembles. In Proceedings of the Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects, LT4VarDial ’15, pages 35–43, Hissar, Bulgaria. find
  23. Shervin Malmasi, Eshrag Refaee, and Mark Dras. 2015. Arabic dialect identification using a parallel multidialectal corpus. In Proceedings of the 14th Conference of the Pacific Association for Computational Linguistics, PACLING ’15, pages 209–217, Bali, Indonesia. find
  24. Jordi Porta and José-Luis Sancho. 2014. Using maximum entropy models to discriminate between similar languages and varieties. In Proceedings of the 1st Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects, VarDial ’14, pages 120–128, Dublin, Ireland. find
  25. Matthew Purver. 2014. A simple baseline for discriminating similar language. In Proceedings of the 1st Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialect, VarDial ’14, pages 155–160, Dublin, Ireland. find
  26. Bali Ranaivo-Malançon. 2006. Automatic identification of close languages - case study: Malay and Indonesian. ECTI Transactions on Computer and Information Technology, 2:126–134. find
  27. Fatiha Sadat, Farnazeh Kazemi, and Atefeh Farzindar. 2014. Automatic identification of Arabic language varieties and dialects in social media. In Proceedings of the Second Workshop on Natural Language Processing for Social Media, SocialNLP ’14, pages 22–27, Dublin, Ireland. find
  28. Alberto Simões, José João Almeida, and Simon D Byers. 2014. Language identification: a neural network approach. In Proceedings of the 3rd Symposium on Languages, Applications and Technologies, SLATE ’14, pages 252–265, Dagstuhl, Germany. find
  29. Noah A. Smith, Claire Cardie, Anne L. Washington, and John D. Wilkerson. 2014. Overview of the 2014 NLP unshared task in PoliInformatics. In Proceedings of the Workshop on Language Technologies and Computational Social Science, pages 5–7, Baltimore, Maryland, USA. find
  30. Thamar Solorio, Elizabeth Blair, Suraj Maharjan, Steven Bethard, Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Alison Chang, and Pascale Fung. 2014. Overview for the first shared task on language identification in code-switched data. In Proceedings of the First Work… find
  31. Liling Tan, Marcos Zampieri, Nikola Ljubešić, and Jörg Tiedemann. 2014. Merging comparable data sources for the discrimination of similar languages: The DSL corpus collection. In Proceedings of The Workshop on Building and Using Comparable Corpora, BUCC ’14, pages 6–10, Reykjavik, Iceland. find
  32. Jörg Tiedemann and Nikola Ljubešić. 2012. Efficient discrimination between closely related languages. In Proceedings of the 24th International Conference on Computational Linguistics, COLING ’12, pages 2619–2634, Mumbai, India. find
  33. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the International Conference on Language Resources and Evaluation, LREC ’12, pages 2214–2218, Istanbul, Turkey. find
  34. Francis Tyers and Murat Serdar Alperen. 2010. South-East European Times: A parallel corpus of Balkan languages. In Proceedings of the LREC workshop on Exploitation of multilingual resources and tools for Central and (South) Eastern European Languages, Valetta, Malta. find
  35. Fei Xia, Carrie Lewis, and William D Lewis. 2010. The problems of language identification within hugely multilingual data sets. In Proceedings of the Seventh International Conference on Language Resources and Evaluation, LREC ’10, pages 2790– 2797, Valetta, Malta. find
  36. Yiming Yang and Jan O. Pedersen. 1997. A comparative study on feature selection in text categorization. In Proceedings of the Fourteenth International Conference on Machine Learning, ICML ’97, pages 412–420, Nashville, Tennessee, USA. find
  37. Marcos Zampieri and Binyam Gebrekidan Gebre. 2012. Automatic identification of language varieties: The case of Portuguese. In Proceedings of KONVENS, pages 233–237, Vienna, Austria. find
  38. Marcos Zampieri, Liling Tan, Nikola Ljubešić, and Jörg Tiedemann. 2014. A report on the DSL shared task 2014. In Proceedings of the Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects, VarDial ’14, pages 58–67, Dublin, Ireland. find our card
  39. Marcos Zampieri, Binyam Gebrekidan Gebre, Hernani Costa, and Josef van Genabith. 2015. Comparing approaches to the identification of similar languages. In Proceedings of the Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects, LT4VarDial ’15, pages 66–72, Hiss… find
  40. Arkaitz Zubiaga, Inaki San Vicente, Pablo Gamallo, José Ramom Pichel, Inaki Alegria, Nora Aranberri, Aitzol Ezeiza, and Vıctor Fresno. 2014. Overview of tweetLID: Tweet language identification at SE-PLN 2014. In Proceedings of the Tweet Language Identification Workshop, TweetLID ’14, pages 1–11, Gi… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/zampieri-2015-overview-dsl-shared-task/