Elamur Research⌕
☰
home / papers / 2018 / Classifier Ensembles for Dialect and Language Variety Identification

Classifier Ensembles for Dialect and Language Variety Identification

Summary

The paper tackles automatic identification of closely related language varieties in two VarDial 2018 shared tasks. The first is ADI, which covers four Arabic dialects plus Modern Standard Arabic in transcribed speech broadcasts. The second is DFS, which separates Dutch from Flemish in subtitle excerpts drawn from the SUBTIEL corpus. The authors reuse an approach they had applied to author profiling and native language identification. Each feature family gets its own linear SVM: TF-IDF character n-grams, word n-grams, and word k-skip bigrams. The individual predictions are then merged by an unweighted majority vote. Feature subsets and the regularization constant were chosen by grid search on development data. The ensembles beat the random baselines and their own strongest single member on the test sets. However, they finished in the lower part of both rankings, behind string-kernel and other systems. The work matters mainly as a carefully documented, easy-to-reproduce reference point for dialect and variety classification. It also shows candidly that a recipe that won earlier editions does not automatically transfer to new data.

Key points

  • DFS data: 320,500 SUBTIEL instances, about 11 million tokens in total. The split is 300,000 for training, 500 for development and 20,000 for testing.
  • ADI data and test design: Participants received ASR output, acoustic features and phonetic features. The test set quietly mixed in-domain and out-of-domain material without telling participants.
  • Implementation: Built with scikit-learn, using LinearSVC (Liblinear) members combined by VotingClassifier. Ties are resolved by the sorted order of the labels.
  • Selected configurations:
    • DFS used character 3–6-grams plus word 3-grams with C = 100.
    • ADI used character 3–5-grams with C = 1.
  • Test results:
TaskEnsembleBest single memberRandom baselineRankWinner
ADI0.500 macro F10.4930.200last rankedUnibucKernel, 0.589
DFS0.596 macro F10.5760.5006thTübingen-Oslo, 0.660

Interesting ideas

  • I1. One learner per feature view. Train a separate simple model on each feature view and let them vote, so every decision can be traced to which views agreed or disagreed.
  • I2. Compare against the best member on both splits. On development data the best single classifier slightly beat the selected ensemble, yet on test data the ensemble came out ahead, so picking a model by dev score alone can mislead.
  • I3. Hide out-of-domain items in the test set. Merging out-of-domain items into the test set without labeling them probes robustness rather than tuning to a known domain.
  • I4. Break ties deterministically. Resolving voting ties by a fixed label order keeps results reproducible across runs.
  • I5. Diagnose with per-class confusion matrices. The confusion matrices revealed which specific varieties got mixed up, such as Levantine with North African, and which was hardest, namely Gulf.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): The same pattern fits silver-data agreement, with each witness parser acting as a separate voter. Logging which witnesses formed the majority gives an explainable trust signal per arc.
  • I2 (learn): When evaluating gparse versus voting combinations, report both member and ensemble scores on held-out test data before discarding the ensemble.
  • I3 (answer): Fable QA and school-test benchmarks could secretly include texts from unfamiliar genres. This would check whether readers degrade gracefully.
  • I4 (tree): When multiple parsers or repair rules tie on a head or relation choice, a documented fixed tie-break rule keeps tree outputs stable and explainable.
  • I5 (tokens): A per-label confusion analysis of the tagger and morphology, for example between British and American spelling variants, would point to where targeted rules are needed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Semitic languages

Cite

Cite as Liviu P. Dinu, Alina Maria Ciobanu, Marcos Zampieri and Shervin Malmasi. 2018. Classifier Ensembles for Dialect and Language Variety Identification. arXiv:1808.04800.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1808.04800 · arXiv 1808.04800

@misc{dinu2018classifier,
  title = {{Classifier Ensembles for Dialect and Language Variety Identification}},
  author = {Liviu P. Dinu, Alina Maria Ciobanu, Marcos Zampieri and Shervin Malmasi},
  year = {2018},
  doi = {10.48550/arxiv.1808.04800},
  eprint = {1808.04800},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1808.04800}
}

References

45 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wafia Adouane, Nasredine Semmar, and Richard Johansson. 2016. ASIREM Participation at the Discriminating Similar Languages Shared Task 2016. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages 163–169, Osaka, Japan. find
  2. Ahmed Ali, Najim Dehak, Patrick Cardinal, Sameer Khurana, Sree Harsha Yella, James Glass, Peter Bell, and Steve Renals. 2016. Automatic Dialect Detection in Arabic Broadcast Speech. In Proceedings of INTERSPEECH, pages 2934–2938. find
  3. Ahmed Ali, Stephan Vogel, and Steve Renals. 2017. Speech Recognition Challenge in the Wild: Arabic MGB-3. arXiv preprint arXiv:1709.07276. ↗ our card
  4. Mohamed Ali. 2018. Character Level Convolutional Neural Network for Arabic Dialect Identification. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  5. Adrien Barbaresi. 2018. Computationally Efficient Discrimination Between Language Varieties with Large Feature Vectors and Regularized Classifiers. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  6. Andrei M. Butnaru and Radu Ionescu. 2018. UnibucKernel Reloaded: First Place in Arabic Dialect Identification for the Second Year in a Row. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  7. Çağrı Çöltekin, Taraka Rama, and Verena Blaschke. 2018. Tübingen-Oslo Team at the Var-Dial 2018 Evaluation Campaign: An Analysis of N-gram Features in Language Variety Identification. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  8. Alina Maria Ciobanu and Liviu P Dinu. 2016. A Computational Perspective on the Romanian Dialects. In Proceedings of Language Resources and Evalution (LREC). find
  9. Alina Maria Ciobanu, Marcos Zampieri, Shervin Malmasi, and Liviu P Dinu. 2017. Including Dialects and Language Varieties in Author Profiling. Working Notes of CLEF. find
  10. Alina Maria Ciobanu, Shervin Malmasi, and Liviu P. Dinu. 2018a. German Dialect Identification Using Classifier Ensembles. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  11. Alina Maria Ciobanu, Marcos Zampieri, Shervin Malmasi, Santanu Pal, and Liviu P. Dinu. 2018b. Discriminating between Indo-Aryan Languages Using SVM Ensembles. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  12. Steven Du and Yuan Yuan Wang. 2018. STEVENDU2018’s System in VarDial 2018: Discriminating between Dutch and Flemish in Subtitles. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  13. Mohamed Eldesouki, Fahim Dalvi, Hassan Sajjad, and Kareem Darwish. 2016. QCRI @ DSL 2016: Spoken Arabic Dialect Identification Using Textual Features. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages 221–226, Osaka, Japan. find
  14. Heba Elfardy and Mona T Diab. 2013. Sentence Level Dialect Identification in Arabic. In Proceedings of ACL. find
  15. Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. LIB-LINEAR: A Library for Large Linear Classification. Journal of Machine Learning Research, 9:1871–1874. find
  16. Radu Tudor Ionescu and Andrei Butnaru. 2017. Learning to Identify Arabic and German Dialects using Multiple Kernels. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), pages 200–209, Valencia, Spain, April. find
  17. Radu Tudor Ionescu and Marius Popescu. 2016. UnibucKernel: An Approach for Arabic Dialect Identification Based on Multiple String Kernels. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages 135–144, Osaka, Japan. find
  18. Tommi Jauhiainen, Heidi Jauhiainen, and Krister Lindén. 2018a. HeLI-based Experiments in Discriminating Between Dutch and Flemish Subtitles. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  19. Tommi Jauhiainen, Marco Lui, Marcos Zampieri, Timothy Baldwin, and Krister Lindén. 2018b. Automatic Language Identification in Texts: A Survey. arXiv preprint arXiv:1804.08186. ↗ our card
  20. Tim Kreutz and Walter Daelemans. 2018. Exploring Classifier Combinations for Language Variety Identification. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  21. Martin Kroon, Maria Medvedeva, and Barbara Plank. 2018. When Simple n-gram Models Outperform Syntactic Approaches: Discriminating between Dutch and Flemish. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  22. Marco Lui and Paul Cook. 2013. Classifying English Documents by National Dialect. In Proceedings of ALTA. find
  23. Shervin Malmasi and Mark Dras. 2015. Language Identification using Classifier Ensembles. In Proceedings of the Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects (LT4VarDial), pages 35–43, Hissar, Bulgaria. find
  24. Shervin Malmasi and Marcos Zampieri. 2016. Arabic Dialect Identification in Speech Transcripts. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages 106–113, Osaka, Japan. find
  25. Shervin Malmasi and Marcos Zampieri. 2017a. Arabic Dialect Identification Using iVectors and ASR Transcripts. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), pages 178–183, Valencia, Spain, April. find
  26. Shervin Malmasi and Marcos Zampieri. 2017b. German Dialect Identification in Interview Transcriptions. In Proceedings of the Fourth VarDial Workshop, pages 164–169, Valencia, Spain. find
  27. Shervin Malmasi, Eshrag Refaee, and Mark Dras. 2015. Arabic Dialect Identification using a Parallel Multidialectal Corpus. In Proceedings of the 14th Conference of the Pacific Association for Computational Linguistics (PACLING 2015), pages 209–217, Bali, Indonesia, May. find
  28. Shervin Malmasi, Mark Dras, and Marcos Zampieri. 2016a. LTG at SemEval-2016 Task 11: Complex Word Identification with Classifier Ensembles. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016). our card
  29. Shervin Malmasi, Marcos Zampieri, and Mark Dras. 2016b. Predicting Post Severity in Mental Health Forums. In Proceedings of the Third Computational Linguistics and Clinical Psychology Workshop (CLPsych), pages 133–137, San Diego, California, USA. find
  30. Shervin Malmasi, Marcos Zampieri, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, and Jörg Tiedemann. 2016c. Discriminating between Similar Languages and Arabic Dialect Identification: A Report on the Third DSL Shared Task. In Proceedings of the 3rd Workshop on Language Technology for Closely Related Languages, Varieties and Dialects (VarDial), Osaka, Japan. our card
  31. Elise Michon, Minh Quang Pham, Josep Crego, and Jean Senellart. 2018. Neural Network Architectures for Arabic Dialect Identification. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  32. Rabee Naser and Abualsoud Hanani. 2018. Birzeit Arabic Dialect Identication System for the 2018 VarDial Challenge. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  33. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12:2825–2830. find
  34. Yves Peirsman, Dirk Geeraerts, and Dirk Speelman. 2010. The Automatic Identification of Lexical Variation Between Language Varieties. Natural Language Engineering, 16:469–491. find
  35. Francisco Rangel, Paolo Rosso, Martin Potthast, and Benno Stein. 2017. Overview of the 5th Author Profiling Task at PAN 2017: Gender and Language Variety Identification in Twitter. Working Notes Papers of the CLEF. find
  36. Christoph Tillmann, Saab Mansour, and Yaser Al-Onaizan. 2014. Improved Sentence-Level Arabic Dialect Classification. In Proceedings of the VarDial Workshop (VarDial). find
  37. Chris van der Lee and Antal van den Bosch. 2017. Exploring Lexical and Syntactic Features for Language Variety Identification. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), pages 190–199, Valencia, Spain. find
  38. Hans van Halteren and Nelleke Oostdijk. 2018. Identification of Differences between Dutch Language Varieties with the VarDial2018 Dutch-Flemish Subtitle Data. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  39. Omar F Zaidan and Chris Callison-Burch. 2014. Arabic Dialect Identification. Computational Linguistics, 40(1):171–202. our card
  40. Marcos Zampieri, Liling Tan, Nikola Ljubešić, and Jörg Tiedemann. 2014. A Report on the DSL Shared Task 2014. In Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects (VarDial), pages 58–67, Dublin, Ireland. our card
  41. Marcos Zampieri, Shervin Malmasi, Octavia-Maria Sulea, and Liviu P Dinu. 2016. A Computational Approach to the Study of Portuguese Newspapers Published in Macau. In Proceedings of Workshop on Natural Language Processing Meets Journalism (NLPMJ). find
  42. Marcos Zampieri, Alina Maria Ciobanu, and Liviu P. Dinu. 2017a. Native Language Identification on Text and Speech. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 398–404, Copenhagen, Denmark, September. Association for Computational Linguistics. find
  43. Marcos Zampieri, Shervin Malmasi, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, Jörg Tiedemann, Yves Scherrer, and Noëmi Aepli. 2017b. Findings of the VarDial Evaluation Campaign 2017. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), Valencia, Spain. our card
  44. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Ahmed Ali, Suwon Shon, James Glass, Yves Scherrer, Tanja Samardžić, Nikola Ljubešić, Jörg Tiedemann, Chris van der Lee, Stefan Grondelaers, Nelleke Oostdijk, Antal van den Bosch, Ritesh Kumar, Bornini Lahiri, and Mayank Jain. 2018. Language Identification and Morphosyntactic Tagging: The Second VarDial Evaluation Campaign. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), Santa Fe, USA. our card
  45. Arkaitz Zubiaga, Inaki San Vicente, Pablo Gamallo, José Ramom Pichel, Inaki Alegria, Nora Aranberri, Aitzol Ezeiza, and Vı́ctor Fresno. 2016. TweetLID: A Benchmark for Tweet Language Identification. Language Resources and Evaluation, 50(4):729–766. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dinu-2018-classifier-ensembles-dialect-language-variety/