Elamur Research⌕
☰
home / papers / 2023 / Optimizing the Size of Subword Vocabularies in Dialect Classification

Optimizing the Size of Subword Vocabularies in Dialect Classification

Summary

Pretrained models arrive with a fixed tokenizer, which may fit poorly to low-resource dialect data whose spelling varies a lot. The authors ask whether choosing the right token granularity improves dialect classification, a task that depends on small surface cues such as affixes and character clusters. They use four datasets. GDI (Swiss German) and ADI (Arabic) are transcribed speech with inconsistent spelling. ILI (Indo-Aryan) and AOC (Arabic) are originally written and more consistent. NADI serves as an extra check. They compare character, word and subword inputs, building subword vocabularies of varying size with SentencePiece Unigram and BPE. These feed BiLSTM and CNN classifiers trained from scratch, which are then compared with fine-tuned BERT-style models. A CNN trained from scratch on a tuned subword vocabulary wins on the noisy transcribed datasets. On consistently written data, language-specific pretrained models stay ahead whatever the vocabulary size. The practical lesson is that how noisy the writing is should guide whether to fine-tune or to train small models with a tuned tokenizer.

Key points

  • Datasets: GDI-2018 (4 areas: Basel, Bern, Lucerne, Zurich, from the ArchiMob corpus), ILI (5 languages incl. Hindi, Braj Bhasha, Awadhi, Bhojpuri, Magahi), ADI, AOC and NADI 2022 (18 country dialects, roughly 20K training tweets).
  • On ADI, Kim_CNN with a Unigram vocabulary beats the best fine-tuned model (AraBERT) by 6.1 F1 points. Kim_CNN + Unigram is also best on GDI.
  • On GDI, the strongest fine-tuned model was BERT-base-cased, not German BERT.
  • On ILI and AOC, fine-tuned language-specific models (Indic Transformers, AraBERT) are best. NADI follows the same pattern. BPE beat Unigram on 2 of 3 consistently written datasets.
  • Confusion matrices show the same hard spots for both approaches: Bern vs Lucerne in GDI, and Magahi in ILI.

Interesting ideas

  • I1. Treat subword vocabulary size as a hyperparameter to sweep for each dataset, rather than accepting whatever size a pretrained model ships with. — For Mova (tokens): When splitting rare or noisy English word forms for the tagger and morphology, sweep the size of any subword fallback against tagging accuracy instead of hard-coding it.
  • I2. Hold the architecture fixed and swap only the input unit (character, subword or word), so any change in results can be credited to tokenization alone. — For Mova (tokens): Compare character, subword and word features inside the same tagger and lemmatizer to see which granularity actually drives accuracy on unknown words.
  • I3. Split evaluation by how consistent the writing is (transcribed vs originally written), because the best method flips between the two groups. — For Mova (learn): Report scores separately on clean edited text and on noisy text (forums, transcripts), so a regression on noisy input is not hidden behind a good average.
  • I4. Keep a small model trained from scratch as a serious rival to a fine-tuned large model, since on noisy data the small model can win. — For Mova (tree): Benchmark the perceptron parser against gparse on noisy-text treebanks, and route input to whichever parser proves more reliable for that text type.
  • I5. Put two systems' confusion matrices side by side to tell confusions that are inherently hard (both fail) from errors specific to one model. — For Mova (learn): Compare witness parsers' label confusion matrices to find dependency relations that every parser mixes up; these may need guideline fixes rather than more training data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

7 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 6 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.vardial-1.2

@inproceedings{kanjirangat2023optimizing,
  title = {{Optimizing the Size of Subword Vocabularies in Dialect Classification}},
  author = {Kanjirangat, Vani and Samardžić, Tanja and Dolamic, Ljiljana and Rinaldi, Fabio},
  booktitle = {Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023)},
  year = {2023},
  doi = {10.18653/v1/2023.vardial-1.2},
  url = {https://doi.org/10.18653/v1/2023.vardial-1.2}
}

References

56 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. NADI 2020: The first nuanced Arabic dialect identification shared task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 97–110, Barcelona, Spain (Online). Association for Computational Linguistics. find
  2. Andrei Butnaru. 2019. Bam: A combination of deep and shallow models for german dialect identification. In Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 128–137. find
  3. Andrea Ceolin. 2021. Comparing the performance of cnns and shallow models for language identification. In Proceedings of the Eighth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 102–112. find
  4. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2021. NADI 2021: The second nuanced Arabic dialect identification shared task. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 244–259, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. find
  5. Bharathi Raja Chakravarthi, Mihaela Gaman, Radu Tudor Ionescu, Heidi Jauhiainen, Tommi Jauhiainen, Krister Lindén, Nicola Ljubešić, Niko Partanen, Ruba Priyadharshini, Christoph Purschke, et al. 2021. Findings of the vardial evaluation campaign 2021. In Proceedings of the 8th VarDial Workshop on NLP for Similar Languages, Varieties and Dialects. The Association for Computational Linguistics. find
  6. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. NADI 2022: The Third Nuanced Arabic Dialect Identification Shared Task. In Proceedings of the Seven Arabic Natural Language Processing Workshop (WANLP 2022). find
  7. Colin Cherry, George Foster, Ankur Bapna, Orhan Firat, and Wolfgang Macherey. 2018. Revisiting character-based neural machine translation with capacity and compression. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4295–4305. find our card
  8. Ahmed Ali, Najim Dehak, Patrick Cardinal, Sameer Khurana, Sree Harsha Yella, James Glass, Peter Bell, and Steve Renals. 2016. Automatic dialect detection in arabic broadcast speech. find
  9. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15. find our card
  10. Çağrı Çöltekin and Taraka Rama. 2016. Discriminating similar languages with linear svms and neural networks. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages 15–24. find
  11. Duygu Ataman and Marcello Federico. 2018. An evaluation of two vocabulary reduction methods for neural machine translation. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 97–110, Boston, MA. Association for Machine Translation in the Americas. find
  12. Mathias Creutz and Krista Lagus. 2005. Inducing the morphological lexicon of a natural language from unannotated text. In In Proceedings of the International and Interdisciplinary Conference on Adaptive Knowledge Representation and Reasoning (AKRR’05, pages 106–113. find
  13. Toms Bergmanis and Sharon Goldwater. 2017. From segmentation to analyses: a probabilistic model for unsupervised morphology induction. In Proceedings of the 15th Conference of the European Chapter of Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171– 4186. find
  14. Tommi Jauhiainen, Heidi Jauhiainen, and Krister Lindén. 2018a. Iterative language model adaptation for indo-aryan language identification. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pages 66–75. find
  15. Shuoyang Ding, Adithya Renduchintala, and Kevin Duh. 2019. A call for prudent choice of subword merge operations in neural machine translation. In Proceedings of Machine Translation Summit XVII: Research Track, pages 204–213. find
  16. Tommi Jauhiainen, Krister Lindén, and Heidi Jauhiainen. 2019. Discriminating between Mandarin Chinese and Swiss-German varieties using adaptive language models. In Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 178–187, Ann Arbor, Michigan. Association for Computational Linguistics. find
  17. Miguel Domingo, Mercedes Garcıa-Martınez, Alexandre Helle, Francisco Casacuberta, and Manuel Herranz. 2018. How much does tokenization affect neural machine translation? arXiv preprint arXiv:1812.08621. ↗
  18. Tommi Jauhiainen, Tharindu Ranasinghe, and Marcos Zampieri. 2021. Comparing approaches to dravidian language identification. In Proceedings of the Eighth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 120–127. find
  19. Philip Gage. 1994. A new algorithm for data compression. C Users Journal, 12(2):23–38. find
  20. Mihaela Gaman, Dirk Hovy, Radu Tudor Ionescu, Heidi Jauhiainen, Tommi Jauhiainen, Krister Lindén, Nikola Ljubešić, Niko Partanen, Christoph Purschke, Yves Scherrer, et al. 2020. A report on the vardial evaluation campaign 2020. In Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 1–14. find
  21. Tommi Sakari Jauhiainen, Heidi Annika Jauhiainen, Bo Krister Johan Linden, et al. 2018b. Heli-based experiments in swiss german dialect identification. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018). The Association for Computational Linguistics. find
  22. Koustava Goswami, Rajdeep Sarkar, Bharathi Raja Chakravarthi, Theodorus Fransen, and John Philip McCrae. 2020. Unsupervised deep language and dialect identification for short texts. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1606–1617. find
  23. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. In Thirtieth AAAI conference on artificial intelligence. find our card
  24. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational Linguistics. find
  25. Thamme Gowda and Jonathan May. 2020. Finding the optimal vocabulary size for neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3955–3964, Online. Association for Computational Linguistics. find
  26. Stig-Arne Grönroos, Sami Virpioja, and Mikko Kurimo. 2020. Morfessor EM+Prune: Improved subword segmentation with expectation maximization and pruning. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3944–3953, Marseille, France. European Language Resources Association. find
  27. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Association for Computational Linguistics. find our card
  28. Ximena Gutierrez-Vasques, Christian Bentz, Olga Sozinova, and Tanja Samardzic. 2021. From characters to words: the turning point of bpe merges. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3454–3468. find
  29. Jiahuan Li, Yutong Shen, Shujian Huang, Xinyu Dai, and Jiajun Chen. 2021. When is char better than subword: A systematic study of segmentation algorithms for neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 543–549. find
  30. Hai Hu, Wen Li, He Zhou, Zuoyu Tian, Yiwen Zhang, and Liang Zou. 2019. Ensemble methods to distinguish mainland and taiwan chinese. In Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 165–171. find
  31. Zhouhan Lin, Minwei Feng, Cícero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net. find our card
  32. Kushal Jain, Adwait Deshpande, Kumar Shridhar, Felix Laumann, and Ayushman Dash. 2020. Indic-transformers: An analysis of transformer language models for indian languages. arXiv preprint arXiv:2011.02323. ↗
  33. Shervin Malmasi, Marcos Zampieri, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, and Jörg Tiedemann. 2016. Discriminating between similar languages and arabic dialect identification: A report on the third dsl shared task. In Proceedings of the third workshop on NLP for similar languages, varieties and dialects (VarDial3), pages 1–14. find our card
  34. Jörg Tiedemann and Nikola Ljubešić. 2012. Efficient discrimination between closely related languages. In Proceedings of COLING 2012, pages 2619–2634, Mumbai, India. find
  35. The COLING 2012 Organizing Committee. find
  36. Alper Kursat Uysal and Serkan Gunal. 2014. The impact of preprocessing on text classification. Information processing & management, 50(1):104–112. find
  37. Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. 2019. What kind of language is hard to language-model? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4975–4989, Florence, Italy. Association for Computational Linguistics. find
  38. Clara Vania and Adam Lopez. 2017. From characters to words to in between: Do we capture morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2016–2027, Vancouver, Canada. Association for Computational Linguistics. find
  39. Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. 2015. An unsupervised method for uncovering morphological chains. Transactions of the Association for Computational Linguistics, 3:157– 167. find
  40. Jingjing Xu, Hao Zhou, Chun Gan, Zaixiang Zheng, and Lei Li. 2021. Vocabulary learning via optimal transport for neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7361–7373. find
  41. Hyunji Hayley Park, Katherine J. Zhang, Coleman Haley, Kenneth Steimel, Han Liu, and Lane Schwartz. 2021. Morphology matters: A multilingual language modeling analysis. Transactions of the Association for Computational Linguistics, 9:261–276. find
  42. Li Yang and Yang Xiang. 2019. Naive bayes and bilstm ensemble for discriminating between mainland and taiwan variation of mandarin chinese. In Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 120–127. find
  43. Hoifung Poon, Colin Cherry, and Kristina Toutanova. 2009. Unsupervised morphological segmentation with log-linear models. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 209–217, Boulder, Colorado. Association for Computational Linguistics. find
  44. George-Eduard Zaharia, Andrei-Marius Avram, Dumitru-Clementin Cercel, and Traian Rebedea. 2020. Exploring the power of romanian bert for dialect identification. In Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 232–241. find
  45. Cristian Popa and Vlad S , tefănescu. 2020. Applying multilingual and monolingual transformer-based models for dialect identification. In Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 193–201. find
  46. Omar Zaidan and Chris Callison-Burch. 2011. The arabic online commentary dataset: an annotated dataset of informal arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 37–41. find our card
  47. Tanja Samardzic, Yves Scherrer, and Elvira Glaser. 2016. Archimob-a corpus of spoken swiss german. In Proceedings of the tenth international conference on language resources and evaluation (LREC 2016). European Language Resources Association (ELRA). find
  48. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152. find
  49. Marcos Zampieri, Shervin Malmasi, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, Jörg Tiedemann, Yves Scherrer, and Noëmi Aepli. 2017. Findings of the vardial evaluation campaign 2017. In Proceedings of the fourth workshop on NLP for similar languages, varieties and dialects. find our card
  50. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Linguistics. ↗
  51. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Ahmed Ali, Suwon Shon, James Glass, Yves Scherrer, Tanja Samardžić, Nikola Ljubešić, Jörg Tiedemann, et al. 2018. Language identification and morphosyntactic tagging. the second vardial evaluation campaign. find our card
  52. Bashar Talafha, Mohammad Ali, Muhy Eddin Za’ter, Haitham Seelawi, Ibraheem Tuffaha, Mostafa Samir, Wael Farhan, and Hussein Al-Natsheh. 2020. Multi-dialect arabic bert for country-level dialect identification. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 111–118. find
  53. Marcos Zampieri, Shervin Malmasi, Yves Scherrer, Tanja Samardžić, Francis Tyers, Miikka Silfverberg, Natalia Klyueva, Tung-Le Pan, Chu-Ren Huang, Radu Tudor Ionescu, et al. 2019. A report on the third vardial evaluation campaign. Association for Computational Linguistics. find
  54. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28. find our card
  55. Ayah Zirikly, Bart Desmet, and Mona Diab. 2016. The gw/lt3 vardial 2016 shared task system for dialects and similar languages detection. In COLING, pages 33–41. find
  56. The COLING 2016 Organizing Committee. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kanjirangat-2023-optimizing-size-subword-vocabularies-dialect/