Elamur Research⌕
☰
home / papers / 2026 / Dialectal Filtering: Synthesizing Kurdish Corpora for Low-Resource Varieties by Utilizing “Noise” in Large Textual Data

Dialectal Filtering: Synthesizing Kurdish Corpora for Low-Resource Varieties by Utilizing “Noise” in Large Textual Data

Summary

Kurdish is a dialect continuum with many varieties, but most web-derived text is tagged only with broad labels such as the macro-code kur. Smaller varieties therefore remain hidden and effectively zero-resource. Existing LID tools and large curation pipelines such as FineWeb2 depend on established language codes, so they cannot isolate these dialects. The authors offer a rule-based alternative that relies on lexical evidence instead of LID models. They build compact dialect-specific word lists, discard any items that also occur in high-resource languages to reduce false positives, and use the remaining distinctive words to give text instances multiple language labels. This filtering is applied to large, unlabeled mixed Kurdish datasets and yields 138 variety-specific sub-corpora. The authors describe this as the most extensive set of dialect-level Kurdish resources so far. The work matters because it provides basic data for dialect-specific NLP and sheds light on how Kurdish varieties relate to each other. The method is also intended to transfer to other languages with comparable dialect situations.

Key points

  • The main output is 138 new dialect-specific Kurdish sub-corpora, extracted from previously conflated web-scale data. Code is released at github.com/christianschuler8989/KurdishDialectFilters.
  • The method uses lexicographic features only. It relies on "unique" word lists per variety, excludes words shared with high-resource languages, and assigns multi-labels rather than a single LID label.
  • The authors chose a rule-based design deliberately. ML-based filters are known to disproportionately remove dialectal content, and the framework can later be extended with morphological, syntactic, or phonological features.
  • The surveyed resource landscape (UD, UniMorph, WordNet, NLLB, Wikipedia, FineWeb2, LID, MT, ASR, and others) reaches just six varieties: the Northern, Central, and Southern forms of Kurdish, along with Zazaki, Laki, and Hawrami. Laki and Hawrami are the most neglected.
  • Inconsistent naming adds to the problem. For example, Shabak has at least six spellings, and umbrella names like Kurmanji, Sorani, and Hawrami conceal sub-varieties. (Note: the evaluation details are missing from the provided extract.)

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Low-resource languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/249h3r9tiw52

@inproceedings{schuler2026dialectal,
  title = {{Dialectal Filtering: Synthesizing Kurdish Corpora for Low-Resource Varieties by Utilizing “Noise” in Large Textual Data}},
  author = {Schuler, Christian and Ahmad, Raman and Wáng, Ānrán and Gurgurov, Daniil and Baumann, Timo and Ostermann, Simon and Genabith, Josef van},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/249h3r9tiw52},
  url = {https://doi.org/10.63317/249h3r9tiw52}
}

References

81 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Roshna Omer Abdulrahman, Hossein Hassani, and Sina Ahmadi. 2019. Developing a fine-grained corpus for a less-resourced language: The case of kurdish. find
  2. Raman Ahmad and Christian Schuler. 2023. Analysis of phonology and morphology in the kobani dialect [conference poster & abstract]. find
  3. Sina Ahmadi. 2019. A rule-based kurdish text transliteration system. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 18(2):18:1–18:8. find
  4. Sina Ahmadi. 2020. Building a Corpus for the Zaza–Gorani Language Family. In Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 70–78, Barcelona, Spain (Online). International Committee on Computational Linguistics (ICCL). find
  5. Sina Ahmadi, Milind Agarwal, and Antonios Anastasopoulos. 2023a. Pali: A language identification benchmark for perso-arabic scripts. In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pages 78– 90, Dubrovnik, Croatia. Association for Computational Linguistics. find
  6. Sina Ahmadi, Zahra Azin, Sara Belelli, and Antonios Anastasopoulos. 2023b. Approaches to corpus creation for low-resource language technology: The case of southern kurdish and laki. find
  7. Sina Ahmadi, Hossein Hassani, and Daban Q. Jaff. 2022. Leveraging multilingual news websites for building a kurdish parallel corpus. ACM Transactions on Asian and Low-Resource Language Information Processing, 21(5):1–11. find
  8. Sina Ahmadi, Hossein Hassani, and John P. Mc-Crae. 2019. Towards electronic lexicography for the kurdish language. eLex 2019. find
  9. Sina Ahmadi, Daban Jaff, Md Mahfuz Ibn Alam, and Antonios Anastasopoulos. 2024. Language and speech technology for central kurdish varieties. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 10034–1004… find
  10. Sina Ahmadi and Aso Mahmudi. 2023. Revisiting and amending central kurdish data on unimorph 4.0. In Proceedings of the 20th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 38– 48, Toronto, Canada. Association for Computational Linguistics. find
  11. Sina Ahmadi, Rico Sennrich, Erfan Karami, Ako Marani, Parviz Fekrazad, Gholamreza Akbarzadeh Baghban, Hanah Hadi, Semko Heidari, Mahîr Dogan, Pedram Asadi, Dashne Bashir, Mohammad Amin Ghodrati, Kourosh Amini, Zeynab Ashourinezhad, Mana Baladi, Farshid Ezzati, Alireza Ghasemifar, Daryoush Hosseinpou… find
  12. Md Mahfuz Ibn Alam, Sina Ahmadi, and Antonios Anastasopoulos. 2024a. Codet: A benchmark for contrastive dialectal evaluation of machine translation. find
  13. Md Mahfuz Ibn Alam, Sina Ahmadi, and Antonios Anastasopoulos. 2024b. A morphologically-aware dictionary-based data augmentation technique for machine translation of underrepresented languages. find
  14. Mehdi Ali, Manuel Brack, Max Lübbering, Elias Wendt, Abbas Goher Khan, Richard Rutmann, Alex Jude, Maurice Kraus, Alexander Arno Weber, David Kaczér, Florian Mai, Lucie Flek, Rafet Sifa, Nicolas Flores-Herr, Joachim Köhler, Patrick Schramowski, Michael Fromm, and Kristian Kersting. 2025. Judging qua… find
  15. Hiwa Asadpour. 2023. A corpus analysis of the effects of definiteness and animacy on word order variation. Languages, 8(4):279. find
  16. Harald Baayen. 2001. Word Frequency Distributions. Kluwer Academic Publishers. find
  17. Soran Badawi, Ari M. Saeed, Sara A. Ahmed, Peshraw Ahmed Abdalla, and Diyari A. Hassan. 2023. Kurdish news dataset headlines (kndh) through multiclass classification. Data in Brief, 48:109120. find
  18. Kelsey Ball and Dan Garrette. 2018. Part-of-Speech Tagging for Code-Switched, Transliterated Texts without Explicit Language Identification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3084–3089, Brussels, Belgium. Association for Computational Li… find
  19. Emily M. Bender. 2011. On achieving and evaluating language-independence in nlp. Linguistic Issues in Language Technology, 6. find
  20. Gabriel Bernier-colborne, Cyril Goutte, and Serge Leger. 2023. Dialect and variant identification as a multi-label classification task: A proposal based on near-duplicate analysis. In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pages 142–151, Dubrovnik, Croati… find
  21. Steven Bird. 2020. Decolonising speech and language technology. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3504–3519, Barcelona, Spain (Online). International Committee on Computational Linguistics. find our card
  22. Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016. Demographic dialectal variation in social media: A case study of african-american english. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1119–1130, Austin, Texas. Association for Computational… find our card
  23. J. K. Chambers and Peter Trudgill. 1998. Dialectology, 2 edition. Cambridge Textbooks in Linguistics. Cambridge University Press, Cambridge. find
  24. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2017. find
  25. Conll-sigmorphon 2017 shared task: Universal morphological reinflection in 52 languages. In Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, pages 1– 30, Vancouver. Association for Computational Linguistics. find
  26. Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzman, and Philipp Koehn. 2020. Ccaligned: A massive collection of cross-lingual web-document pairs. find our card
  27. Kyumars Sheykh Esmaili, Donya Eliassi, Shahin Salavati, Purya Aliabadi, Asrin Mohammadi, Somayeh Yosefi, and Shownem Hakimi. 2013. Building a test collection for sorani kurdish. In 2013 ACS International Conference on Computer Systems and Applications (AICCSA), pages 1–7. find
  28. Fahim Faisal, Orevaoghene Ahia, Aarohi Srivastava, Kabir Ahuja, David Chiang, Yulia Tsvetkov, and Antonios Anastasopoulos. 2024. Dialectbench: A nlp benchmark for dialects, varieties, and closely-related languages. find
  29. Songül Gündoğdu, Ergîn Öpengîn, Geoffrey Haig, and Erik Anonby, editors. 2019. Current Issues in Kurdish Linguistics, volume 1 of Bamberg Studies in Kurdish Linguistics (BSKL). University of Bamberg Press. find
  30. Geoffrey Haig. 2001. Corpora of spoken and written varieties of kurdish. find
  31. Geoffrey Haig and Ergin Öpengin. 2018. Kurmanji kurdish in turkey: Structure, varieties, and status. Linguistic minorities in Turkey and Turkic-speaking minorities of the periphery, pages 157–229. find
  32. Geoffrey Haig, Donald Stilo, Mahîr Dogan, and Nils N. Schiborr. 2024. Wowa — word order in western asia: A spoken-language-based corpus for investigating areal effects in word order variation. find
  33. Hossein Hassani. 2021. Can linguistic distance help language classification? assessing hawrami-zaza and kurmanji-sorani. find
  34. Hanar Hoshyar Mustafa and Rebwar Nabi. 2022. Kurdish kurmanji dialect dataset for kurmanji lemmatization and spell-checker with spell-correction. find
  35. Abdullah Incekan and Geoffrey Haig. 2021. The corpus of contemporary written kurdish. find
  36. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the nlp world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Comp… find our card
  37. Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schütze. 2023. Glotlid: Language identification for low-resource languages. find our card
  38. Aram Khaksar and Hossein Hassani. 2024. Shifting from endangerment to rebirth in the artificial intelligence age: An ensemble machine learning approach for hawrami text classification. find
  39. Hewa Salam Khalid. 2020. Kurdish language, its family and dialects. International Journal of Kur-diname. find
  40. Ben King and Steven Abney. 2013. Labeling the languages of words in mixed-language documents using weakly supervised methods. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1110–1119, Atlanta, G… find
  41. Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhur… find our card
  42. Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, and Benjamin Van Durme. 2025. mmbert: A modern multilingual encoder with annealed language learning. find
  43. Tessa Masis, Anissa Neal, Lisa Green, and Brendan O’Connor. 2022. Corpus-guided contrast sets for morphosyntactic feature detection in low-resource english varieties. Proceedings of the first workshop on field linguistics, pages 11– 25. find
  44. Yaron Matras. 2019. Revisiting kurdish dialect geography: Findings from the manchester database. Current Issues in Kurdish Linguistics. Bamberg: Bamberg University Press. 225-241. find
  45. Dastan Maulud, Karwan Jacksi, and Ismael Ali. 2023. Towards a complete kurdish nlp pipeline: Challenges and opportunities. 17:1–17. find
  46. Peshmerge Morad, Sina Ahmadi, and Lorenzo Gatti. 2024. Part-of-Speech Tagging for Northern Kurdish. In Proceedings of the Joint Workshop on Multiword Expressions and Universal Dependencies (MWE-UD) @ LREC-COLING 2024, pages 70–80, Torino, Italia. ELRA and ICCL. find
  47. Rudra Murthy, Anoop Kunchukuttan, and Pushpak Bhattacharyya. 2019. Addressing word-order divergence in multilingual neural machine translation for extremely low resource languages. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: H… find
  48. Maria Nădejde, Siva Reddy, Rico Sennrich, Tomasz Dwojak, Marcin Junczys-Dowmunt, Philipp Koehn, and Alexandra Birch. 2017. Predicting target language ccg supertags improves neural machine translation. In Proceedings of the Second Conference on Machine Translation, pages 68–79, Copenhagen, Denmark. A… find
  49. Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddeen Muhammad, Salomon Kabongo Kabenamualu, Salomey Osei, Freshia Sackey, Rubungo Andre Niyongabo, Ricky Macharm, Perez Ogayo, Orevaoghene Ahia, Musie Meressa Berhe, Mofetoluwa A… find our card
  50. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9–16, Mannheim.… find
  51. Fatih Ozek, Bilgit Saglam, and Charlotte Gooskens. 2021. Mutual intelligibility of a kurmanji and a zazaki dialect spoken in the province of elazığ, turkey. Applied Linguistics Review. find
  52. Partha Pakray, Alexander Gelbukh, and Sivaji Bandyopadhyay. 2025. Natural language processing applications for low-resource languages. Natural Language Processing, 31(2):183–197. find
  53. Guilherme Penedo, Hynek Kydlíček, Loubna B. Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024a. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811–30849. find our card
  54. Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024b. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-Eight Conference on Neural Information Processing Systems Datasets a… find our card
  55. Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. 2025. Fineweb2: One pipeline to scale them all – adapting pre-training data processing to every language. find
  56. Steven T. Piantadosi. 2014. Zipf’s word frequency law in natural language: A critical review and future directions. Psychonomic Bulletin & Review, 21(5):1112–1130. find
  57. Paul Rayson. 2008. From key words to key semantic domains. International Journal of Corpus Linguistics, 13(4):519–549. find
  58. Víctor M. Sánchez-Cartagena, Juan Antonio Pérez-Ortiz, and Felipe Sánchez-Martínez. 2020. Understanding the effects of word-level linguistic annotations in under-resourced neural machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3938–3950, B… find
  59. Nathan C Sanders. 2010. A Statistical Method for Syntactic Dialectometry. Doctor of philosophy, Indiana University. find
  60. Yves Scherrer. 2012. Generating Swiss German Sentences from Standard German: A Multi-Dialectal Approach. Ph.D. thesis, Université de Genève. find
  61. Christian Schuler and Raman Ahmad. 2023. Towards a complete mapping of kurdish dialectology [conference poster & abstract]. find
  62. Thamar Solorio and Yang Liu. 2008. Learning to predict code-switching points. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 973– 981, Honolulu, Hawaii. Association for Computational Linguistics. find
  63. Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2025. Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset. find
  64. Givi Tavadze. 2019. Spreading of the kurdish language dialects and writing systems used in the middle east. 13(1). find
  65. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hof… find
  66. Mildred C. Templin. 1957. Certain Language Skills in Children: Their Development and Interrelationships, ned - new edition edition, volume 26. University of Minnesota Press. find
  67. Georgelle Thomas, R. Derald Hartley, and J. Peter Kincaid. 1975. Test-retest and inter-analyst reliability of the automated readability index, flesch reading ease score, and the fog count. Journal of Reading Behavior, 7(2):149–154. find
  68. Jörg Tiedemann. 2012. Parallel data, tools and In Proceedings of the interfaces in opus. Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA). find
  69. Hadi Veisi, Mohammad MohammadAmini, and Hawre Hosseini. 2020. Toward kurdish language processing: Experiments in collecting and processing the asosoft text corpus. Digital Scholarship in the Humanities, 35(1):176–193. find
  70. Wikimedia. 2025. List of wikipedias by language group - meta. find
  71. Roy Xie, Orevaoghene Ahia, Yulia Tsvetkov, and Antonios Anastasopoulos. 2024. Extracting lexical features from dialects via interpretable dialect classifiers. find
  72. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling… find
  73. Omar F. Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. find our card
  74. Marcos Zampieri, Shervin Malmasi, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, Jörg Tiedemann, Yves Scherrer, and Noëmi Aepli. 2017. Findings of the VarDial Evaluation Campaign 2017. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), pages 1–15, Valen… find our card
  75. G. K. Zipf. 1935. The Psycho-Biology of Language. The Psycho-Biology of Language. Houghton, Mifflin, Oxford, England. 10. find
  76. Language Resource References Ahmadi, Sina and Q. Jaff, Daban and Ibn Alam, Md Mahfuz and Anastasopoulos, Antonios. 2024. CORDI — Corpus of Dialogues in Central Kurdish. ELRA Language Resources Association. PID https://github.com/sinaahmadi/CORDI. ↗
  77. Ahmadi, Sina and Sennrich, Rico and Karami, Erfan and Marani, Ako and Fekrazad, Parviz and Akbarzadeh Baghban, Gholamreza and Hadi, Hanah and Heidari, Semko and Dogan, Mahîr and Asadi, Pedram and Bashir, Dashne and Ghodrati, Mohammad Amin and Amini, Kourosh and Ashourinezhad, Zeynab and Baladi, Mana… find
  78. Haig, Geoffrey and Stilo, Donald and Dogan, Mahîr C. and Schiborr, Nils. 2024. Word Order in Western Asia: A spoken-language-based corpus for investigating areal effects in word order variation. PID https://fd-repo.uni-bamberg.de/records/gyws0-g4218. ↗
  79. Kurdish Twitter Data. https://github.com/ftkurt/kurdish-twitter-data. ↗
  80. Matras, Yaron. 2016. The Dialects of Kurdish. PID https://www.kratylos.org/ raphael/kurdish/database/. ↗
  81. Guilherme Penedo and Hynek Kydlíček and Vinko Sabolčec and Bettina Messmer and Negar Foroutan and Amir Hossein Kargaran and Colin Raffel and Martin Jaggi and Leandro Von Werra and Thomas Wolf. 2025. FineWeb2. PID https://huggingface.co/datasets/HuggingFaceFW/fineweb-2. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schuler-2026-dialectal-filtering-synthesizing-kurdish-corpora/