Elamur Research⌕
☰
home / papers / 2022 / Optimizing Naive Bayes for Arabic Dialect Identification

Optimizing Naive Bayes for Arabic Dialect Identification

Paper · 2022 · Tommi Jauhiainen, Heidi Jauhiainen and Krister Lindén

Summary

This paper reports the SUKI team's entry in the NADI 2022 shared task. The task was to assign Arabic tweets to one of 18 country-level dialects, with macro F1 as the ranking metric. The authors wanted to know whether their shallow Naive Bayes identifier, which had won several other dialect-identification competitions, could compete on closely related Arabic varieties. The model scores text with log relative frequencies of character n-grams and charges a tuned penalty for unseen features. Their main engineering addition was an automatic optimizer that searches the n-gram range and the penalty value, replacing earlier semi-manual tuning. They also tried three other things: Farasa morphological segmentation, language model adaptation, and unsupervised topic clustering of the training tweets. Only segmentation helped on the development set. The team's one submitted run placed 18th of 19 teams, reaching a macro F1 of 0.1963 for test set A and 0.1058 for test set B. The result matters as an honest negative finding: on this task, character-level NB trails BERT-style models by a wide margin.

Key points

  • Data came from NADI 2022 labeled tweets: 20,398 for training, 4,871 for development, 4,758 in test A and 1,474 in test B. The classes are highly imbalanced, from 4,283 Egyptian tweets down to 215 each for Sudan, Qatar and Bahrain. The organizers also offered 10M unlabeled tweet IDs, of which about 2M had become unavailable. The team did not use the unlabeled data in the final run.
  • Development-set macro F1 rose with each tuning step:
    • manual tuning: 0.2046 (n-grams 2–4, penalty 1.40);
    • automatic optimizer: 0.2070 (1–4, penalty 1.4375);
    • optimizer with Farasa-segmented data: 0.2119 (1–4, penalty 1.375).
  • Language model adaptation and clustering did not beat the baseline. Clustering produced 214 clusters across the dialects, and the scores improved as clusters were merged back together.
  • Final ranks were 19/19 on test A and 15/19 on test B, for 18th place overall. The code is published as TunPRF-NADI on GitHub.

Interesting ideas

  • I1. The optimizer is a greedy neighbourhood search over hyperparameters. It keeps a cache of every configuration already tried and expands only around the current top ten. Penalty steps are halved between known neighbours, and the search stops once the top-ten list no longer changes. This gives cheap, reproducible tuning without a full grid.
  • I2. A 0.002 gap between manual and automatic results came down to one mismatch: test tweets were padded with boundary spaces, but training tweets were not. Preprocessing therefore has to be identical on both sides.
  • I3. Unseen features get a tunable penalty scaled by corpus size rather than a fixed smoothing constant. This turns out-of-vocabulary handling into an explicit, measurable parameter.
  • I4. The authors caution that their 2.4% gain from segmentation might simply be noise. They also hold back adaptation because base accuracy is too low for self-training to help. Both points argue against trusting small gains or bootstrapping from a weak model.
  • I5. The clustering builds pseudo-classes by repeatedly seeding a new model from the item farthest from all existing models. Models that claim only themselves are dropped as outliers.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova could tune parser or tagger knobs this way (beam width, feature cutoffs, rule thresholds), logging every trial so each chosen setting can be traced and explained.
  • I2 (tokens): Mova should enforce one shared normalization function for training and inference in the tokenizer and tagger, with a regression test that flags any divergence.
  • I3 (tokens): Mova could expose a similar tuned backoff weight for unknown words in morphology and lemma lookup, so OOV behaviour is visible and adjustable rather than hidden.
  • I4 (learn): Mova's silver-data and self-repair loops should only switch on above a base-accuracy threshold, and small improvements should be checked with negative controls before they are adopted.
  • I5 (world): Mova could use a similar farthest-first grouping of texts or entity mentions to discover sub-domains in its knowledge graph, keeping the seed items as human-readable cluster explanations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of HelsinkiHelsinki · Finland50 works · 5,318 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Tommi Jauhiainen, Heidi Jauhiainen and Krister Lindén. 2022. Optimizing Naive Bayes for Arabic Dialect Identification. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.40

@inproceedings{jauhiainen2022optimizing,
  title = {{Optimizing Naive Bayes for Arabic Dialect Identification}},
  author = {Tommi Jauhiainen, Heidi Jauhiainen and Krister Lindén},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.40},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.40}
}

References

19 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for Arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16, San Diego, California. Association for Computational Linguistics.
  2. Bharathi Raja Chakravarthi, Mihaela Gaman, Radu Tudor Ionescu, Heidi Jauhiainen, Tommi Jauhiainen, Krister Lindén, Nikola Ljubešić, Niko Partanen, Ruba Priyadharshini, Christoph Purschke, Eswari Rajagopal, Yves Scherrer, and Marcos Zampieri. 2021. Findings of the VarDial evaluation campaign 2021. In Proceedings of the Eighth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 1– 11, Kyiv, Ukraine. Association for Computational Linguistics. find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  4. Abdellah El Mekki, Ahmed Alami, Hamza Alami, Ahmed Khoumsi, and Ismail Berrada. 2020. Weighted combination of BERT and n-GRAM features for nuanced Arabic dialect identification. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 268–274, Barcelona, Spain (Online). Association for Computational Linguistics.
  5. Ashraf Elnagar, Sane M. Yagi, Ali Bou Nassif, Ismail Shahin, and Said A. Salloum. 2021. Systematic Literature Review of Dialectal Arabic: Identification and Detection. IEEE Access, 9:31010–31042. ↗
  6. Tommi Jauhiainen, Heidi Jauhiainen, Tero Alstola, and Krister Lindén. 2019a. Language and Dialect Identification of Cuneiform Texts. In Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 89–98, Ann Arbor, Michigan. Association for Computational Linguistics. find
  7. Tommi Jauhiainen, Heidi Jauhiainen, and Krister Lindén. 2019b. Discriminating between Mandarin Chinese and Swiss-German varieties using adaptive language models. In Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 178–187, Ann Arbor, Michigan. Association for Computational Linguistics. find
  8. Tommi Jauhiainen, Heidi Jauhiainen, and Krister Lindén. 2021. Naive Bayes-based experiments in Romanian dialect identification. In Proceedings of the Eighth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 76–83, Kyiv, Ukraine. Association for Computational Linguistics. find
  9. Tommi Jauhiainen, Heidi Jauhiainen, and Krister Lindén. 2022a. Italian language and dialect identification and regional French variety detection using adaptive naive bayes. In Proceedings of the Ninth Workshop on NLP for Similar Languages, Varieties and Dialects, Gyeongju, Republic of Korea. find
  10. Tommi Jauhiainen, Marco Lui, Marcos Zampieri, Timothy Baldwin, and Krister Lindén. 2019d. Automatic Language Identification in Texts: A Survey. Journal of Artificial Intelligence Research, 65:675–782. ↗
  11. Tommi Jauhiainen, Jussi Piitulainen, Erik Axelson, and Krister Lindén. 2022b. Language identification as part of the text corpus creation pipeline at the language bank of finland. In Digital Humanities in Nordic and Baltic Countries conference (DHNB 2022). ↗
  12. Francisco Rangel, Paolo Rosso, Martin Potthast, and Benno Stein. 2017. Overview of the 5th Author Profiling Task at PAN 2017: Gender and Language Variety Identification in Twitter. In Working Notes Papers of CLEF 2017 Evaluation Labs and Workshop, Dublin, Ireland. CEUR-WS.org. find
  13. Bashar Talafha, Mohammad Ali, Muhy Eddin Za’ter, Haitham Seelawi, Ibraheem Tuffaha, Mostafa Samir, Wael Farhan, and Hussein Al-Natsheh. 2020. Multi-dialect Arabic BERT for country-level dialect identification. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 111–118, Barcelona, Spain (Online). Association for Computational Linguistics. ↗
  14. Anshul Wadhawan. 2021. Dialect identification in nuanced Arabic tweets using farasa segmentation and AraBERT. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 291–295, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  15. Marcos Zampieri, Shervin Malmasi, Yves Scherrer, Tanja Samardžić, Francis Tyers, Miikka Silfverberg, Natalia Klyueva, Tung-Le Pan, Chu-Ren Huang, Radu Tudor Ionescu, Andrei M. Butnaru, and Tommi Jauhiainen. 2019. A report on the third VarDial evaluation campaign. In Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 1–16, Ann Arbor, Michigan. Association for Computational Linguistics. find
  16. Tong Zhang. 2004. Solving large scale linear prediction problems using stochastic gradient descent algorithms. In Proceedings of the twenty-first international conference on Machine learning, page 116. find
  17. Tommi Jauhiainen, Krister Lindén, and Heidi Jauhiainen. 2016. HeLI, a word-based backoff method for language identification. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages 153–162, Osaka, Japan. find
  18. The COLING 2016 Organizing Committee. find
  19. Tommi Jauhiainen, Krister Lindén, and Heidi Jauhiainen. 2019c. Language model adaptation for language and dialect identification of text. Natural Language Engineering, 25(5):561–583. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jauhiainen-2022-optimizing-naive-bayes-arabic-dialect/