Elamur Research⌕
☰
home / papers / 2019 / The SMarT Classifier for Arabic Fine-Grained Dialect Identification

The SMarT Classifier for Arabic Fine-Grained Dialect Identification

Paper · 2019 · Karima Meftouh, Karima Abidi, Salima Harrat et al.

Summary

The paper describes the SMarT group's entry to subtask 1 of the MADAR shared task. The task was to identify, at the sentence level, which of 25 Arab city dialects or Modern Standard Arabic a sentence is written in. The authors trained only on the official Corpus-26 and Corpus-6 data. Two neural options, an LSTM and a Word2vec-based model, did poorly on such a small corpus. They therefore used a Multinomial Naïve Bayes classifier in scikit-learn with Tf-Idf weighting and a tuned smoothing value. Its features were word unigrams and bigrams, two kinds of character 1–5-grams (word-boundary space padding on or off), and likelihood scores from 26 per-class unigram language models. Character n-grams were consistently stronger than word n-grams. The submitted system reached a macro-averaged F1 of 67.31% and a macro accuracy of 67.73% on the test set, which ranked first of 19 participants. The result shows that a carefully engineered, transparent n-gram model can beat neural approaches when the dialect classes are very close and training data is scarce.

Key points

  • Data:
    • Corpus-26 holds parallel sentences in 25 city dialects plus MSA, with 1600 training and 200 development sentences per class.
    • Corpus-6 adds 9,000 training and 1,000 development sentences each for Beirut, Cairo, Doha, Tunis, Rabat and MSA.
  • Rejected baselines on Corpus-26 (macro F1):
    • Word2vec reached 49.74.
    • LSTM reached 58.33.
  • Classifier setup:
    • Multinomial Naïve Bayes was used with smoothing fixed at 0.093 for all systems.
    • Punctuation was stripped from all data.
    • A special symbol marked the start of each sentence.
  • Feature results:
    • Word n-grams peaked at unigrams plus bigrams, and higher orders suffered from sparsity.
    • Character 1–5-grams beat word n-grams by at least 3 F1 points.
  • Final scores:
    • The best combination (word 1–2 + char 1–5 with and without spaces) scored 69.06 dev / 67.34 test.
    • Adding the 26 LM likelihoods gave 69.09 dev / 67.31 test, and this slightly worse-on-test version was the one submitted.

Interesting ideas

  • I1. Extracting character n-grams twice, once with word edges padded by spaces and once without, lets one model see both word-internal fragments and prefix/suffix boundary cues; the two views together beat either alone.
  • I2. The authors scored each sentence with one small language model per class and fed those likelihoods into the main classifier as compact, interpretable extra features.
  • I3. They tried neural models first and dropped them once a simple Naïve Bayes model proved stronger on the limited data.
  • I4. Their own numbers show the LM features helping on dev while slightly hurting on test, so tiny dev gains can be noise rather than real improvement.
  • I5. They tuned a single smoothing constant and kept it fixed across all feature configurations, so the comparison between feature sets stayed clean.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The tagger and lemmatizer could expose boundary-padded and unpadded character n-grams as separate explainable features for guessing the POS and morphology of unknown words.
  • I2 (tokens): Per-variety unigram LMs (e.g., archaic fable English vs. modern school text) could give a cheap, inspectable register signal before tagging and parsing.
  • I3 (learn): Every new learned component should be evaluated against a plain n-gram/NB baseline, so that a complex model only enters the pipeline if it actually beats the simple control.
  • I4 (learn): Repair rules or features that improve silver/dev scores by only a hair should be confirmed on a held-out set before adoption.
  • I5 (learn): When comparing parser or tagger feature sets, freeze shared hyperparameters so that score differences can be attributed to the features themselves.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Laboratoire Lorrain de Recherche en Informatique et ses ApplicationsNancy · France30 works · 3,899 citations🎓Université de LorraineNancy · France20 works · 945 citations

Also on the byline: Badji Mokhtar-Annaba University · École Normale Supérieure de Bouzareah Alger.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Karima Meftouh, Karima Abidi, Salima Harrat and Kamel Smaili. 2019. The SMarT Classifier for Arabic Fine-Grained Dialect Identification. Proceedings of the Fourth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-4633

@inproceedings{meftouh2019smart,
  title = {{The SMarT Classifier for Arabic Fine-Grained Dialect Identification}},
  author = {Karima Meftouh, Karima Abidi, Salima Harrat and Kamel Smaili},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Workshop},
  year = {2019},
  doi = {10.18653/v1/w19-4633},
  url = {https://doi.org/10.18653/v1/w19-4633}
}

References

21 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, Mona Diab, and Sandra Kbler. 2014. SAMAR: Subjectivity and sentiment analysis for Arabic social media. Computer Speech & Language, 28:2037. ↗
  2. Heba Elfardy and Mona Diab. 2013. Sentence Level Dialect Identification in Arabic. In ACL (2), pages 456–461.
  3. Cyril Goutte, Serge LÉger, and Marine Carpuat. 2014. The nrc system for Discriminating Similar Languages. In Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects, pages 139–145. find
  4. Mohamed Al-Badrashiny and Mona Diab. 2016. Lili: A Simple Language Independent Approach for Language Identification. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1211– 1219. find
  5. Nizar Habash, Owen Rambow, Mona Diab, and Reem Kanjawi-Faraj. 2008. Guidelines for annotation of Arabic dialectness. In Proceedings of the Lrec workshop on hlt and nlp within the Arabic world, pages 49–53. find
  6. Mohamed Al-Badrashiny, Heba Elfardy, and Mona Diab. 2015. Aida2: A hybrid approach for token and sentence level dialect identification in arabic. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning, pages 42– 51.
  7. Salima Harrat, Karima Meftouh, and Kamel Smaı̈li. 2017. Machine translation for Arabic dialects (survey). Information Processing and Management. ↗
  8. Brigitte Bigi, Armelle Brun, Jean-Paul Haton, Kamel Smaı̈li, and Imed Zitouni. 2001. A comparative study of Topic Identification on Newspaper and Email. In Proceedings of the 8th International Symposium on String Processing and Information Retrieval - SPIRE’01, pages 238–241, Laguna de San Rafael, Chili. Shervin Malmasi, Eshrag. Refaee, and Mark. find
  9. Dras. 2015. Arabic dialect identification using a parallel multidialectal corpus. In In International Conference of the Pacific Association for Computational Linguistics, pages 35–53. Springer. find
  10. Andrew McCallum and Kamal Nigam. 1998. A comparison of event models for naive bayes text classification. In IN AAAI-98 WORKSHOP ON LEARN-ING FOR TEXT CATEGORIZATION, pages 41–48. AAAI Press.
  11. Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A Multidialectal Parallel Corpus of Arabic. In Proceedings of the Language Resources and Evaluation Conference, LREC-2014, pages 1240– 1245.
  12. Karima Meftouh, Salima Harrat, and Kamel Smaı̈li. 2018. PADIC: extension and new experiments. In 7th International Conference on Advanced Technologies ICAT, Antalya, Turkey. find
  13. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In ICLR (Workshop). Fabian. Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent.
  14. Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. find
  15. Toqir Rana, Yu-N Cheah, and Sukumar Letchmunan. 2016. Topic modeling in sentiment analysis: A systematic review. Journal of ICT Research and Applications, 10:76–93. ↗
  16. Motaz Saad, David Langlois, and Kamel Smaı̈li. 2013. Comparing Multilingual Comparable Articles Based On Opinions. In Proceedings of the 6th Workshop on Building and Using Comparable Corpora, pages 105–111, Sofia, Bulgaria. Association for Computational Linguistics ACL. find
  17. Fatiha Sadat, Farzindar Kazemi, and Atef Farzindar. 2014. Automatic identification of arabic dialects in social media. In In Proceedings of the first international workshop on Social media retrieval and analysis, page 3540. ACM. find
  18. Hasim Sak, Andrew Senior, and Franoise Beaufays. 2015. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In Google, USA. find
  19. Mohamed Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained Arabic Dialect Identification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1332–1344. Association for Computational Linguistics.
  20. Karen Spärck Jones. 1972. A Statistical Interpretation of Term Specificity and Its Application in Retrieval. Journal of Documentation, 28:11–21. ↗
  21. Omar Zaidan and Chris Callison-Burch. 2012. Arabic Dialect Identification. Association for Computational Linguistics, Volume 1, pages 1–35. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/meftouh-2019-smart-classifier-arabic-fine-grained/