Elamur Research⌕
☰
home / papers / 2016 / Arabic Dialect Identification in Speech Transcripts

Arabic Dialect Identification in Speech Transcripts

Paper · 2016 · Shervin Malmasi and Marcos Zampieri

Summary

The paper addresses the task of separating Modern Standard Arabic from four regional dialects (Egyptian, Gulf, Levantine, North African). The input is automatic speech recognition output from broadcast speech, not written text. It describes the MAZA entry to the Arabic sub-task of the DSL 2016 shared task (VarDial3). The training data comes from Ali et al. (2016) and has 7,619 sentences in Buckwalter transliteration. A separate test set of 1,540 sentences was used for official scoring.

Following Malmasi and Dras (2015b), the authors train one linear SVM per feature type, with no preprocessing. The feature types are character n-grams of orders 1–6 and word unigrams. They then combine these members in three ways: plurality voting, median probability and mean probability.

In cross-validation, character 4-grams were the strongest single feature, at about 66%. Every ensemble beat every individual model, and mean probability was best at 68%. Scores dropped sharply on the test set. Even so, mean-probability fusion reached a weighted F1 of 0.51 and ranked first of 18 closed-track teams. The result shows that combined simple linear models remain competitive for close varieties, and that dialect identification from transcripts is far from solved.

Key points

  • Training class sizes: EGY 1,578, GLF 1,672, LAV 1,758, MSA 999, NOR 1,612. The test majority baseline was 22.79%.
  • Single features in cross-validation:
    • Character 4-grams scored 65.95% and 5-grams 65.70%.
    • Character 4-, 5- and 6-grams all beat word unigrams.
    • The learning curve for 4-grams was still rising, so more data should help.
  • Ensembles in cross-validation scored 0.6755 for voting, 0.6782 for median and 0.6800 for mean, against a 0.2307 majority baseline. Voting tied on 344 samples (4.52%), and ties were broken arbitrarily.
  • Test results for run 3 (mean probability): accuracy 0.5117, weighted F1 0.5132, macro F1 0.5088. This is roughly 2 points above runs 1 and 2.
  • Per-class F1 on the test set was highest for MSA (0.61) and lowest for Gulf (0.35). The authors suspect the gap between cross-validation and test comes from the test set being drawn from a different source.

Interesting ideas

  • I1. Averaging class probabilities lets a model's second-ranked answer still add support, which hard voting throws away; this made mean fusion the best combiner here.
  • I2. Training one model per evidence family and fusing their outputs keeps each family's contribution separately inspectable.
  • I3. Reporting how often a voting combiner ties shows how much of the outcome is effectively arbitrary.
  • I4. A learning curve on the best single feature tells you whether collecting more data is worth it before building anything more complex.
  • I5. A large drop from cross-validation to the official test points to a shift in data source, not to the method itself.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When witness parsers build silver data, average their per-arc score distributions instead of counting votes. A parser that ranked the correct head second would then still contribute.
  • I2 (tokens): The tagger could keep separate scorers for suffix, word form and context, each producing a distribution before fusion. Each tag decision could then be explained by which evidence pushed it.
  • I3 (learn): Log the tie rate among witness parsers for each sentence. Drop or flag silver examples where agreement was settled by a coin flip.
  • I4 (tree): Plot gparse and perceptron LAS against the fraction of the treebank used for training. This shows whether extra silver or converted UD data is likely to pay off.
  • I5 (answer): Evaluate the readers on fables or school tests from a collection not used in development. Random splits of one source may overstate accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬German Research Centre for Artificial IntelligenceKaiserslautern · Germany77 works · 29,479 citations🏥Brigham and Women's HospitalBoston · United States3 works · 1,425 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Shervin Malmasi and Marcos Zampieri. 2016. Arabic Dialect Identification in Speech Transcripts. Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{malmasi2016arabic,
  title = {{Arabic Dialect Identification in Speech Transcripts}},
  author = {Shervin Malmasi and Marcos Zampieri},
  booktitle = {Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3)},
  year = {2016},
  url = {https://aclanthology.org/W16-4814/}
}

References

32 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rania Al-Sabbagh and Roxana Girju. 2012. YADAC: Yet another Dialectal Arabic Corpus. In Proceedings of LREC.
  2. Ahmed Ali, Najim Dehak, Patrick Cardinal, Sameer Khurana, Sree Harsha Yella, James Glass, Peter Bell, and Steve Renals. 2016. Automatic Dialect Detection in Arabic Broadcast Speech. In Proceedings of Interspeech.
  3. Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A Multidialectal Parallel Corpus of Arabic. In Proceedings of LREC.
  4. David Chiang, Mona T Diab, Nizar Habash, Owen Rambow, and Safiullah Shareef. 2006. Parsing Arabic Dialects. In Proceedings of EACL.
  5. Alina Maria Ciobanu and Liviu P. Dinu. 2016. A Computational Perspective on Romanian Dialects. In Proceedings of LREC. find
  6. Ryan Cotterell and Chris Callison-Burch. 2014. A Multi-dialect, Multi-genre Corpus of Informal Written Arabic. In Proceedings LREC.
  7. Kareem Darwish, Hassan Sajjad, and Hamdy Mubarak. 2014. Verifiably Effective Arabic Dialect Identification. In Proceedings of EMNLP.
  8. Heba Elfardy and Mona T Diab. 2013. Sentence Level Dialect Identification in Arabic. In Proceedings of ACL.
  9. Cyril Goutte, Serge Léger, and Marine Carpuat. 2014. The NRC System for Discriminating Similar Languages. In Proceedings of the VarDial Workshop. find
  10. Cyril Goutte, Serge Léger, Shervin Malmasi, and Marcos Zampieri. 2016. Discriminating Similar Languages: Evaluations and Explorations. In Proceedings of LREC. find
  11. Nizar Y Habash. 2010. Introduction to Arabic Natural Language Processing. Synthesis Lectures on Human Language Technologies, 3(1):1–187. ↗
  12. Josef Kittler, Mohamad Hatef, Robert PW Duin, and Jiri Matas. 1998. On combining classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(3):226–239. ↗
  13. Ludmila I Kuncheva. 2004. Combining Pattern Classifiers: Methods and Algorithms. John Wiley & Sons. find
  14. Ludmila I Kuncheva. 2014. Combining Pattern Classifiers: Methods and Algorithms. Wiley, second edition. find
  15. Nikola Ljubešić, Nives Mikelic, and Damir Boras. 2007. Language Identification: How to Distinguish Similar Languages? In Proceedings of the International Conference on Information Technology Interfaces. find
  16. Marco Lui and Paul Cook. 2013. Classifying English Documents by National Dialect. In Proceedings of ALTW.
  17. Shervin Malmasi and Mark Dras. 2015a. Automatic Language Identification for Persian and Dari Texts. In Proceedings PACLING. find
  18. Shervin Malmasi and Mark Dras. 2015b. Language Identification using Classifier Ensembles. In Proceedings of the LT4VarDial workshop. find
  19. Shervin Malmasi, Eshrag Refaee, and Mark Dras. 2015. Arabic Dialect Identification using a Parallel Multidialectal Corpus. In Proceedings of PACLING. find
  20. Shervin Malmasi, Mark Dras, and Marcos Zampieri. 2016a. LTG at SemEval-2016 Task 11: Complex Word Identification with Classifier Ensembles. In Proceedings of SemEval. find
  21. Shervin Malmasi, Marcos Zampieri, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, and Jörg Tiedemann. 2016b. Discriminating between Similar Languages and Arabic Dialect Identification: A Report on the Third DSL Shared Task. In Proceedings of the VarDial Workshop.
  22. Gustavo Henrique Paetzold and Lucia Specia. 2016. Semeval 2016 task 11: Complex Word Identification. Proceedings of SemEval, pages 560–569.
  23. Fatiha Sadat, Farnazeh Kazemi, and Atefeh Farzindar. 2014. Automatic Identification of Arabic Language Varieties and Dialects in Social Media. In Proceedings of the SocialNLP Workshop.
  24. Abdulhadi Shoufan and Sumaya Al-Ameri. 2015. Natural Language Processing for Dialectical Arabic: A Survey. In Proceedings of the Arabic NLP Workshop.
  25. Liling Tan, Marcos Zampieri, Nikola Ljubešic, and Jörg Tiedemann. 2014. Merging Comparable Data Sources for the Discrimination of Similar Languages: The DSL Corpus Collection. In Proceedings of the BUCC Workshop. find
  26. Christoph Tillmann, Saab Mansour, and Yaser Al-Onaizan. 2014. Improved Sentence-Level Arabic Dialect Classification. In Proceedings of the VarDial Workshop.
  27. Omar F Zaidan and Chris Callison-Burch. 2011. The Arabic Online Commentary Dataset: An Annotated Dataset of Informal Arabic with High Dialectal Content. In Proceedings of ACL.
  28. Omar F Zaidan and Chris Callison-Burch. 2014. Arabic Dialect Identification. Computational Linguistics. ↗
  29. Marcos Zampieri and Binyam Gebrekidan Gebre. 2012. Automatic Identification of Language Varieties: The Case of Portuguese. In Proceedings of KONVENS. find
  30. Marcos Zampieri, Liling Tan, Nikola Ljubešić, and Jörg Tiedemann. 2014. A Report on the DSL Shared Task 2014. In Proceedings of the VarDial Workshop.
  31. Marcos Zampieri, Liling Tan, Nikola Ljubešić, Jörg Tiedemann, and Preslav Nakov. 2015. Overview of the DSL Shared Task 2015. In Proceedings of the LT4VarDial Workshop.
  32. Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar F Zaidan, and Chris Callison-Burch. 2012. Machine translation of Arabic Dialects. In Proceedings of NAACL-HLT.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/malmasi-2016-arabic-dialect-identification-speech-transcripts/