Elamur Research⌕
☰
home / papers / 2018 / UnibucKernel Reloaded: First Place in Arabic Dialect Identification for the Second Year in a Row

UnibucKernel Reloaded: First Place in Arabic Dialect Identification for the Second Year in a Row

Paper · 2018 · Andrei Butnaru and Radu Tudor Ionescu

Summary

The paper describes UnibucKernel, the system that won the closed Arabic Dialect Identification (ADI) task at VarDial 2018. The task is a five-way classification that separates Modern Standard Arabic from four Arabic dialects in broadcast speech. Organizers supplied three kinds of input: ASR speech transcripts, phonetic transcripts, and audio-derived dialectal embeddings.

The authors build on their 2017 winning system. They represent each view with a kernel:

  • character p-gram string kernels (presence bits, histogram intersection, and a Local Rank Distance–based kernel) on both transcript types;
  • an RBF kernel on the audio embeddings.

They then sum these kernels and train Kernel Ridge Regression, which did slightly better than Kernel Discriminant Analysis in development tests.

The best run reached 58.65% accuracy and 58.92% macro-F1 on the test set. Organizer tests found this significantly better than the runner-up (57.59%). With audio embeddings released after the competition, macro-F1 rose to 62.28%.

The work shows that a shallow, carefully tuned kernel combination over several complementary views can beat other approaches on a noisy dialect task.

Key points

  • 2018 ADI closed task: MSA plus four dialects, 6 participating teams, ranked by macro-F1. Run 1 scored 58.92% macro-F1 and 58.65% accuracy and finished first. A random baseline gets 19.95% macro-F1.
  • The kernels are p-grams presence bits, histogram intersection, LRD (blended over p-gram ranges such as 3–6), and an RBF kernel on audio embeddings from Shon et al. (2018). The combination is a plain sum of kernels, with KRR at regularization 10⁻³.
  • On the development set:
    • speech-transcript kernels alone reached 55.17% accuracy and 54.72% macro-F1;
    • adding the audio kernel raised this to 59.52% accuracy and 59.12% macro-F1;
    • adding phonetic-transcript kernels gave roughly another 1% and pushed KRR past 60%.
  • Two single-change variants did worse on the test set. Using non-squared kernels (run 2) gave 54.91% macro-F1. Lowering KRR regularization to 10⁻⁴ (run 3) gave 58.55%. Post-competition runs with new audio embeddings reached 62.02% (run 4) and 62.28% (run 5).
  • The confusion matrix shows over 400 Levantine samples labeled as Maghrebi. Levantine–Egyptian and Egyptian–Maghrebi were also frequently confused.

Interesting ideas

  • I1. Each input view (ASR text, phonetic text, audio) gets its own similarity kernel, and the kernels are simply summed. Complementary but noisy evidence is fused without a learned fusion layer, and each view's contribution can be measured separately.
  • I2. Character p-grams that record only presence rather than frequency proved strong on noisy, error-laden transcripts. Presence bits make the model robust to repetition and to spelling or ASR noise.
  • I3. The authors added one input view at a time and reported the gain at each step: speech, then audio (+~4–5%), then phonetic (+~1%). This makes clear which resource actually pays off.
  • I4. Results were checked with a significance test against the runner-up and compared with a random baseline, rather than reported as raw score differences. A small margin (58.92 vs 57.59) becomes a defensible claim.
  • I5. A confusion matrix exposed a specific, lopsided error: Levantine samples pulled toward Maghrebi. That points to where the model fails, which an aggregate score hides.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): witness parsers' agreement on silver data could be scored per view (lexical, morphological, structural) and summed transparently, so the contribution of each witness stays inspectable.
  • I2 (tokens): presence-bit character n-gram signatures could back up the tagger and lemmatizer on unknown or misspelled words, giving an explainable guess about which known words a form resembles.
  • I3 (learn): new knowledge sources or repair rules could be admitted through the same stepwise ablation on held-out data, keeping only those with a measurable gain over a baseline.
  • I4 (learn): regression checks between parser or reader versions could require a significance test plus a chance-level negative control before a change counts as an improvement.
  • I5 (tree): per-label confusion tables for dependency relations, for example by treebank or English variety, could show systematic attachment or label swaps worth targeting with repair rules.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Andrei Butnaru and Radu Tudor Ionescu. 2018. UnibucKernel Reloaded: First Place in Arabic Dialect Identification for the Second Year in a Row. Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{butnaru2018unibuckernel,
  title = {{UnibucKernel Reloaded: First Place in Arabic Dialect Identification for the Second Year in a Row}},
  author = {Andrei Butnaru and Radu Tudor Ionescu},
  booktitle = {Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018)},
  year = {2018},
  url = {https://aclanthology.org/W18-3909/}
}

References

41 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Ali, Najim Dehak, Patrick Cardinal, Sameer Khurana, Sree Harsha Yella, James Glass, Peter Bell, and Steve Renals. 2016. Automatic dialect detection in arabic broadcast speech. In Proceedings of Interspeech, pages 2934–2938.
  2. Ahmed Ali, Stephan Vogel, and Steve Renals. 2017. Speech Recognition Challenge in the Wild: Arabic MGB-3. arXiv preprint arXiv:1709.07276. ↗
  3. Fadi Biadsy, Julia Hirschberg, and Nizar Habash. 2009. Spoken Arabic Dialect Identification Using Phonotactic Modeling. In Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages, pages 53–61, Stroudsburg, PA, USA. Association for Computational Linguistics.
  4. Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A Multidialectal Parallel Corpus of Arabic. In Proceedings of LREC, pages 1240–1245. European Language Resources Association (ELRA), May.
  5. Mădălina Cozma, Andrei Butnaru, and Radu Tudor Ionescu. 2018. Automated essay scoring with string kernels and word embeddings. In Proceedings of ACL. find
  6. Kareem Darwish, Hassan Sajjad, and Hamdy Mubarak. 2014. Verifiably Effective Arabic Dialect Identification. In Proceedings of EMNLP, pages 1465–1468. ACL.
  7. Liviu P. Dinu, Radu Tudor Ionescu, and Alexandru I. Tomescu. 2014. A rank-based sequence aligner with applications in phylogenetic analysis. PLoS ONE, 9(8):e104006, 08. ↗
  8. Heba Elfardy and Mona T. Diab. 2013. Sentence Level Dialect Identification in Arabic. In Proceedings of ACL, pages 456–461. Association for Computer Linguistics.
  9. Hugo Jair Escalante, Thamar Solorio, and Manuel Montes-y-Gómez. 2011. Local Histograms of Character Ngrams for Authorship Attribution. In Proceedings of ACL: HLT, volume 1, pages 288–298. find
  10. Rosa M. Giménez-Pérez, Marc Franco-Salvador, and Paolo Rosso. 2017. Single and Cross-domain Polarity Classification using String Kernels. In Proceedings of EACL, pages 558–563, April. find
  11. Trevor Hastie and Robert Tibshirani. 2003. The Elements of Statistical Learning. Springer, corrected edition, July. find
  12. David Haussler. 1999. Convolution Kernels on Discrete Structures. Technical Report UCS-CRL-99-10, University of California at Santa Cruz, Santa Cruz, CA, USA. find
  13. Radu Tudor Ionescu and Andrei Butnaru. 2017. Learning to Identify Arabic and German Dialects using Multiple Kernels. In Proceedings of VarDial Workshop of EACL, pages 200–209.
  14. Radu Tudor Ionescu and Marius Popescu. 2016a. Knowledge Transfer between Computer Vision and Text Mining. Advances in Computer Vision and Pattern Recognition. Springer International Publishing. ↗
  15. Radu Tudor Ionescu and Marius Popescu. 2016b. UnibucKernel: An Approach for Arabic Dialect Identification based on Multiple String Kernels. In Proceedings of VarDial Workshop of COLING, pages 135–144.
  16. Radu Tudor Ionescu and Marius Popescu. 2017. Can string kernels pass the test of time in native language identification? In Proceedings of BEA-12 Workshop of EMNLP, pages 224–234. find
  17. Radu Tudor Ionescu, Marius Popescu, and Aoife Cahill. 2014. Can characters reveal your native language? A language-independent approach to native language identification. In Proceedings of EMNLP, pages 1363–1373. Association for Computational Linguistics, October. find
  18. Radu Tudor Ionescu, Marius Popescu, and Aoife Cahill. 2016. String kernels for native language identification: Insights from behind the curtains. Computational Linguistics, 42(3):491–525. ↗
  19. Radu Tudor Ionescu. 2013. Local Rank Distance. In Proceedings of SYNASC, pages 221–228, Timisoara, Romania. IEEE Computer Society. find
  20. Radu Tudor Ionescu. 2015. A Fast Algorithm for Local Rank Distance: Application to Arabic Native Language Identification. In Proceedings of ICONIP, volume 9490, pages 390–400. Springer LNCS.
  21. Tommi Jauhiainen, Marco Lui, Marcos Zampieri, Timothy Baldwin, and Krister Lindén. 2018. Automatic language identification in texts: A survey. arXiv preprint arXiv:1804.08186. ↗
  22. Rohit J. Kate and Raymond J. Mooney. 2006. Using String-kernels for Learning Semantic Parsers. In Proceedings of ACL, pages 913–920, Stroudsburg, PA, USA. Association for Computational Linguistics.
  23. Huma Lodhi, Craig Saunders, John Shawe-Taylor, Nello Cristianini, and Christopher J. C. H. Watkins. 2002. Text Classification using String Kernels. Journal of Machine Learning Research, 2:419–444. find
  24. Subhransu Maji, Alexander C. Berg, and Jitendra Malik. 2008. Classification using intersection kernel support vector machines is efficient. In Proceedings of CVPR, pages 1–8. IEEE Computer Society. find
  25. Shervin Malmasi and Marcos Zampieri. 2016. Arabic Dialect Identification in Speech Transcripts. In Proceedings of VarDial Workshop of COLING, pages 106–113. ACL Anthology.
  26. Shervin Malmasi and Marcos Zampieri. 2017. Arabic Dialect Identification Using iVectors and ASR Transcripts. In Proceedings of VarDial Workshop of EACL, pages 178–183, Valencia, Spain, April.
  27. Shervin Malmasi, Eshrag Refaee, and Mark Dras. 2015. Arabic Dialect Identification using a Parallel Multidialectal Corpus. In Proceedings of PACLING, pages 209–217, Bali, Indonesia, May. find
  28. Shervin Malmasi, Marcos Zampieri, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, and Jörg Tiedemann. 2016. Discriminating between Similar Languages and Arabic Dialect Identification: A Report on the Third DSL Shared Task. In Proceedings of VarDial Workshop of COLING, Osaka, Japan.
  29. Maryam Najafian, Sameer Khurana, Suwon Shon, Ahmed Ali, and James Glass. 2018. Exploiting convolutional neural networks for phonotactic based dialect identification. In Proceedings of ICASSP, pages 5174–5175. find
  30. Marius Popescu and Cristian Grozea. 2012. Kernel methods and string kernels for authorship analysis. In Pamela Forner, Jussi Karlgren, and Christa Womser-Hacker, editors, CLEF (Online Working Notes/Labs/Workshop), Rome, Italy, September. find
  31. Marius Popescu and Radu Tudor Ionescu. 2013. The Story of the Characters, the DNA and the Native Language. In Proceedings of BEA-8 Workshop of NAACL, pages 270–278, Atlanta, Georgia, June. Association for Computational Linguistics. find
  32. Marius Popescu, Cristian Grozea, and Radu Tudor Ionescu. 2017. HASKER: An efficient algorithm for string kernels. Application to polarity classification in various languages. In Proceedings of KES, pages 1755–1763. find
  33. Francisco Rangel, Paolo Rosso, Martin Potthast, and Benno Stein. 2017. Overview of the 5th author profiling task at pan 2017: Gender and language variety identification in twitter. Working Notes Papers of the CLEF. find
  34. Conrad Sanderson and Simon Guenter. 2006. Short text authorship attribution via sequence kernels, markov chains and author unmasking: An investigation. In Proceedings of EMNLP, pages 482–491, Sydney, Australia, July. Association for Computational Linguistics. find
  35. John Shawe-Taylor and Nello Cristianini. 2004. Kernel Methods for Pattern Analysis. Cambridge University Press. ↗
  36. Suwon Shon, Ahmed Ali, and James Glass. 2018. Convolutional Neural Networks and Language Embeddings for End-to-End Dialect Recognition. In Proceedings of Odyssey. ↗
  37. Andrea Vedaldi and Andrew Zisserman. 2010. Efficient additive kernels via explicit feature maps. In Proceedings of CVPR, pages 3539–3546, San Francisco, CA, USA. IEEE Computer Society. ↗
  38. Omar F. Zaidan and Chris Callison-Burch. 2011. The Arabic Online Commentary Dataset: An Annotated Dataset of Informal Arabic with High Dialectal Content. In Proceedings of ACL: HLT, volume 2, pages 37–41, Stroudsburg, PA, USA. Association for Computational Linguistics.
  39. Omar F. Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗
  40. Marcos Zampieri, Shervin Malmasi, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, Jörg Tiedemann, Yves Scherrer, and Noëmi Aepli. 2017. Findings of the VarDial Evaluation Campaign 2017. In Proceedings of VarDial Workshop of EACL, Valencia, Spain.
  41. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Ahmed Ali, Suwon Shon, James Glass, Yves Scherrer, Tanja Samardžić, Nikola Ljubešić, Jörg Tiedemann, Chris van der Lee, Stefan Grondelaers, Nelleke Oostdijk, Antal van den Bosch, Ritesh Kumar, Bornini Lahiri, and Mayank Jain. 2018. Language Identification and Morphosyntactic Tagging: The Second VarDial Evaluation Campaign. In Proceedings of VarDial Workshop of COLING, Santa Fe, USA.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/butnaru-2018-unibuckernel-reloaded-first-place-arabic/