Elamur Research⌕
☰
home / papers / 2019 / ZCU-NLP at MADAR 2019: Recognizing Arabic Dialects

ZCU-NLP at MADAR 2019: Recognizing Arabic Dialects

Paper · 2019 · Pavel Přibáň and Stephen Taylor

Summary

The paper describes the University of West Bohemia entry to the MADAR shared task on fine-grained Arabic dialect identification. Subtask 1 (S-1) assigns a sentence to one of 25 city dialects or MSA. Subtask 2 (S-2) infers a Twitter user's country, out of 21, from their tweets.

For S-1 the authors used only organizer data: about 41K training and 5.2K development sentences. They built one 5-gram kenlm language model per dialect, in both word and character versions. Each model's score for a sentence becomes a feature, and these features feed conventional classifiers. Their best system was a layered sklearn voting ensemble that mixes language-model features with n-gram naive Bayes and an SVM. A small MLP over the 26 language-model scores came close, and BiLSTM variants did clearly worse.

The ensemble reached 0.658 macro F1, ranking 6th of 19. The S-2 system, a k-nearest-neighbour classifier with per-user voting, scored 0.475 and ranked 7th.

The work shows that per-class language-model scores are a compact, strong signal for telling closely related varieties apart.

Key points

  • S-1 language models: 26 per-dialect kenlm models of order 5, in word and character variants. For the character models, characters were separated by spaces and each word was wrapped in <w></w> markers.

  • Tortuous Classifier: a hard-voting combination of three parts:

    • a soft-voting ensemble of naive Bayes models on word 1–2grams, char 3–5grams and language-model scores;
    • an SVM with a degree-2 polynomial kernel;
    • a multinomial naive Bayes model combining language-model and TF-IDF features.

    It scored 0.658 macro F1 on test.

  • Neural Network Classifier: an MLP with one 400-unit ReLU hidden layer over the 26 language-model probabilities. It scored 0.661 macro F1 on dev and 0.648 on test. Adding char n-gram BiLSTM inputs performed worse, with a best dev score of 0.555 when tested on their own.

  • S-2 system: kenlm scores for the 21 countries were exponentiated and normalised, then classified with KNeighborsClassifier (k=31). Each user received the country most often predicted for their tweets. Roughly 10% of train/dev tweets and 12% of test tweets could no longer be downloaded.

  • Error analysis: the confusion matrix shows errors that make geographic sense, for example among Egyptian cities (Aswan, Alexandria, Cairo) and among Levantine cities.

Interesting ideas

  • I1. Instead of feeding raw n-grams to a classifier, train one language model per class and use the vector of model scores as a small, interpretable feature set; each feature reads as how much the text resembles a given class.
  • I2. In character-level models, mark word edges with explicit begin/end tags rather than a space symbol, so prefixes and suffixes are visible without spending n-gram context on neighbouring words.
  • I3. Wrap a non-standard predictor (here, a folder of language models) behind a standard fit/predict/predict-proba interface, so it plugs directly into generic voting ensembles.
  • I4. Make decisions at two levels: classify each item, then take a plurality over all items belonging to one entity (here, a user).
  • I5. Data cleaning that improved per-tweet accuracy by about 5% actually lowered per-user accuracy, which shows that gains must be measured at the level of the final decision.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Per-treebank or per-genre language models could score incoming text, so Mova can explain which UD treebank style or register the input resembles before choosing parser settings.
  • I2 (tokens): A character model with word-boundary tags could help the morphology and lemma guesser for unknown words, since English affixes sit at word edges.
  • I3 (learn): Giving every witness parser one common scoring interface would make agreement voting for silver data uniform and easy to extend with new witnesses.
  • I4 (world): Sentence-level guesses, such as who the narrator is or which time frame applies, could be pooled across a whole text into one document-level decision that can be traced back to its votes.
  • I5 (learn): Evaluate repair rules and data filters on end answers, not only on intermediate metrics, and treat improvements that do not transfer as warnings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of West Bohemia in PilsenPilsen · Czechia14 works · 38,760 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Pavel Přibáň and Stephen Taylor. 2019. ZCU-NLP at MADAR 2019: Recognizing Arabic Dialects. Proceedings of the Fourth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-4623

@inproceedings{priban2019zcu,
  title = {{ZCU-NLP at MADAR 2019: Recognizing Arabic Dialects}},
  author = {Pavel Přibáň and Stephen Taylor},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Workshop},
  year = {2019},
  doi = {10.18653/v1/w19-4623},
  url = {https://doi.org/10.18653/v1/w19-4623}
}

References

18 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martı́n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org. find
  2. Alex Graves and Jürgen Schmidhuber. 2005. Frame-wise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, 18(5-6):602–610. find
  3. Abualsoud Hanani, Hanna Basha, Yasmeen Sharaf, and Stephen Taylor. 2015. Palestinian Arabic regional accent recognition. In The 8th International Conference on Speech Technology and Human-Computer Dialogue. find
  4. Abualsoud Hanani, Martin J. Russell, and Michael J. Carey. 2013. Human and computer recognition of regional accents and ethnic groups from British english speech. Computer Speech and Language, 27(1):5974. ↗
  5. Ahmed Ali, Najim Dehak, Patrick Cardinal, Sameer Khurana, Sree Harsha Yella, James Glass, Peter Bell, and Steve Renals. 2016. Automatic dialect detection in Arabic broadcast speech. In Proceedings of Interspeech 2016, pages 2934–2938.
  6. Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, , and Philipp Koehn. 2013. Scalable modified kneser-ney language model estimation. In ACL.
  7. Radu Tudor Ionescu and Marius Popescu. 2016. UnibucKernel: An Approach for Arabic Dialect Identification Based on Multiple String Kernels. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages 135–144, Osaka, Japan.
  8. Fadi Biadsy, Julia Hirschberg, and Nizar Habash. 2009. Spoken arabic dialect identification using phonotactic modeling. In Proceedings of the EACL Workshop on Computational Approaches to Semitic Languages, pages 53–61.
  9. Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A multidialectal parallel corpus of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC14). European Language Resources Association (ELRA).
  10. Diederik P. Kingma and Jimmy Lei Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980v9. ↗
  11. Shervin Malmasi, Eshrag Refaee, and Mark Dras. 2015. Arabic dialect identification using a parallel multidialectal corpus. In Proceedings of the 14th Conference of the Pacific Association for Computational Linguistics (PACLING 2015), pages 209–217, Bali, Indonesia. find
  12. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The madar arabic dialect corpus and lexicon. In Proceedings of the 11th International Conference on Language Resources and Evaluation. find
  13. Shervin Malmasi, Marcos Zampieri, Nikola Ljubei, Preslav Nakov, Ahmed Ali, and Jrg Tiedemann. 2016. Discriminating between similar languages and arabic dialect identification: A report on the third dsl shared task. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2016).
  14. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained arabic dialect identification. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 1332–1344, Santa Fe, New Mexico, USA.
  15. Omar F. Zaidan and Chris Callison-Burch. 2011. The Arabic Online Commentary dataset: An annotated dataset of informal Arabic with high dialectal content. In Proceedings of ACL, pages 37–41.
  16. Omar F. Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗
  17. Marcos Zampieri, Shervin Malmasi, Nikola Ljubei, Preslav Nakov, Ahmed Ali, Jrg Tiedemann, Yves Scherrer, and Nomi Aepli. 2017. Findings of the vardial evaluation campaign 2017. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2017.
  18. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Ahmed Ali, Suwon Shon, James Glass, Yves Scherrer, Tanja Samardžić, Nikola Ljubešić, Jörg Tiedemann, Chris van der Lee, Stefan Grondelaers, Nelleke Oostdijk, Dirk Speelman, Antal van den Bosch, Ritesh Kumar, Bornini Lahiri, and Mayank Jain. 2018. Language identification and morphosyntactic tagging: The second VarDial evaluation campaign. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pages 1–17, Santa Fe, New Mexico, USA. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/priban-2019-zcu-nlp-madar-2019-recognizing/