Elamur Research⌕
☰
home / papers / 2019 / Arabic Dialect Identification for Travel and Twitter Text

Arabic Dialect Identification for Travel and Twitter Text

Paper · 2019 · Pruthwik Mishra and Vandan Mujadia

Summary

This paper describes the IIIT Hyderabad entry to the MADAR Shared Task on fine-grained Arabic dialect identification at WANLP 2019. There were two subtasks. The first was to label travel-domain sentences with one of 26 city-level dialects (MADAR-Corpus-26). The second was to infer a Twitter user's country from that user's tweets. The authors built on the Salameh and Bouamor (2018) feature set. It combines word and character n-gram TF-IDF vectors with scores from per-dialect word and character 5-gram language models trained on MADAR-Corpus-26 and MADAR-Corpus-6. They fed these features to several scikit-learn classifiers (Linear SVM, Multinomial Naive Bayes, Logistic Regression) and to a small Keras MLP. For Twitter, they classified each tweet on its own and then gave each user the country predicted most often across their tweets. On the dev set, all classical models scored about 67.5% accuracy on subtask1 and slightly beat the MLP. Test accuracy was 66.31% for subtask1 and 67.20% for subtask2. The work shows that well-engineered sparse features remain competitive with neural models when labelled data is limited.

Key points

  • Data:
    • Subtask1 uses MADAR-Corpus-26: 1600 training and 200 dev sentences per dialect (41,600 / 5,200 / 5,200 for train/dev/test).
    • Subtask2 covers 2,180 / 300 / 500 users and roughly 218k / 30k / 50k tweets.
  • Features and preprocessing:
    • Word unigrams combined with character 2–5-grams worked better than either kind of n-gram alone.
    • Per-class LM scores (word and char 5-gram, summed) came from 6 and 26 separate LMs.
    • Tokenization was plain whitespace splitting.
  • What mattered most:
    • The LM scores were the strongest contributor in subtask1.
    • TF-IDF and the coarse Corpus-6 probabilities added little on top.
    • Logistic Regression and mNB did poorly on subtask2, so only SVM results are reported for it.
  • Test results:
    • Subtask1 (mNB): F1 66.21, accuracy 66.31.
    • Subtask2 (SVM): precision 83.37, recall 47.73, F1 57.90, accuracy 67.20.
  • Main error source in subtask2: 5,992 test tweets were unavailable. They were all labelled "Saudi Arabia", the most frequent training country, which the authors name as the main reason for the low recall.

Interesting ideas

  • I1. Train one small language model per class and use each model's score as a dense feature next to sparse n-grams; this gave the biggest gain here and is cheap and easy to inspect.
  • I2. Decide a label for a whole entity (a user) by majority vote over its many short items (tweets), instead of classifying all of the entity's text at once.
  • I3. Route inputs that lack a key feature (tweets with no Arabic, hence no dialect probabilities) to a separate classifier trained without that feature, rather than imputing values.
  • I4. Before deleting "noise" such as handles and URLs, measure the effect; here removing them reduced accuracy, so the authors kept the raw tweets.
  • I5. A default label based on the class prior quietly caused most of the recall loss, which shows how one fallback rule can dominate an evaluation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Per-variety character/word LMs (e.g. British vs. American, tweets vs. formal prose) could produce an explainable register/dialect score for the tokenizer and tagger before they pick normalization rules.
  • I2 (world): Attributes of an entity in a text's world (origin, stance, role) could be gathered as a vote over per-mention evidence, so the final value can be traced back to the mentions that support it.
  • I3 (learn): When a witness parser or a knowledge source is missing for a sentence, a fallback model trained on the reduced feature set avoids silent garbage values in silver-data agreement.
  • I4 (tokens): Cleaning rules for web text should be ablated, not assumed, because tokens that look like junk can carry signal for downstream tagging.
  • I5 (learn): Evaluations should report how many items reach default or fallback paths and score those items separately, as a kind of negative control.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓International Institute of Information Technology, HyderabadHyderabad · India42 works · 2,317 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Pruthwik Mishra and Vandan Mujadia. 2019. Arabic Dialect Identification for Travel and Twitter Text. Proceedings of the Fourth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-4628

@inproceedings{mishra2019arabic,
  title = {{Arabic Dialect Identification for Travel and Twitter Text}},
  author = {Pruthwik Mishra and Vandan Mujadia},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Workshop},
  year = {2019},
  doi = {10.18653/v1/w19-4628},
  url = {https://doi.org/10.18653/v1/w19-4628}
}

References

18 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, Hassan Alhuzali, and Mohamed Elaraby. 2018. You tweet what you speak: A city-level dataset of arabic dialects. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018).
  2. Manar Alkhatib, May El Barachi, and Khaled Shaalan. 2019. An arabic social media based framework for incidents and events monitoring in smart cities. Journal of Cleaner Production, 220:771–785. ↗
  3. Areej Alshutayri and Eric Atwell. 2017. Exploring twitter as a source of an arabic dialect corpus. International Journal of Computational Linguistics (IJCL), 8(2):37–44. find
  4. Fadi Biadsy, Julia Hirschberg, and Nizar Habash. 2009. Spoken arabic dialect identification using phonotactic modeling. In Proceedings of the eacl 2009 https://keras.io ↗
  5. Mohammad Salameh and Houda Bouamor. 2018. Fine-grained arabic dialect identification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1332–1344. workshop on computational approaches to semitic languages, pages 53–61. Association for Computational Linguistics.
  6. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018. The madar arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018). find
  7. Suwon Shon, Ahmed Ali, and James Glass. 2017. Mit-qcri arabic dialect identification system for the 2017 multi-genre broadcast challenge. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 374–380. IEEE.
  8. Omar F Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗
  9. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR Shared Task on Arabic Fine-Grained Dialect Identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop (WANLP19), Florence, Italy.
  10. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Ahmed Ali, Suwon Shon, James Glass, Yves Scherrer, Tanja Samardžić, Nikola Ljubešić, Jörg Tiedemann, et al. 2018. Language identification and morphosyntactic tagging: The second vardial evaluation campaign. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects. Association for Computational Linguistics.
  11. Andrei M Butnaru and Radu Tudor Ionescu. 2018. Unibuckernel reloaded: First place in arabic dialect identification for the second year in a row. arXiv preprint arXiv:1805.04876. ↗
  12. Mohamed Elaraby and Muhammad Abdul-Mageed. 2018. Deep models for arabic dialect identification on benchmarked data. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pages 263–274.
  13. Imane Guellil, Houda Saâdane, Faical Azouaou, Billel Gueni, and Damien Nouvel. 2019. Arabic natural language processing: An overview. Journal of King Saud University-Computer and Information Sciences. ↗
  14. Tommi Jauhiainen, Marco Lui, Marcos Zampieri, Timothy Baldwin, and Krister Lindén. 2018. Automatic language identification in texts: A survey. arXiv preprint arXiv:1804.08186. ↗
  15. Yuanzhi Li and Yang Yuan. 2017. Convergence analysis of two-layer neural networks with relu activation. In Advances in Neural Information Processing Systems, pages 597–607. find
  16. Wang Ling, Guang Xiang, Chris Dyer, Alan Black, and Isabel Trancoso. 2013. Microblogs as parallel corpora. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 176–186. find
  17. Maryam Najafian, Sameer Khurana, Suwon Shan, Ahmed Ali, and James Glass. 2018. Exploiting convolutional neural networks for phonotactic based dialect identification. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5174–5178. IEEE. find
  18. Fabian Pedregosa, Gael Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mishra-2019-arabic-dialect-identification-travel-twitter/