Elamur Research⌕
☰
home / papers / 2019 / Mawdoo3 AI at MADAR Shared Task: Arabic Tweet Dialect Identification

Mawdoo3 AI at MADAR Shared Task: Arabic Tweet Dialect Identification

Paper · 2019 · Bashar Talafha, Wael Farhan, Ahmed Altakrouri et al.

Summary

The paper addresses user-level Arabic dialect identification for MADAR shared subtask-2. Given up to 100 tweets per Twitter user, the system must assign one of 21 country dialects. The data is difficult in several ways. Classes are heavily skewed toward Saudi Arabia. Most tweets are retweets, so the text often comes from an author with a different dialect than the user. About a tenth of the tweets could no longer be retrieved.

The authors built features from three sources:

  • word and character n-gram Tf-idf,
  • character Tf-idf over usernames and other Twitter-specific fields,
  • city/MSA language-model probabilities supplied by the organizers.

They also tried AraVec and fastText embeddings. Several deep architectures (SepCNN, LSTMs, char-CNNs and hybrids) were compared against a LinearSVC. The linear model beat every neural model. The largest gain came from aggregating tweet-level predictions per user, using either frequency-aware weighted voting or majority voting with a stricter threshold for the dominant class. The best system reached roughly 71–72% Macro-F1 on dev and 69.86% on test, placing second. The result shows that curated features plus careful aggregation can beat deeper models on noisy social-media data.

Key points

  • Dataset: The MADAR Twitter data covers 21 countries.
    • Users never appear in more than one split.
    • About 61% of tweets are retweets and about 10% are unavailable.
    • Saudi Arabia accounts for about 35% of tweets; Djibouti for 0.08%.
  • Features:
    • word unigram+bigram Tf-idf and character 1–5-gram Tf-idf;
    • character-unigram Tf-idf over usernames and mentions;
    • probabilities for 25 cities + MSA from the Salameh et al. (2018) model;
    • 300-d Twitter skip-gram AraVec;
    • a 300-d fastText model trained on AutoTweet, ArapTweet, DART, PADIC and MADAR data.
  • Tweet-level results (dev): LinearSVC reached 54.26% accuracy / 38.95 F1. The best neural model, a word LSTM with fastText, reached 50.59 / 34.65.
  • User-level results:
    • Concatenating each user's tweets into one document gave 77.33% accuracy / 70.43 F1 on dev.
    • Weighted user voting gave 81.67% / 71.60 F1 on dev and 69.86 F1 on test.
    • A Saudi-threshold vote (Saudi chosen only if at least 75% of a user's tweets are predicted Saudi) is reported at 71.84 F1 in the text but 70.72 in the table.
  • Dialect weights: Each dialect gets an integer weight from 1 to 6. Counts are binned on a cube-root scale between the largest and smallest class, and rarer dialects receive larger weights. The retweet weight is given as 6 in the text and 8 in the table.

Interesting ideas

  • I1. Instead of classifying a whole collection at once, the authors classify each item and then vote, with rarer classes weighted more heavily via cube-root bins; this was the biggest single improvement and suits any setting where many weak per-item guesses must become one label.
  • I2. The dominant class is only accepted when it wins a supermajority (75%), which curbs a classifier's drift toward the majority label and helps macro-averaged metrics.
  • I3. Retweets carry the retweeter's label but another author's language, a reminder that the person who reproduces content is not necessarily its source.
  • I4. A linear SVM over hand-chosen word and character n-grams beat every neural architecture tried, showing that a cheap, transparent baseline can be very hard to beat on noisy, short, skewed text.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When several witness parsers vote on a silver label, votes for rare relations or tags could be up-weighted with similar binned weights, so silver data is not swamped by the most frequent labels.
  • I2 (tokens): The tagger or lemmatizer could require extra confidence before falling back to its most frequent default analysis, sending close calls to a richer disambiguation step instead.
  • I3 (frames): Quoted or reported speech should be attributed to its original speaker rather than the narrator, so speech acts and stances are not assigned to the wrong agent.
  • I4 (learn): Every new learned component should be compared against a simple linear n-gram baseline as a control, keeping the complex model only if it clearly wins.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Bashar Talafha, Wael Farhan, Ahmed Altakrouri and Hussein Al-Natsheh. 2019. Mawdoo3 AI at MADAR Shared Task: Arabic Tweet Dialect Identification. Proceedings of the Fourth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-4629

@inproceedings{talafha2019mawdoo3,
  title = {{Mawdoo3 AI at MADAR Shared Task: Arabic Tweet Dialect Identification}},
  author = {Bashar Talafha, Wael Farhan, Ahmed Altakrouri and Hussein Al-Natsheh},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Workshop},
  year = {2019},
  doi = {10.18653/v1/w19-4629},
  url = {https://doi.org/10.18653/v1/w19-4629}
}

References

19 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. ↗
  2. Hind Almerekhi and Tamer Elsayed. 2015. Detecting automatically-generated arabic tweets. In AIRS, pages 123–134. Springer. find
  3. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. Journal of machine learning research, 12(Oct):2825–2830. ↗
  4. Israa Alsarsour, Esraa Mohamed, Reem Suwaileh, and Tamer Elsayed. 2018. Dart: A large dataset of dialectal arabic tweets. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018).
  5. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. ↗
  6. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained Arabic dialect identification. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 1332–1344, Santa Fe, New Mexico, USA.
  7. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic Dialect Corpus and Lexicon. In Proceedings of the Language Resources and Evaluation Conference (LREC), Miyazaki, Japan. find
  8. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. Twitter. Api reference index twitter developers. find
  9. Wajdi Zaghouani and Anis Charfi. 2018. Arap-tweet: A large multi-dialect twitter corpus for gender, age and language variety identification. arXiv preprint arXiv:1808.07674. ↗
  10. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR Shared Task on Arabic Fine-Grained Dialect Identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop (WANLP19), Florence, Italy.
  11. Omar F Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗
  12. Timo I. Denk. 2017. Text classification with separable convolutional neural networks. find
  13. Shiwei Zhang, Xiuzhen Zhang, and Jeffrey Chan. 2017. A word-character convolutional neural network for language-agnostic twitter sentiment analysis. In Proceedings of the 22nd Australasian Document Computing Symposium, page 12. ACM. find
  14. Mohamed Elaraby and Muhammad Abdul-Mageed. 2018. Deep models for arabic dialect identification on benchmarked data. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pages 263–274.
  15. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pages 649–657. ↗
  16. Salima Harrat, Karima Meftouh, Mourad Abbas, and Kamel Smaili. 2014. Building resources for algerian arabic dialects. In 15th Annual Conference of the International Communication Association Interspeech. find
  17. Fei Huang. 2015. Improved arabic dialect classification with social media data. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2118–2126.
  18. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431. Association for Computational Linguistics. find
  19. Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. arXiv preprint arXiv:1605.05101. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/talafha-2019-mawdoo3-ai-madar-shared-task/