Elamur Research⌕
☰
home / papers / 2019 / Team JUST at the MADAR Shared Task on Arabic Fine-Grained Dialect Identification

Team JUST at the MADAR Shared Task on Arabic Fine-Grained Dialect Identification

Paper · 2019 · Bashar Talafha, Ali Fadel, Mahmoud Al-Ayyoub et al.

Summary

The paper addresses fine-grained Arabic dialect identification in the MADAR shared task (travel-domain subtask). The goal is to assign a sentence to one of 25 city dialects or MSA. The training set is small and many dialects are close to one another. The authors first study the parallel data and find that very few words are shared across all dialects, while some pairs, mostly within the Levant, repeat many identical sentences. Their system strips punctuation and emoticons. It then enlarges the training data by merging part of Corpus-6 with Corpus-26 and, for the remaining dialects, adding word-shuffled copies of each sentence, which yields a balanced set. Features are per-dialect KenLM scores plus word- and character-level n-gram TF-IDF, fed to a Multinomial Naive Bayes classifier. Several neural models, BERT among them, did worse on dev. On the test set the system is close to the top team (Macro-F1 66.33%). However, a broken submission file left it at the bottom of the official ranking. The work shows that cheap statistical models can still compete when data is scarce.

Key points

  • Data:
    • Corpus-26 holds 41,600 train, 5,200 dev and 5,200 test parallel sentences (1,600 training sentences per dialect).
    • Corpus-6 adds 54,000 training sentences for BEI, CAI, DOH, MSA, RAB and TUN.
    • Only 84 of 27,501 unique words occur in every dialect.
  • Augmentation:
    • For the six Corpus-6 dialects, 8,000 Corpus-6 sentences are combined with the 1,600 from Corpus-26.
    • Each of the other 20 dialects gets five shuffled variants per sentence, using a different random seed for each variant.
    • The result is 9,600 sentences per class and 249,600 in total.
  • Features and classifier:
    • A 26-dimensional vector of KenLM probabilities, one per dialect.
    • Word TF-IDF n-grams from 1 to 6 and character n-grams from 1 to 5.
    • MNB with α = 0.5, trained one-vs-rest.
  • Test results:
    • Macro-F1 66.33%, Precision 66.56%, Recall 66.42%.
    • City accuracy 66.42%, country accuracy 74.71%, region accuracy 84.54%.
    • This places the system 4th–5th among teams, against a top F1 of 67.32%.
  • Error analysis:
    • The dev ablation shows only a small gain from augmentation (F1 67.51 with it vs 66.83 without).
    • The worst confusions are CAI→ASW (39 samples) and RAB→FES (38).
    • ALG and MOS are the easiest dialects to identify.

Interesting ideas

  • I1. Word-order shuffling is a valid augmentation only when the label truly ignores order; the authors justify it on exactly that basis, which makes the assumption explicit and checkable.
  • I2. Training one small language model per class and using the vector of their scores as features gives a compact, readable signal of which variety a sentence resembles.
  • I3. Measuring how many identical items two classes share ahead of modelling predicted which classes would later be confused (Levant pairs, CAI vs ASW).
  • I4. Rebalancing by capping the large corpus and inflating the small classes to an equal size kept the extra six-dialect data from dominating training.
  • I5. An ablation that reports a small augmentation gain honestly, and links it to the classifier's independence assumptions, shows that the value of augmentation depends on the model.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Reverse the trick into a negative control: feed shuffled sentences to the parser and frame extractor, and treat any component whose output barely changes as secretly behaving like a bag of words.
  • I2 (tokens): Per-register or per-dialect n-gram models could flag the text variety (e.g. informal or regional English) before choosing tokenizer and tagger rules, with the score vector logged as part of the explanation.
  • I3 (tree): Before UD dialect conversion or treebank merging, count overlapping sentences and constructions between sources to anticipate where the parsers will disagree, and focus repair rules there.
  • I4 (learn): When silver data from different witness parsers or genres differs greatly in volume, cap and balance the sources so that one genre does not bias the perceptron or gparse training.
  • I5 (learn): Run every augmentation through a with/without ablation per model type, because a trick that helps gparse may do nothing for the perceptron parser.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Jordan University of Science and TechnologyIrbid · Jordan15 works · 1,840 citations

Also on the byline: Duquesne University.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Bashar Talafha, Ali Fadel, Mahmoud Al-Ayyoub, Yaser Jararweh, Mohammad AL-Smadi and Patrick Juola. 2019. Team JUST at the MADAR Shared Task on Arabic Fine-Grained Dialect Identification. Proceedings of the Fourth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-4638

@inproceedings{talafha2019team,
  title = {{Team JUST at the MADAR Shared Task on Arabic Fine-Grained Dialect Identification}},
  author = {Bashar Talafha, Ali Fadel, Mahmoud Al-Ayyoub, Yaser Jararweh, Mohammad AL-Smadi and Patrick Juola},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Workshop},
  year = {2019},
  doi = {10.18653/v1/w19-4638},
  url = {https://doi.org/10.18653/v1/w19-4638}
}

References

20 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mahmoud Al-Ayyoub, Aya Nuseir, Kholoud Alsmearat, Yaser Jararweh, and Brij Gupta. 2018. Deep learning for arabic nlp: A survey. Journal of computational science, 26:522–531. ↗
  2. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018. The madar arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018). find
  3. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR Shared Task on Arabic Fine-Grained Dialect Identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop (WANLP19), Florence, Italy.
  4. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained Arabic dialect identification. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 1332–1344, Santa Fe, New Mexico, USA.
  5. Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Return of the devil in the details: Delving deep into convolutional nets. arXiv preprint arXiv:1405.3531. ↗
  6. Francois Chollet. 2016. Building powerful image classification models using very little data. The Keras Blog. Https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html. find
  7. Bashar Talafha and Mahmoud Al-Ayyoub. 2019. Ioh-rcnn: Pursue the ingredients of happiness using recurrent convolutional neural networks. In Proceedings of the 2nd Workshop on Affective Content Analysis (AffCon 2019) co-located with Thirty-Third AAAI Conference on Artificial Intelligence (AAAI 2019), Honolulu, USA, January 27, 2019., pages 191–197. find
  8. François Chollet. 2017. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251–1258. ↗
  9. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  11. Omar F. Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗
  12. Maad Ebrahim, Mohammad Alsmirat, and Mahmoud Al-Ayyoub. 2018. Performance study of augmentation techniques for hep2 cnn classification. In 2018 9th International Conference on Information and Communication Systems (ICICS), pages 163– 168. IEEE. find
  13. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Ahmed Ali, Suwon Shon, James Glass, Yves Scherrer, Tanja Samardžić, Nikola Ljubešić, Jörg Tiedemann, et al. 2018. Language identification and morphosyntactic tagging: The second vardial evaluation campaign. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects. Association for Computational Linguistics.
  14. Heba Elfardy and Mona Diab. 2013. Sentence level dialect identification in arabic. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 456–461.
  15. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. arXiv preprint ar-Xiv:1705.00440.
  16. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778. ↗
  17. Kushal Kafle, Mohammed Yousefhussien, and Christopher Kanan. 2017. Data augmentation for visual question answering. In Proceedings of the 10th International Conference on Natural Language Generation, pages 198–202. find
  18. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint ar-Xiv:1408.5882.
  19. Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In International conference on machine learning, pages 1188– 1196. ↗
  20. Fatiha Sadat, Farnazeh Kazemi, and Atefeh Farzindar. 2014. Automatic identification of arabic dialects in social media. In Proceedings of the first international workshop on Social media retrieval and analysis, pages 35–40. ACM. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/talafha-2019-team-just-madar-shared-task/