Elamur Research⌕
☰
home / papers / 2020 / The QMUL/HRBDT contribution to the NADI Arabic Dialect Identification Shared Task

The QMUL/HRBDT contribution to the NADI Arabic Dialect Identification Shared Task

Paper · 2020 · Abdulrahman Aloraini, Massimo Poesio and Ayman Alhelbawy

Summary

The paper presents a system for the country-level subtask of the NADI 2020 shared task. The goal is to tell which of 21 Arab countries a tweet comes from, using noisy Twitter text with very uneven class sizes. The authors clean and normalise the tweets and up-sample minority classes. Each tweet is then represented by three feature sets joined together: a BiLSTM-CNN encoder, character 1–5-gram TF-IDF, and LDA topic proportions. A small feed-forward classifier makes the prediction. The word vectors come from FastText trained on 10 million unlabeled NADI tweets. A second run adds a rule: if a tweet names an Arab country, it gets that country's label. This run reached 20.77% macro-F1 and 34.32% accuracy on the test set. The same architecture beat earlier published accuracies on the coarser AOC benchmark. The authors' error analysis matters beyond the scores: a large share of NADI tweets are entirely MSA or classical Arabic, so they carry no dialect signal and make the labels noisy.

Key points

  • NADI country-level data has 21,000 train, 4,957 dev and 5,000 unlabeled test tweets across 21 countries. Egypt dominates the training set, and several countries have only about 210 tweets each.
  • The architecture uses three feature blocks feeding a two-layer classifier with dropout:
    • a BiLSTM-CNN in the style of Ma and Hovy, with character embeddings of size 30;
    • char 1–5-gram TF-IDF (wider ranges did not help);
    • LDA with 50 topics, the best value from a sweep.
  • The country-name rule raised test macro-F1 from 0.194 to 0.207 and accuracy from 0.337 to 0.343. City names were tried and left out because they did not improve results.
  • On AOC, test accuracy reached 87.64 (binary), 89.94 (three-way) and 84.23 (four-way). These numbers beat the BiGRU and attention-BiLSTM baselines of Elaraby and Abdul-Mageed.
  • Using AraVec vocabulary as an MSA lexicon, 6,291 of 21,000 training tweets and 1,508 of 4,957 dev tweets are fully covered by MSA words. A greeting common to all dialects was learned as a Somali marker.

Interesting ideas

  • I1. A simple lexicon-coverage score (the share of a text's words found in a "standard" vocabulary) reveals training items that contain no class evidence at all, which helps explain label noise.
  • I2. A transparent gazetteer rule that overrides the statistical model was kept only for the parts that ablation proved useful (countries kept, cities dropped).
  • I3. Character n-grams sit next to learned word encoders to catch affix-level patterns, which suits noisy, unstandardised spelling.
  • I4. Embeddings trained on in-domain unlabeled text beat generic pretrained ones, even though the domain corpus is smaller.
  • I5. Error analysis showed that up-sampling plus universal phrases can turn a neutral expression into a fake signal for a minority class.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): score silver sentences by how much of them is generic boilerplate, and use the score to down-weight or flag them before training.
  • I2 (world): run place-name and entity lookups as explicit, logged rules over the classifier, and keep each lookup type only if its ablation shows a gain.
  • I3 (tokens): add char-n-gram features to the tagger and lemmatizer as a fallback for misspelled or unseen words in informal English.
  • I4 (tree): pretrain or adapt gparse representations on unlabeled text from the target genre (fables, school texts) before fine-tuning on treebanks.
  • I5 (learn): use phrases known to be class-neutral as negative controls, and check that no rare label or frame gets attached to them after rebalancing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Abdulrahman Aloraini, Massimo Poesio and Ayman Alhelbawy. 2020. The QMUL/HRBDT contribution to the NADI Arabic Dialect Identification Shared Task. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{aloraini2020qmul,
  title = {{The QMUL/HRBDT contribution to the NADI Arabic Dialect Identification Shared Task}},
  author = {Abdulrahman Aloraini, Massimo Poesio and Ayman Alhelbawy},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.31/}
}

References

22 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martín Abadi, Ashish Agarwal, Paul Barham, et al. 2015. TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org. find
  2. Muhammad Abdul-Mageed, Hassan Alhuzali, and Mohamed Elaraby. 2018. You tweet what you speak: A city-level dataset of arabic dialects. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
  3. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. NADI 2020: The First Nuanced Arabic Dialect Identification Shared Task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop (WANLP2020), Barcelona, Spain. ↗
  4. Maha Althobaiti, Udo Kruschwitz, and Massimo Poesio. 2014. Aranlp: A java-based library for the processing of arabic text.
  5. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018. The madar arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  6. Ryan Cotterell and Chris Callison-Burch. 2014. A multi-dialect, multi-genre corpus of informal written arabic. In LREC, pages 241–245.
  7. Kareem Darwish, Hassan Sajjad, and Hamdy Mubarak. 2014. Verifiably effective arabic dialect identification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1465–1468.
  8. Cicero Dos Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In International Conference on Machine Learning, pages 1818–1826. find
  9. Mohamed Elaraby and Muhammad Abdul-Mageed. 2018. Deep models for arabic dialect identification on benchmarked data. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pages 263–274.
  10. Heba Elfardy and Mona Diab. 2013. Sentence level dialect identification in arabic. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 456–461.
  11. Asmaa Etman and AA Louis Beex. 2015. Language and dialect identification: A survey. In 2015 SAI Intelligent Systems Conference (IntelliSys), pages 220–231. IEEE. find
  12. Nizar Habash, Abdelhadi Soudi, and Timothy Buckwalter. 2007. On arabic transliteration. In Arabic computational morphology, pages 15–22. Springer. ↗
  13. Fei Huang. 2015. Improved arabic dialect classification with social media data. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2118–2126.
  14. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. arXiv preprint arXiv:1603.01354. ↗
  15. Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45–50, Valletta, Malta, May. ELRA. http://is.muni.cz/publication/884893/en. ↗
  16. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained arabic dialect identification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1332–1344.
  17. Younes Samih, Hamdy Mubarak, Ahmed Abdelali, Mohammed Attia, Mohamed Eldesouki, and Kareem Darwish. 2019. Qc-go submission for madar shared task: Arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 290–294.
  18. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
  19. Toshiyuki Takezawa, Genichiro Kikui, Masahide Mizushima, and Eiichiro Sumita. 2007. Multilingual spoken language corpus development for communication research. In International Journal of Computational Linguistics & Chinese Language Processing, Volume 12, Number 3, September 2007: Special Issue on Invited Papers from ISCSLP 2006, pages 303–324. find
  20. Christoph Tillmann, Saab Mansour, and Yaser Al-Onaizan. 2014. Improved sentence-level arabic dialect classification. In Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects, pages 110–119.
  21. Wajdi Zaghouani and Anis Charfi. 2018. Arap-tweet: A large multi-dialect twitter corpus for gender, age and language variety identification. arXiv preprint arXiv:1808.07674. ↗
  22. Omar F Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aloraini-2020-qmul-hrbdt-contribution-nadi-arabic/