Elamur Research⌕
☰
home / papers / 2020 / Improving Arabic Text Categorization Using Transformer Training Diversification

Improving Arabic Text Categorization Using Transformer Training Diversification

Paper · 2020 · Shammur Absar Chowdhury, Ahmed Abdelali, Kareem Darwish et al.

Summary

The paper asks how well Arabic short-text topic classifiers carry over when the training text and the test text differ in source, platform, or variety of Arabic (MSA vs. dialects). The authors built two new labelled resources. The first, ASND, holds Aljazeera posts from Twitter, Facebook and YouTube, mostly in MSA. The second, AITD, contains tweets from influential accounts and includes a lot of dialectal Arabic. They also used the existing SANAD news collection. They trained SVM, mBERT, AraBERT and QARiB classifiers and evaluated them across every train/test pairing. QARiB is a BERT model pre-trained on a mix of formal news and informal tweets, while AraBERT was pre-trained on formal text only. The main finding is that diversity in the data helps generalization, and it can enter at two points: in the task-specific training set, and in the corpus used for pre-training. QARiB matched AraBERT on news-style data and clearly beat it on social media posts. The takeaway for low-resource settings is practical: mixing registers during pre-training can stand in for expensive in-domain annotation.

Key points

  • ASND: 10k posts collected from February 2017 to September 2019 (about 6k tweets, 2k Facebook posts, 2k YouTube titles).
    • Labels come from Amazon Mechanical Turk across 12 categories.
    • Roughly 1.7k posts were dropped because annotators disagreed.
    • Fleiss's kappa is 0.69.
    • Final splits: 6,676 train, 567 dev, 1,103 test.
  • AITD: a large tweet test set of about 115.7k items, evaluated on 10 labels.
  • SANAD: used as a news-headline resource with 6 overlapping categories.
  • Cross-domain gap: trained on ASND and tested on SANAD, QARiB scored 81 macro-F1 against AraBERT's 60.
  • AITD results: trained on ASND+SANAD, QARiB reached 60 against 54 for AraBERT and 51 for mBERT.
  • Overall gains (from the abstract): diversifying task training data gave about +21% macro-F1, and diversifying pre-training data gave about +26%.
  • Ranking quality: with training and testing on ASND, QARiB placed the correct category in its top two 90% of the time (P@2), compared with 69% for AraBERT.
  • Confusions: politics often got mixed up with crime-war-conflict and human-rights-press-freedom. On AITD, roughly 40% of human-rights tweets were labelled politics.

Interesting ideas

  • I1. Pre-training on a deliberate blend of formal and informal registers produced an encoder that transfers across genres, without losing accuracy on formal text.
  • I2. Crowd annotation was protected by three layers: a language-skill qualification quiz, hidden gold items inside each work batch, and a requirement that two of three annotators agree, with unresolved items discarded rather than forced.
  • I3. As a sanity check on labels, the authors computed a per-class token valence score, which finds words that occur almost exclusively in one class, and confirmed that those words fit the category.
  • I4. Evaluation was a full cross-matrix of training source against test source, which made domain-shift failures visible that single-split testing would hide.
  • I5. The error analysis showed that some classes, such as politics and human rights, are semantically close, and the authors suggest a hierarchical label ontology instead of a flat one.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): The training mix for gparse could be built to cover edited prose, dialogue and informal web English together, so the parser holds up on fable dialogue and on noisy inputs.
  • I2 (learn): Silver data from witness parsers or LLM teacher tiers could be filtered the same way, by hiding known-gold sentences in each batch and dropping items where the witnesses fail to agree.
  • I3 (learn): Applying valence to features of each frame or repair-rule class could expose labelling artefacts before training, acting as a cheap negative control.
  • I4 (answer): School tests, maths problems and fable QA could be scored with systems tuned on one domain and tested on the others, to measure how much each component actually generalizes.
  • I5 (world): Frame and ontology categories could be arranged as a hierarchy, so that a near-miss counts as a parent-level match and the explanation can say which branch was uncertain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Shammur Absar Chowdhury, Ahmed Abdelali, Kareem Darwish, Jung Soon-Gyo, Joni Salminen and Bernard J. Jansen. 2020. Improving Arabic Text Categorization Using Transformer Training Diversification. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{chowdhury2020improving,
  title = {{Improving Arabic Text Categorization Using Transformer Training Diversification}},
  author = {Shammur Absar Chowdhury, Ahmed Abdelali, Kareem Darwish, Jung Soon-Gyo, Joni Salminen and Bernard J. Jansen},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.21/}
}

References

39 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mourad Abbas and Kamel Smaili. 2005. Comparison of topic identification methods for arabic language. In Proceedings of International Conference on Recent Advances in Natural Language Processing, RANLP, pages 14–17. find
  2. Mourad Abbas, Kamel Smaı̈li, and Daoud Berkani. 2011. Evaluation of topic identification methods on arabic corpora. J. Digit. Inf. Manag., 9(5):185–192. find
  3. Muhammad Abdul-Mageed, Hassan Alhuzali, and Mohamed Elaraby. 2018. You tweet what you speak: A city-level dataset of arabic dialects. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
  4. Malak Abdullah and Samira Shaikh. 2018. Teamuncc at semeval-2018 task 1: Emotion detection in english and arabic tweets using deep learning. In Proceedings of the 12th international workshop on semantic evaluation, pages 350–357.
  5. Deem AlSaleh, Mashael Bin AlAmir, and Souad Larabi-Marie-Sainte. 2020. Snad arabic dataset for deep learning. In Proceedings of SAI Intelligent Systems Conference, pages 630–640. Springer. find
  6. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  7. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. ↗
  8. Amina Chouigui, Oussama Ben Khiroun, and Bilel Elayeb. 2017. Ant corpus: an arabic news text collection for textual classification. In 2017 IEEE/ACS 14th International Conference on Computer Systems and Applications (AICCSA), pages 135–142. IEEE. find
  9. Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Abdelali, Soon-gyo Jung, Bernard J Jansen, and Joni Salminen. 2020. A multi-platform arabic news comment dataset for offensive language detection. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 6203–6212.
  10. Michael D Conover, Jacob Ratkiewicz, Matthew Francisco, Bruno Gonçalves, Filippo Menczer, and Alessandro Flammini. 2011. Political polarization on twitter. In Fifth international AAAI conference on weblogs and social media. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  12. Ibrahim Abu El-Khair. 2016. 1.5 billion words arabic corpus. arXiv preprint arXiv:1611.04033. ↗
  13. Obeida ElJundi, Wissam Antoun, Nour El Droubi, Hazem Hajj, Wassim El-Hajj, and Khaled Shaban. 2019. hulmona: The universal language model in arabic. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 68–77.
  14. Ashraf Elnagar, Ridhwan Al-Debsi, and Omar Einea. 2020. Arabic text classification using deep learning models. Information Processing & Management, 57(1):102121. ↗
  15. Alexander Erdmann, Nasser Zalmout, and Nizar Habash. 2018. Addressing noise in multidialectal word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 558–565. find
  16. Rosa Falotico and Piero Quatto. 2015. Fleiss’ kappa statistic without paradoxes. Quality & Quantity, 49(2):463– 470. ↗
  17. Ibrahim Abu Farha and Walid Magdy. 2019. Mazajak: An online arabic sentiment analyser. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 192–198.
  18. Sabit Hassan, Younes Samih, Hamdy Mubarak, Ahmed Abdelali, Ammar Rashed, and Shammur Absar Chowdhury. 2020. Alt submission for osact shared task on offensive language detection. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 61–65.
  19. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146. ↗
  20. Yaser Keneshloo, Shuguang Wang, Eui-Hong Han, and Naren Ramakrishnan. 2016. Predicting the popularity of news articles. In Proceedings of the 2016 SIAM International Conference on Data Mining, pages 441–449. SIAM. find
  21. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. ↗
  22. David D Lewis. 2001. Applying support vector machines to the trec-2001 batch filtering and routing tasks. In TREC. find
  23. Pierre Lison and Jörg Tiedemann. 2016. Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France, may. European Language Resources Association (ELRA).
  24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  25. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. ↗
  26. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2017. Advances in pre-training distributed word representations. arXiv preprint arXiv:1712.09405. ↗
  27. Saif M Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. Nrc-canada: Building the state-of-the-art in sentiment analysis of tweets. arXiv preprint arXiv:1308.6242. ↗
  28. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. SemEval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, pages 31–41, San Diego, California.
  29. Hamdy Mubarak and Kareem Darwish. 2019. Arabic offensive language classification on twitter. In International Conference on Social Informatics, pages 269–276. Springer. find
  30. Hamdy Mubarak, Ammar Rashed, Kareem Darwish, Younes Samih, and Ahmed Abdelali. 2020. Arabic offensive language on twitter: Analysis and experiments. arXiv preprint arXiv:2004.02192. ↗
  31. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. ↗
  32. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. ↗
  33. John Platt. 1998. Sequential minimal optimization: A fast algorithm for training support vector machines. find
  34. Marco Polignano, Pierpaolo Basile, Marco de Gemmis, Giovanni Semeraro, and Valerio Basile. 2019. Alberto: Italian bert language understanding model for nlp challenging tasks based on tweets. In CLiC-it.
  35. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
  36. Bashar Talafha, Mohammad Ali, Muhy Eddin Za’ter, Haitham Seelawi, Ibraheem Tuffaha, Mostafa Samir, Wael Farhan, and Hussein T Al-Natsheh. 2020. Multi-dialect arabic bert for country-level dialect identification. arXiv preprint arXiv:2007.05612. ↗
  37. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th annual meeting of the association for computational linguistics, pages 384–394.
  38. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5753–5763. ↗
  39. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. Osian: Open source international arabic news corpus-preparation and integration into the clarin-infrastructure. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175–182. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chowdhury-2020-improving-arabic-text-categorization-transformer/