Elamur Research⌕
☰
home / papers / 2023 / Arabic Topic Classification in the Generative and AutoML Era

Arabic Topic Classification in the Generative and AutoML Era

Paper · 2023 · Doha Albared, Hadi Hamoud and Fadi Zaraket

Summary

Arabic topic classifiers have mostly been fine-tuned encoders working with a handful of news categories. The authors ask whether newer options, an instruction-tuned multilingual generative model or commercial AutoML services, work well for Arabic with a broader label set. They built ArBNTopic, a 14-topic dataset that pairs SANAD news articles with recent humanities and social-science books published by ACRPS, an Arab institute devoted to research and policy analysis. Because book passages often drift between subjects, they kept only the opening sentences after headings. They then adapted bloomz-7b-mt in two stages: first on unlabeled Arabic text, then on labeled classification data. The resulting model, ArGTC, was compared with classifiers produced by Google Vertex AI, H2O AutoML and HuggingFace AutoTrain. ArGTC reached 83% accuracy, slightly behind AutoTrain at 84%, and beat Vertex and H2O. It was also the cheapest option to train. The study shows that a single rented GPU can produce a competitive open classifier for a lower-resourced language, and it releases both the data and the models on HuggingFace.

Key points

  • ArBNTopic has 14 unified labels, including Religion, Finance-and-Economy, Politics, Medical, Law, Philosophy and History. News comes from SANAD (AlKhaleej, AlArabiya, Akhbarona), cut into segments of at most 250 characters. Book data comes from about 500 books of uneven category sizes, from 6 to 174 per category.

  • ArGTC was trained from bloomz-7b-mt in two phases:

    • continued training on 58,682 tokens of MSA books and news;
    • fine-tuning on 833,642 labeled tokens.

  • Accuracy / precision / recall:

    ModelAccuracyPrecisionRecall
    ArGTC838181
    HF AutoTrain848484
    Vertex AI767776
    Best H2O (AraGPT2 embeddings)767674
    H2O with AraBERT embeddings676767

  • Training costs differed sharply:

    • ArGTC: 4–5 hours on one A100-80GB at $1.86/h, about $9.3, with gradient checkpointing to fit memory.
    • AutoTrain: about $46 for 19,784 samples (free only up to 3,000).
    • Vertex AI: $22.
    • H2O: free.
    • ArGTC inference runs on a 32 GiB GPU for about $0.3/h.

  • The dataset (dru-ac/ArBNTopic), the intermediate model (FTArBloom) and ArGTC are public on HuggingFace.

Interesting ideas

  • I1. Long, multi-topic documents give noisy sentence labels, so the authors kept only the sentence(s) right after a title or heading, where authors usually state the thesis. For scarce categories they also took the opening sentence of each long paragraph.
  • I2. Category names from two sources (news and books) were reconciled through an explicit mapping table into one final label inventory, rather than training on mixed label sets.
  • I3. Adaptation was split into a cheap unsupervised step on in-domain text followed by supervised task tuning.
  • I4. Results were reported together with the dollar cost of training and inference for each approach, which made the "best" choice depend on a performance-per-cost trade-off.
  • I5. Within the same AutoML tool, only the embedding source was varied (AraGPT2 vs AraBERT), which isolated a 9-point accuracy effect of the representation alone.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When building silver data from books or textbooks, prefer heading-adjacent and paragraph-initial sentences as the cleanest examples of what a section is about. Fall back to paragraph openers only to fill thin classes.
  • I2 (tree): A small, inspectable mapping table between treebank or UD-dialect label inventories, documented in one place, keeps conversion explainable and auditable.
  • I3 (tree): gparse could first be adapted on raw in-domain English (school texts, fables) before supervised parser training, which should help domain fit without extra treebank annotation.
  • I4 (learn): Evaluation reports for LLM teacher tiers and witness parsers could log cost or compute next to accuracy, so cheaper tiers are chosen when the gain from expensive ones is marginal.
  • I5 (learn): Run controlled ablations that swap a single component, such as one witness parser or feature source, while holding the rest of the pipeline fixed, to attribute gains honestly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Arab Center for Research and Policy StudiesDoha · Qatar8 works · 69 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Doha Albared, Hadi Hamoud and Fadi Zaraket. 2023. Arabic Topic Classification in the Generative and AutoML Era. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.32

@inproceedings{albared2023arabic,
  title = {{Arabic Topic Classification in the Generative and AutoML Era}},
  author = {Doha Albared, Hadi Hamoud and Fadi Zaraket},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.32},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.32}
}

References

40 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. ↗
  2. Ahmed Abdelali, Hamdy Mubarak, Shammur Absar Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Yousseif Elshahawy, Ahmed Ali, Nadir Durrani, Natasa Milic-Frayling, and Firoj Alam. 2023. Benchmarking arabic ai with large language models. find
  3. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020. Arbert & marbert: deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. ↗
  4. Ashutosh Adhikari, Achyudh Ram, Raphael Tang, and Jimmy Lin. 2019. Docbert: Bert for document classification. arXiv preprint arXiv:1904.08398. ↗
  5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020b. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  6. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. AraGPT2: Pre-trained transformer for Arabic language generation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 196–207, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  7. Salvador Balkus and Donghui Yan. 2022a. Improving short text classification with augmented data using gpt-3. ↗
  8. Salvador V Balkus and Donghui Yan. 2022b. Improving short text classification with augmented data using gpt-3. Natural Language Engineering, pages 1–30. ↗
  9. Enkhbold Bataa and Joshua Wu. 2019. An investigation of transfer learning-based sentiment analysis in japanese. arXiv preprint arXiv:1905.09642. ↗
  10. Shammur Absar Chowdhury, Ahmed Abdelali, Kareem Darwish, Jung Soon-Gyo, Joni Salminen, and Bernard J Jansen. 2020. Improving arabic text categorization using transformer training diversification. In Proceedings of the fifth arabic natural language processing workshop, pages 226–236.
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  12. Omar Einea, Ashraf Elnagar, and Ridhwan Al Debsi. 2019. Sanad: Single-label arabic news articles dataset for automatic text categorization. Data in brief, 25:104076. ↗
  13. Jaouhar Fattahi and Mohamed Mejri. 2021. Spaml: a bimodal ensemble learning spam detector based on nlp techniques. In 2021 IEEE 5th international conference on cryptography, security and privacy (CSP), pages 107–112. IEEE. find
  14. Qurat Tul Ain, Mubashir Ali, Amna Riaz, Amna Noureen, Muhammad Kamran, Babar Hayat, and A Rehman. 2017. Sentiment analysis using deep learning techniques: a review. International Journal of Advanced Computer Science and Applications, 8(6). ↗
  15. Google-Vertex-AI. 2023. Google vertex ai. find
  16. Abdullah Hamid, Nasrullah Shiekh, Naina Said, Kashif Ahmad, Asma Gul, Laiq Hassan, and Ala Al-Fuqaha. 2020. Fake news detection in social media using graph neural networks and nlp techniques: A covid-19 use-case. arXiv preprint arXiv:2012.07517. ↗
  17. Ali Saleh Alammary. 2022. Bert models for arabic text classification: A systematic review. Applied Sciences, 12(11). ↗
  18. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020a. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  19. Safaa SI Ismail, Romany F Mansour, Abd El-Aziz, M Rasha, Ahmed I Taloba, et al. 2022. Efficient e-mail spam detection strategy using genetic decision tree processing with nlp features. ↗
  20. Computational Intelligence and Neuroscience, 2022. find
  21. Rie Johnson and Tong Zhang. 2017. Deep pyramid convolutional neural networks for text categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 562–570. find
  22. Abdullah Salem Khered, Ingy Yasser Hassan Abdou Abdelhalim, and Riza Theresa Batista-Navarro. 2022. Building an ensemble of transformer models for arabic dialect classification and sentiment analysis. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 479–484.
  23. Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Gptaraeval: A comprehensive evaluation of chatgpt on arabic nlp.
  24. Sheng How Kong, Li Mei Tan, Keng Hoon Gan, and Nur Hana Samsudin. 2020. Fake news detection using deep learning. In 2020 IEEE 10th symposium on computer applications & industrial electronics (ISCAIE), pages 102–107. IEEE. find
  25. Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Recurrent convolutional neural networks for text classification. In Proceedings of the AAAI conference on artificial intelligence, volume 29. find
  26. Erin LeDell and Sebastien Poirier. 2020. H2O AutoML: Scalable automatic machine learning. 7th ICML Workshop on Automated Machine Learning (AutoML). find
  27. Qian Li, Hao Peng, Jianxin Li, Congying Xia, Renyu Yang, Lichao Sun, Philip S. Yu, and Lifang He. 2022. A survey on text classification: From traditional to deep learning. ACM Trans. Intell. Syst. Technol., 13(2). ↗
  28. Phayung Meesad. 2021. Thai fake news detection based on information retrieval, natural language processing and machine learning. SN Computer Science, 2(6):425. ↗
  29. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2022a. Crosslingual generalization through multitask finetuning. ↗
  30. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022b. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786. ↗
  31. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2021. Arat5: Text-to-text transformers for arabic language generation. arXiv preprint arXiv:2109.12068. ↗
  32. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628– 647, Dublin, Ireland. Association for Computational Linguistics.
  33. Marco Polignano, Pierpaolo Basile, Marco De Gemmis, Giovanni Semeraro, Valerio Basile, et al. 2019. Alberto: Italian bert language understanding model for nlp challenging tasks based on tweets. In CEUR Workshop Proceedings, volume 2481, pages 1–6. CEUR.
  34. Jun Qian, Zhendong Niu, and Chongyang Shi. 2018. Sentiment analysis model on weather related tweets with deep neural network. In Proceedings of the 2018 10th international conference on machine learning and computing, pages 31–35. find
  35. Maria Razno. 2019. Machine learning text classification model with nlp approach. Computational Linguistics and Intelligent Systems, 2:71–73. find
  36. Sahar Sohangir, Dingding Wang, Anna Pomeranets, and Taghi M Khoshgoftaar. 2018. Big data: Deep learning for financial sentiment analysis. Journal of Big Data, 5(1):1–25. ↗
  37. Shrawan Kumar Trivedi. 2016. A study of machine learning classifiers for spam detection. In 2016 4th international symposium on computational and business intelligence (ISCBI), pages 176–180. IEEE. find
  38. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Huggingface’s transformers: State-of-the-art natural language processing. ↗
  39. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual bert?
  40. Fatima zahra El-Alami, Said Ouatik El Alaoui, and Noureddine En Nahnahi. 2022. Contextual semantic embeddings based on fine-tuned arabert model for arabic text multi-class categorization. Journal of King Saud University - Computer and Information Sciences, 34(10, Part A):8422–8428. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/albared-2023-arabic-topic-classification-generative-automl/