Elamur Research⌕
☰
home / papers / 2020 / AraDIC: Arabic Document Classification Using Image-Based Character Embeddings and Class-Balanced Loss

AraDIC: Arabic Document Classification Using Image-Based Character Embeddings and Class-Balanced Loss

Paper · 2020 · Mahmoud Daif, Shunsuke Kitada and Hitoshi Iyatomi

Summary

Arabic text classification usually depends on a heavy preprocessing pipeline: morphological analysis, word segmentation, stemming and hand-built features such as TF-IDF. Segmentation is especially difficult since spaces do not reliably mark where one Arabic word ends and the next begins. The authors propose AraDIC, which avoids this pipeline by treating each character as a small image. A depth-wise CNN character encoder turns each glyph image into a vector. A classifier, either a CLCNN or a BiGRU, sits on top, and the whole system is trained end to end. To handle skewed label distributions, they use the class-balanced (CB) loss of Cui et al., which was originally developed for vision tasks. They also release two new datasets that need no manual annotation: AWT, built from Wikipedia titles, and AraP, built from poetry. Together these cover modern standard, colloquial and classical Arabic. AraDIC beats both classical and deep baselines, improving micro F-score by 12.29% and macro F-score by 23.05%. This suggests that glyph-level representations can replace fragile segmentation for a morphologically rich language.

Key points

  • Datasets:
    • AWT contains 444,911 Arabic Wikipedia titles in 11 categories, mostly modern standard Arabic. Labels come from the category hierarchy, so some are noisy.
    • AraP contains 41,264 poems from five eras, spanning the 6th to 21st centuries, crawled from the Adab website.
    • Both datasets are long-tailed and are split 80/20 into train and test.
  • Character encoder: it maps a 36×36 character image to a 128-dimensional embedding.
  • Classifiers: a CLCNN uses inputs capped at 60 characters for AWT and 128 for AraP. A BiGRU (3 layers, hidden size 128, mean-pooled hidden states) reads the full text.
  • Training setup: Adam with learning rate 0.001 and batch size 64, CB loss with β = 0.99, and wildcard training at 10%.
  • Baselines: a majority-class classifier, an SVM using TF-IDF unigrams with the Khoja stemmer and Farasa segmentation, a sentence CNN with AraVec embeddings, and a one-hot CLCNN.
  • Results:
    • The BiGRU clearly outperforms the CLCNN.
    • CB loss raises macro F-score for both classifiers.
    • t-SNE plots show that letters with similar shapes mostly cluster together.

Interesting ideas

  • I1. Weighting each class by the inverse of its "effective number" of samples avoids the over-correction of plain inverse-frequency weights, which can hurt the frequent classes.
  • I2. Wildcard training randomly drops whole positions in the embedding sequence, so each epoch sees a slightly different input; this is cheap augmentation that builds robustness to unseen units.
  • I3. A large labelled benchmark can be built without annotators by harvesting an existing hierarchy, such as Wikipedia categories, and accepting some label noise.
  • I4. The authors report a majority-class baseline together with both micro and macro F-scores, which exposes models that only learn the dominant class.
  • I5. Training the character encoder end to end with the task lets the representation adapt to the task, and a t-SNE plot serves as a quick sanity check that similar units end up close together.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Apply CB weighting when training gparse or the tagger so that rare UD relations and rare tags get more weight without degrading common ones like nsubj or det.
  • I2 (tree): Randomly blank token embeddings during parser training so the parser learns to attach unknown or misspelled words from their context.
  • I3 (world): Derive silver entity-type or topic labels from the Wikidata/WordNet hierarchy to test and train the ontology layer, while tracking items that fall under several categories.
  • I4 (learn): Add majority-class and frequency baselines as standard negative controls in evaluation, and always report macro scores next to micro ones.
  • I5 (tokens): Train a character-level encoder for out-of-vocabulary words jointly with the tagger, then inspect its neighbourhoods to confirm that it groups similar spellings and affixes sensibly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hosei UniversityTokyo · Japan3 works · 4 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Stemming · 17 works

Cite

Cite as Mahmoud Daif, Shunsuke Kitada and Hitoshi Iyatomi. 2020. AraDIC: Arabic Document Classification Using Image-Based Character Embeddings and Class-Balanced Loss. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-srw.29

@inproceedings{daif2020aradic,
  title = {{AraDIC: Arabic Document Classification Using Image-Based Character Embeddings and Class-Balanced Loss}},
  author = {Mahmoud Daif, Shunsuke Kitada and Hitoshi Iyatomi},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop},
  year = {2020},
  doi = {10.18653/v1/2020.acl-srw.29},
  url = {https://doi.org/10.18653/v1/2020.acl-srw.29}
}

References

35 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for arabic. In Proc. of NAACL, pages 11– 16.
  2. Mohammed N Al-Kabi, Saif A Kazakzeh, Belal M Abu Ata, Saif A Al-Rababah, and Izzat M Alsmadi. 2015. A novel root based arabic stemmer. Journal of King Saud University-Computer and Information Sciences, 27(2):94–103. ↗
  3. Nora Al-Twairesh, Hend Al-Khalifa, and AbdulMalik Al-Salman. 2016. Arasenti: Large-scale twitter-specific arabic sentiment lexicons. In Proc. of ACL, pages 697–705.
  4. Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. 2018. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249–259. ↗
  5. Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321–357. ↗
  6. Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. In Proc. of NIPS Workshop on Deep Learning. ↗
  7. Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. 2019. Class-balanced loss based on effective number of samples. In Proc. of CVPR, pages 9268–9277. find
  8. Alaa M El-Halees. 2009. Filtering spam e-mail from mixed arabic and english messages: A comparison of machine learning techniques. Filtering Spam E-Mail from Mixed Arabic and English Messages: A Comparison of Machine Learning Techniques., 6(1). find
  9. Yonatan Geifman and Ran El-Yaniv. 2017. Deep active learning over the long tail. CoRR preprint arXiv:1711.00941. ↗
  10. Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. 2016. Learning deep representation for imbalanced classification. In Proc. of CVPR, pages 5375– 5384. find
  11. Chen Huang, Yining Li, Change Loy Chen, and Xiaoou Tang. 2019. Deep imbalanced learning for face recognition and attribute prediction. IEEE transactions on pattern analysis and machine intelligence. ↗
  12. Salman H Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A Sohel, and Roberto Togneri. 2017. Cost-sensitive learning of deep feature representations from imbalanced data. IEEE transactions on neural networks and learning systems, 29(8):3573–3587. ↗
  13. Shereen Khoja. 2001. Apt: Arabic part-of-speech tagger. In Proc. of the Student Workshop at NAACL, pages 20–25. find
  14. Yoon Kim. 2014. Convolutional neural networks for sentence classification. CoRR preprint arXiv:1408.5882. ↗
  15. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR preprint arXiv:1412.6980. ↗
  16. Shunsuke Kitada, Ryunosuke Kotani, and Hitoshi Iyatomi. 2018. End-to-end text classification via image-based embedding using character-level networks. In Proc. of IEEE AIPR Workshop, pages 1–4. IEEE. find
  17. Frederick Liu, Han Lu, Chieh Lo, and Graham Neubig. 2017. Learning character-level compositionality with visual features. In Proc. of ACL, pages 2059–2068. find
  18. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605. find
  19. Jonathan Masci, Ueli Meier, Dan Cireşan, and Jürgen Schmidhuber. 2011. Stacked convolutional autoencoders for hierarchical feature extraction. In International conference on artificial neural networks, pages 52–59. Springer. find
  20. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proc. of NIPS, pages 3111–3119. ↗
  21. Fuchun Peng, Xiangji Huang, Dale Schuurmans, and Shaojun Wang. 2003. Text classification in asian languages without word segmentation. In Proc. IRAL workshop, pages 41–48. find
  22. Dania Sagheer and Fadel Sukkar. 2018. Arabic sentences classification via deep learning. International Journal of Computer Applications, 182(5):40–46. ↗
  23. Hossam S Ibrahim, Sherif M Abdou, and Mervat Gheith. 2015. Sentiment analysis for modern standard arabic and colloquial. CoRR preprint arXiv:1505.03105. ↗
  24. Said A Salloum, Ahmad Qasim AlHamad, Mostafa Al-Emran, and Khaled Shaalan. 2018. A survey of arabic text mining. In Intelligent Natural Language Processing: Trends and Applications, pages 417– 431. Springer. ↗
  25. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167. ↗
  26. Mohammed A Shehab, Omar Badarneh, Mahmoud Al-Ayyoub, and Yaser Jararweh. 2016. A supervised approach for multi-label classification of arabic news articles. In Proc. of CSIT, pages 1–6. IEEE. find
  27. Li Shen, Zhouchen Lin, and Qingming Huang. 2016. Relay backpropagation for effective learning of deep convolutional neural networks. In Proc. of ECCV, pages 467–482. Springer. find
  28. Daiki Shimada, Ryunosuke Kotani, and Hitoshi Iyatomi. 2016. Document classification through image-based character embedding and wildcard training. In Proc. of IEEE Big Data, pages 3922– 3927. IEEE. find
  29. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
  30. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958. find
  31. Kai Ming Ting. 2000. A comparative study of cost-sensitive boosting algorithms. In Proc. of ICML. Citeseer. find
  32. Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. 2017. Learning to model the tail. In Advances in Neural Information Processing Systems, pages 7029–7039. find
  33. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proc. of NIPS, pages 649–657. ↗
  34. Zhi-Hua Zhou and Xu-Ying Liu. 2005. Training cost-sensitive neural networks with methods addressing the class imbalance problem. IEEE Transactions on knowledge and data engineering, 18(1):63–77. ↗
  35. Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. 2018. Domain adaptation for semantic segmentation via class-balanced self-training. arXiv preprint arXiv:1810.07911. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/daif-2020-aradic-arabic-document-classification-image/