Elamur Research⌕
☰
home / papers / 2021 / Evaluating Various Tokenizers for Arabic Text Classification

Evaluating Various Tokenizers for Arabic Text Classification

Summary

Subword tokenizers such as BPE or SentencePiece are usually language-agnostic and do not take morphology into account. They are also hard to judge outside the tasks they feed into. This matters for Arabic, whose rich inflection and derivation attach morphemes before, inside and after the stem. The authors propose three new tokenizers for Arabic: Stochastic, Disjoint-letter and Morphological. They compare these with three baselines (word, character and SentencePiece) in two ways. The first is unsupervised, measuring tokenization speed and compression factor. The second is supervised, measuring accuracy on sentiment analysis, news classification and poem-meter classification across six public datasets and several vocabulary sizes. No tokenizer wins everywhere. Whole-word tokens do best on small datasets. Characters clearly win on poetry meter. On large datasets most reasonably coarse tokenizers end up with similar accuracy. Even segmentations with no linguistic meaning perform surprisingly well. The study shows that the right choice of tokenizer depends on dataset size, task type and how much morphology the data contains.

Key points

  • Six tokenizers were compared: word, character and SentencePiece as baselines, plus the new Stochastic, Disjoint-letter and Morphological tokenizers.
  • The datasets named in the text include AJGT and LABR for sentiment, DSAC for news and MetRec for poetry meter. Accuracy was reported across vocabulary sizes from 5K to 30K.
  • On small sentiment data, word and morphological tokenizers worked best. On one small dataset, SentencePiece effectively ran out of material at a 10K vocabulary, since that dataset has only about 11K tokens.
  • The character tokenizer was by far the best on MetRec, because meter depends on character-level information. It was the worst on sentiment, due to sparsity.
  • On DSAC (about 2 million tokens, about 140K unique), all tokenizers converged near 0.87 accuracy at 30K. SentencePiece showed a dip at 10K before recovering.

Interesting ideas

  • I1. The paper pairs cheap intrinsic measures (speed and compression factor) with downstream task accuracy, because neither alone tells you whether a tokenizer is good. — For Mova (tokens): Report the speed and compression of the crate en tokenizer and lemmatizer alongside downstream parse quality, so changes to segmentation are judged on both axes.
  • I2. A deliberately random (stochastic) segmentation still reached decent accuracy, which shows that downstream scores can reward segmentations that carry no linguistic sense. — For Mova (learn): Add a random-segmentation or shuffled-lemma baseline as a negative control, so improvements credited to real morphology can be told apart from what any arbitrary split would give.
  • I3. Results were swept over vocabulary sizes and over low- versus high-resource datasets, which revealed reversals that a single setting would hide. — For Mova (learn): When evaluating parser or tagger variants, vary both training-data size and lexicon size, since conclusions drawn at one operating point may flip at another.
  • I4. The best granularity depends on the task: meter classification needs characters, while topic and sentiment do well with words. — For Mova (tokens): Keep character-level features available next to lemmas, so that spelling-, sound- or rhyme-sensitive questions (for example, about poems in tales) are not lost after word-level analysis.
  • I5. Small datasets favor whole words or morphology-aware units, because pieces without meaning need more data to become informative. — For Mova (tokens): This supports Mova's choice of explicit morphological analysis and lemmas over learned subwords, especially for rare words in small school-text domains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

18 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 11 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2106.07540 · arXiv 2106.07540

@misc{alyafeai2021evaluating,
  title = {{Evaluating Various Tokenizers for Arabic Text Classification}},
  author = {Alyafeai, Zaid and Al-shaibani, Maged S. and Ghaleb, Mustafa and Ahmad, Irfan},
  year = {2021},
  doi = {10.48550/arxiv.2106.07540},
  eprint = {2106.07540},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2106.07540}
}

References

13 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gheith A Abandah, Mohammed Z Khedher, Mohammad R Abdel-Majeed, Hamdi M Mansour, Salma F Hulliel, and Lara M Bisharat. Classifying and diacritizing arabic poems using deep recurrent neural networks. Journal of King Saud University-Computer and Information Sciences, 2020. find
  2. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. Farasa: A fast and furious segmenter for arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16, 2016. find our card
  3. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. Pre-training bert on arabic tweets: Practical considerations. arXiv preprint arXiv:2102.10684, 2021. ↗ our card
  4. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. Arbert & marbert: deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785, 2020. ↗ our card
  5. Ibrahim Abu Farha and Walid Magdy. A comparative study of effective approaches for arabic sentiment analysis. Information Processing & Management, 58(2):102438, 2021. ISSN 0306-4573. doi: https://doi.org/10.1016/j.ipm. 2020.102438. URL http://www.sciencedirect.com/science/article/pii/S0306457320309316. ↗
  6. Mahmoud Al-Ayyoub, Abed Allah Khamaiseh, Yaser Jararweh, and Mohammed N Al-Kabi. A comprehensive survey of arabic sentiment analysis. Information processing & management, 56(2):320–342, 2019. find
  7. Baligh M Al-Helali and Sabri A Mahmoud. Arabic online handwriting recognition (aohr) a survey. ACM Computing Surveys (CSUR), 50(3):1–35, 2017. Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3159–3166, 2019. Maged S Al-shaibani, Zaid Alyafeai, and Irfan Ahmad. Meter classification of arabic poems using deep bidirectional recurrent neural networks. find
  8. Pattern Recognition Letters, 2020. Maged S Al-Shaibani, Zaid Alyafeai, and Irfan Ahmad. Metrec: A dataset for meter classification of arabic poetry. Data in Brief, 33:106497, 2020. Mohamed Alkaoud and Mairaj Syed. On the importance of tokenization in arabic embedding models. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 119–129, 2020. Khaled Mohammad Alomari, Hatem M ElSherif, and Khaled Shaalan. Arabic tweets sentimental analysis using machine learning. In International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, pages 602–610. find our card
  9. Springer, 2017. Mohamed Aly and Amir Atiya. Labr: A large scale arabic book reviews dataset. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 494–498, 2013. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104, 2020. AL-Shatnawi Atallah and Khairuddin Omar. A comparative study between methods of arabic baseline detection. In 2009 International Conference on Electrical Engineering and Informatics, volume 1, pages 73–77. ↗ our card
  10. IEEE, 2009. Gilbert Badaro, Ramy Baly, Hazem Hajj, Wassim El-Hajj, Khaled Bashir Shaban, Nizar Habash, Ahmad Al-Sallab, and Ali Hamdi. A survey of opinion mining in arabic: a comprehensive system perspective covering challenges and advances in tools, resources, models, applications, and visualizations. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 18(3):1–52, 2019. mohamed BINIZ. Dataset for arabic classification. 2018. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146, 2017. ISSN 2307-387X. Kaj Bostrom and Greg Durrett. Byte pair encoding is suboptimal for language model pretraining. arXiv preprint arXiv:2004.03720, 2020. Rohan Chitnis and John DeNero. Variable-length word encodings for neural translation models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2088–2093, 2015. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. Ibrahim Abu El-Khair. 1.5 billion words arabic corpus. arXiv preprint arXiv:1611.04033, 2016. Obeida ElJundi, Wissam Antoun, Nour El Droubi, Hazem Hajj, Wassim El-Hajj, and Khaled Shaban. hulmona: The universal language model in arabic. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 68–77, 2019. Imane Guellil, Faical Azouaou, and Marcelo Mendoza. Arabic sentiment analysis: studies, resources, and tools. Social Network Analysis and Mining, 9(1):56, 2019. Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146, 2018. Mahdi Nsaif Jasim. Arabic optical characters recognition by neural network based arabic unicode. 2020. Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. arXiv preprint arXiv:1804.10959, 2018. Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018. Anoop Kunchukuttan and Pushpak Bhattacharyya. Orthographic syllable as basic unit for smt between related languages. arXiv preprint arXiv:1610.00634, 2016. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. ↗ our card
  11. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. Mai Oudah, Amjad Almahairi, and Nizar Habash. The impact of preprocessing on arabic-english statistical and neural machine translation. arXiv preprint arXiv:1906.11751, 2019. Arfath Pasha, Mohamed Al-Badrashiny, Mona T Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In LREC, volume 14, pages 1094–1101, 2014. Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9, 2019. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019. Hassan Sajjad, Fahim Dalvi, Nadir Durrani, Ahmed Abdelali, Yonatan Belinkov, and Stephan Vogel. Challenging language-dependent segmentation for arabic: An application to machine translation and part-of-speech tagging. arXiv preprint arXiv:1709.00616, 2017. Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152. ↗ our card
  12. IEEE, 2012. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015. Pamela Shapiro and Kevin Duh. Bpe and charcnns for translation of morphology: A cross-lingual comparison and analysis. arXiv preprint arXiv:1809.01301, 2018a. Pamela Shapiro and Kevin Duh. Morphological word embeddings for arabic neural machine translation in low-resource settings. In Proceedings of the Second Workshop on Subword/Character LEvel Models, pages 1–11, 2018b. Peter Smit, Sami Virpioja, Stig-Arne Grönroos, and Mikko Kurimo. Morfessor 2.0: Toolkit for statistical morphological segmentation. page 4. ↗ our card
  13. Aalto University, 2014. URL http://urn.fi/URN:NBN:fi:aalto-201409292677. Yi Tay, Vinh Q Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. Charformer: Fast character transformers via gradient-based subword tokenization. arXiv preprint arXiv:2106.12672, 2021. Xinyi Wang, Sebastian Ruder, and Graham Neubig. Multi-view subword regularization. arXiv preprint arXiv:2103.08490, 2021. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. Byt5: Towards a token-free future with pre-trained byte-to-byte models. arXiv preprint arXiv:2105.13626, 2021. Waleed A Yousef, Omar M Ibrahime, Taha M Madbouly, and Moustafa A Mahmoud. Learning meters of arabic and english poems with recurrent neural networks: a step forward for language understanding and synthesis. arXiv preprint arXiv:1905.05700, 2019. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alyafeai-2021-evaluating-various-tokenizers-arabic-text/