Elamur Research⌕
☰
home / papers / 2019 / hULMonA: The Universal Language Model in Arabic

hULMonA: The Universal Language Model in Arabic

Paper · 2019 · Obeida ElJundi, Wissam Antoun, Nour El Droubi et al.

Summary

Before this work, Arabic NLP lacked a pretrained language model built only for Arabic. Transfer learning in Arabic had mostly stopped at reusing word embeddings, and the multilingual BERT had seen little testing on Arabic. The authors built hULMonA, an AWD-LSTM language model trained on Arabic Wikipedia, following the ULMFiT recipe. The pipeline has three stages:

  • pretrain the model on general text;
  • adapt it to the unlabeled text of each target dataset;
  • train a classifier on top with gradual unfreezing.

To control the large Arabic vocabulary, they normalized the text and used MADAMIRA to split off clitics. They also fine-tuned BERT Base Multilingual Cased on the same tasks for comparison. On four sentiment benchmarks covering MSA and dialects, hULMonA beat the previously reported best results. mBERT was strongest on ASTD despite poor Arabic tokenization. The study matters because it shows that full-model transfer, rather than embeddings alone, helps a low-resource, morphologically rich language. The model, data and code were released through the OMA project website.

Key points

  • Pretraining corpus: Arabic Wikipedia up to March 2019, keeping articles of at least 100 characters. This gave 600,559 articles, about 108M words and roughly 4M distinct word forms.

  • Preprocessing:

    • numbers mapped to a special token;
    • Alif and Ta-marbota normalized;
    • punctuation separated, diacritics and non-Arabic tokens dropped;
    • MADAMIRA used to segment prefixes such as the article and suffixes such as possessive pronouns;
    • tokens seen fewer than 5 times replaced by an unknown token.

    Table 3 reports a drop in unique tokens from 4.1M to 9.1K.

  • LM training: a three-layer AWD-LSTM trained for 4 epochs, about 200 minutes per epoch on a GTX 1080. Settings were dropout 0.1, learning rate 3e-3 and batch size 32, with 10% of the data held out.

  • Results (F1 / accuracy) against prior best:

    DatasetPrior besthULMonAmBERT
    HARD93.1 / 93.295.7 / 95.795.7 / 95.7
    ASTD62.0 / 68.767.7 / 69.967.0 / 77.1
    ASTD-B82.5 / 82.485.8 / 86.580.0 / 80.1
    ArSenTD-Lev50.0 / 51.051.1 / 52.451.0 / 51.0

  • mBERT setup and caveat: BERT Base Multilingual Cased was fine-tuned on Colab with batch size 32 and per-dataset learning rates and epoch counts. Its WordPiece tokenizer visibly mangled Arabic sentences, which the authors suspect limited its scores.

Interesting ideas

  • I1. Segmenting clitics with a morphological analyzer before training turns a huge, sparse word inventory into a compact set of stems and affixes, so the model wastes fewer parameters on rare surface forms.
  • I2. Between general pretraining and the supervised task, the model first continues next-token training on the raw target-domain text, absorbing dialect words and domain senses without any labels.
  • I3. During fine-tuning, layers get different learning rates and are unfrozen one at a time from the output end, which protects general knowledge in lower layers when the labeled set is small.
  • I4. Checking what a pretrained tokenizer actually does to the input showed that the multilingual model fragmented Arabic badly, giving a concrete explanation for weak scores.
  • I5. Letting the pretrained model continue short prompts gives a quick, human-readable check that it learned fluent language before any downstream evaluation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The en crate could split English clitics such as 's, n't and 'll into UD-style pieces before any statistical model sees them. This would keep vocabulary small and make each piece map cleanly to a tree node.
  • I2 (learn): gparse's encoder could get a short unlabeled pass over fable, tale and school-test text before parser training, adapting it to that register at no annotation cost.
  • I3 (tree): When gparse is tuned on a small or converted treebank, this schedule could reduce forgetting of what the pretrained encoder already knows.
  • I4 (tokens): A routine audit could compare gparse's subword splits against Mova's own tokens and flag misaligned words. Parse errors could then be traced back to tokenization.
  • I5 (learn): A similar cheap probe could serve as a sanity step before costly training runs, alongside the negative controls.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓American University of BeirutBeirut · Lebanon34 works · 2,561 citations🎓Qatar UniversityDoha · Qatar48 works · 1,026 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Obeida ElJundi, Wissam Antoun, Nour El Droubi, Hazem Hajj, Wassim El-Hajj and Khaled Shaban. 2019. hULMonA: The Universal Language Model in Arabic. Proceedings of the Fourth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-4608

@inproceedings{eljundi2019hulmona,
  title = {{hULMonA: The Universal Language Model in Arabic}},
  author = {Obeida ElJundi, Wissam Antoun, Nour {El Droubi}, Hazem Hajj, Wassim El-Hajj and Khaled Shaban},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Workshop},
  year = {2019},
  doi = {10.18653/v1/w19-4608},
  url = {https://doi.org/10.18653/v1/w19-4608}
}

References

38 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed and Mona Diab. 2012. Toward building a large-scale arabic sentiment lexicon. In Proceedings of the 6th international global Word-Net conference, pages 18–22. *Table 5: Learning rate and number of epochs used for training each dataset* Muhammad Abdul-Mageed and Mona T Diab. 2014. Sana: A large scale multi-genre, multi-dialect lexicon for arabic subjectivity and sentiment analysis. In LREC, pages 1162–1169. https://github.com/google-research/ bert/blob/master/multilingual.md 5 https://tfhub.dev/f/google ↗
  2. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175. ↗
  3. Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual nlp. arXiv preprint arXiv:1307.1662. ↗
  4. Ahmad Al-Sallab, Ramy Baly, Hazem Hajj, Khaled Bashir Shaban, Wassim El-Hajj, and Gilbert Badaro. 2017. Aroma: A recursive deep learning model for opinion mining in arabic as a low resource language. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 16(4):25. find
  5. Abdelghani Dahou, Mohamed Abd Elaziz, Junwei Zhou, and Shengwu Xiong. 2019. Arabic sentiment classification using convolutional neural network and differential evolution algorithm. ↗
  6. Computational Intelligence and Neuroscience, 2019. find
  7. Abdelghani Dahou, Shengwu Xiong, Junwei Zhou, Mohamed Houcine Haddoud, and Pengfei Duan. 2016. Word embeddings and convolutional neural network for arabic sentiment classification. In Proceedings of coling 2016, the 26th international conference on computational linguistics: Technical papers, pages 2418–2427.
  8. Ahmad Al Sallab, Hazem Hajj, Gilbert Badaro, Ramy Baly, Wassim El Hajj, and Khaled Bashir Shaban. 2015. Deep learning models for sentiment analysis in arabic. In Proceedings of the second workshop on Arabic natural language processing, pages 9–17. ↗
  9. Gilbert Badaro, Ramy Baly, Hazem Hajj, Wassim El-Hajj, Khaled Bashir Shaban, Nizar Habash, Ahmad Al-Sallab, and Ali Hamdi. 2019. A survey of opinion mining in arabic: A comprehensive system perspective covering challenges and advances in tools, resources, models, applications, and visualizations. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 18(3):27. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  11. Gilbert Badaro, Ramy Baly, Hazem Hajj, Nizar Habash, and Wassim El-Hajj. 2014. A large scale arabic sentiment lexicon for arabic opinion mining. In Proceedings of the EMNLP 2014 workshop on arabic natural language processing (ANLP), pages 165–173.
  12. Ashraf Elnagar, Yasmin S Khalifa, and Anas Einea. 2018. Hotel arabic-reviews dataset construction for sentiment analysis applications. In Intelligent Natural Language Processing: Trends and Applications, pages 35–52. Springer. ↗
  13. Rehab M Duwairi. 2015. Sentiment analysis for dialectical arabic. In 2015 6th International Conference on Information and Communication Systems (ICICS), pages 166–170. IEEE. find
  14. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. arXiv preprint arXiv:1803.11138. ↗
  15. Gilbert Badaro, Obeida El Jundi, Alaa Khaddaj, Alaa Maarouf, Raslan Kain, Hazem Hajj, and Wassim El-Hajj. 2018. Ema at semeval-2018 task 1: Emotion mining for arabic. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 236–244.
  16. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146. ↗
  17. Ramy Baly, Hazem Hajj, Nizar Habash, Khaled Bashir Shaban, and Wassim El-Hajj. 2017. A sentiment treebank and morphologically enriched recursive deep models for effective sentiment analysis in arabic. ACM Transactions on Asian and Low-Resource Language Information Processing (TAL-LIP), 16(4):23. ↗
  18. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar. Association for Computational Linguistics. ↗
  19. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302. ↗
  20. Ramy Baly, Alaa Khaddaj, Hazem Hajj, Wassim El-Hajj, and Khaled Shaban. 2018. Arsentd-lev: A multi-topic corpus for target-based sentiment analysis in arabic levantine tweets. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France. European Language Resources Association (ELRA). ↗
  21. Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In International conference on machine learning, pages 1188– 1196. ↗
  22. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535. ↗
  23. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. you need. In Advances in neural information processing systems, pages 5998–6008. ↗
  24. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2017. Regularizing and optimizing lstm language models. arXiv preprint arXiv:1708.02182. ↗
  25. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. ↗
  26. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. ↗
  27. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. ↗
  28. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. Semeval-2018 task 1: Affect in tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 1–17.
  29. Mahmoud Nabil, Mohamed Aly, and Amir Atiya. 2015. Astd: Arabic sentiment tweets dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2515–2519. ↗
  30. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? In Advances in neural information processing systems, pages 3320–3328. ↗
  31. Arfath Pasha, Mohamed Al-Badrashiny, Mona T Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In LREC, volume 14, pages 1094–1101.
  32. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. ↗
  33. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. ↗
  34. Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. 2017. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444. ↗
  35. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2.amazonaws. com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf. ↗
  36. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
  37. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th annual meeting of the association for computational linguistics, pages 384–394. Association for Computational Linguistics.
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/eljundi-2019-hulmona-universal-language-model-arabic/