Elamur Research⌕
☰
home / papers / 2022 / Morphosyntactic Tagging with Pre-trained Language Models for Arabic and its Dialects

Morphosyntactic Tagging with Pre-trained Language Models for Arabic and its Dialects

Paper · 2022 · Go Inoue, Salam Khalifa and Nizar Habash

Summary

The paper asks how well fine-tuned pre-trained transformers handle full morphosyntactic tagging in Arabic. In this setting the tag combines POS, inflectional features and clitics, so the tagset can run into the thousands. The authors cover four varieties with very different resources: MSA, Gulf, Egyptian and Levantine. Using CAMeLBERT models, they compare several fine-tuning setups under controlled conditions:

  • different amounts of training data;
  • one complex tag versus separate per-feature predictions (unfactored vs factored);
  • with or without an external morphological analyzer;
  • first training on other dialects before the target one.

Their systems beat the previous best results in every variety, with the largest jump in low-resource Levantine. Per-feature modelling and good analyzers help most when data is scarce. Borrowing annotated data from better-resourced dialects also helps a weak dialect. The work matters because morphological tagging feeds parsing, translation and other tasks, and it gives practical guidance for low-resource language varieties. Models and data are released publicly.

Key points

  • Absolute improvements over prior best results by variety: MSA +2.6%, GLF +2.8%, EGY +1.6%, LEV +8.3%.
  • Annotated data sizes in tokens:
    • PATB (MSA): 629k
    • Gumar (GLF, CODA spelling): 202k
    • ARZTB (EGY): 175k
    • Curras (LEV): 57k
  • Analyzers: MSA uses the hand-built SAMA database and EGY uses CALIMA. GLF and LEV rely on analyzers derived automatically from training data via paradigm completion.
  • The factored model beats the unfactored one with little data, but the gap shrinks as data grows. Adding an analyzer helps everywhere except LEV, where noisy spelling and a weaker analyzer make it hurt.
  • Error analysis:
    • Case is the main error source in MSA, while POS dominates in the dialects.
    • Pronominal enclitic errors are about 2% in MSA versus about 17% in dialects, linked to a much less skewed person distribution.
    • Some gold annotation errors, such as genitive diptotes marked accusative, were found.

Interesting ideas

  • I1. Predicting each morphological feature as its own output, instead of one giant combined tag, works better when data is small, since rare feature combinations no longer become unseen labels.
  • I2. An external lexical resource helps only when it is high quality, and a noisy, automatically built analyzer can lower accuracy, so the resource should be validated rather than trusted by default.
  • I3. Training first on a related, better-resourced variety and then on the small target variety gives a low-resource boost without needing new annotation.
  • I4. Errors on the full tag are broken down by which features failed and tied back to label distribution skew, which explains why some features are harder in some varieties.
  • I5. When every system, old and new, makes the same "error", the gold annotation may be wrong, so such shared errors flag candidates for treebank revision.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The en tagger could predict UD FEATS slot by slot (Number, Tense, Person…) rather than as whole feature strings, which should help on rare inflection combinations.
  • I2 (tokens): Lexicon or morphology constraints on the tagger should be switchable and measured per domain, keeping them only where they actually improve dev accuracy.
  • I3 (tree): gparse could be trained on a large UD English treebank first, then fine-tuned on a small converted or in-domain treebank (e.g., tales and school texts) instead of mixing everything at once.
  • I4 (learn): Evaluation reports could break down parse or tag errors by individual feature or relation and relate them to label frequencies, guiding where repair rules are worth inducing.
  • I5 (learn): Tokens where all witness parsers agree but contradict gold could be queued as suspected annotation errors before being counted against the model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates122 works · 4,602 citations🎓Stony Brook UniversityStony Brook · United States43 works · 2,571 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksMorphology · 405 worksPart-of-speech tagging · 126 works

Cite

Cite as Go Inoue, Salam Khalifa and Nizar Habash. 2022. Morphosyntactic Tagging with Pre-trained Language Models for Arabic and its Dialects. Findings of the Association for Computational Linguistics: ACL 2022.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.findings-acl.135

@inproceedings{inoue2022morphosyntactic,
  title = {{Morphosyntactic Tagging with Pre-trained Language Models for Arabic and its Dialects}},
  author = {Go Inoue, Salam Khalifa and Nizar Habash},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.135},
  url = {https://doi.org/10.18653/v1/2022.findings-acl.135}
}

References

43 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. ↗
  2. Ibrahim Abu Farha, Wajdi Zaghouani, and Walid Magdy. 2021. Overview of the WANLP 2021 shared task on sarcasm and sentiment detection in Arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 296–305, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  3. Bashar Alhafni, Nizar Habash, and Houda Bouamor. 2020. Gender-aware reinflection using linguistically enhanced neural models. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 139–150, Barcelona, Spain (Online). Association for Computational Linguistics. find
  4. Ramy Baly, Hazem Hajj, Nizar Habash, Khaled Bashir Shaban, and Wassim El-Hajj. 2017. A sentiment treebank and morphologically enriched recursive deep models for effective sentiment analysis in Arabic. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 16(4):23. ↗
  5. Kareem Darwish. 2014. Arabizi Detection and Conversion to Arabic. In Proceedings of the Workshop for Arabic Natural Language Processing (WANLP), pages 217–224, Doha, Qatar. ↗
  6. Kareem Darwish, Hamdy Mubarak, Ahmed Abdelali, Mohamed Eldesouki, Younes Samih, Randah Alharbi, Mohammed Attia, Walid Magdy, and Laura Kallmeyer. 2018. Multi-dialect Arabic pos tagging: A CRF approach. In Proceedings of the Language Resources and Evaluation Conference (LREC), Miyazaki, Japan.
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  8. Mona Diab, Nizar Habash, Owen Rambow, and Ryan Roth. 2013. LDC Arabic treebanks and associated corpora: Data divisions manual. arXiv preprint arXiv:1309.5652. ↗
  9. Mona Diab, Kadri Hacioglu, and Daniel Jurafsky. 2004. Automatic Tagging of Arabic Text: From Raw Text to Base Phrase Chunks. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 149–152, Boston, MA.
  10. Ahmed El Kholy and Nizar Habash. 2012. Orthographic and morphological processing for English– Arabic statistical machine translation. Machine Translation, 26(1-2):25–45. ↗
  11. Ramy Eskander, Nizar Habash, and Owen Rambow. 2013. Automatic extraction of morphological lexicons from morphologically annotated corpora. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1032–1043, Seattle, Washington, USA. find
  12. Ramy Eskander, Nizar Habash, Owen Rambow, and Arfath Pasha. 2016. Creating resources for Dialectal Arabic from a single annotation: A case study on Egyptian and Levantine. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 3455–3465, Osaka, Japan.
  13. Abbas Ghaddar, Yimeng Wu, Ahmad Rashid, Khalil Bibi, Mehdi Rezagholizadeh, Chao Xing, Yasheng Wang, Duan Xinyu, Zhefeng Wang, Baoxing Huai, Xin Jiang, Qun Liu, and Philippe Langlais. 2022. Jaber and saber: Junior and senior arabic bert. ↗
  14. David Graff, Mohamed Maamouri, Basma Bouziri, Sondos Krouna, Seth Kulick, and Tim Buckwalter. 2009. Standard Arabic Morphological Analyzer (SAMA) Version 3.1. Linguistic Data Consortium LDC2009E73. find
  15. Nizar Habash, Fadhl Eryani, Salam Khalifa, Owen Rambow, Dana Abdulrahim, Alexander Erdmann, Reem Faraj, Wajdi Zaghouani, Houda Bouamor, Nasser Zalmout, Sara Hassan, Faisal Al shargi, Sakhar Alkhereyf, Basma Abdulkareem, Ramy Eskander, Mohammad Salameh, and Hind Saddiki. 2018. Unified guidelines and resources for Arabic dialect orthography. In Proceedings of the Language Resources and Evaluation Conference (LREC), Miyazaki, Japan.
  16. Nizar Habash, Ramy Eskander, and Abdelati Hawwari. 2012. A Morphological Analyzer for Egyptian Arabic. In Proceedings of the Workshop of the Special Interest Group on Computational Morphology and Phonology (SIGMORPHON), pages 1–9, Montréal, Canada.
  17. Nizar Habash, Abdelhadi Soudi, and Tim Buckwalter. 2007. On Arabic Transliteration. In A. van den Bosch and A. Soudi, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods, pages 15–22. Springer, Netherlands. ↗
  18. Nizar Y Habash. 2010. Introduction to Arabic natural language processing. Morgan & Claypool Publishers. ↗
  19. Nawar Halabi. 2016. Modern standard Arabic phonetics for speech synthesis. Ph.D. thesis, University of Southampton. find
  20. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  21. Go Inoue, Hiroyuki Shindo, and Yuji Matsumoto. 2017. Joint Prediction of Morphosyntactic Categories for Fine-Grained Arabic Part-of-Speech Tagging Exploiting Tag Dictionary Information. In Proceedings of the Conference on Computational Natural Language Learning (CoNLL), pages 421–431, Vancouver, Canada.
  22. Mustafa Jarrar, Nizar Habash, Diyam Akra, and Nasser Zalmout. 2014. Building a Corpus for Palestinian Arabic: A Preliminary Study. In Proceedings of the Workshop for Arabic Natural Language Processing (WANLP), pages 18–27, Doha, Qatar.
  23. Muhammad Khalifa, Muhammad Abdul-Mageed, and Khaled Shaalan. 2021. Self-training pre-trained language models for zero- and few-shot multi-dialectal Arabic sequence labeling. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 769–782, Online. Association for Computational Linguistics.
  24. Salam Khalifa, Nizar Habash, Fadhl Eryani, Ossama Obeid, Dana Abdulrahim, and Meera Al Kaabi. 2018. A morphologically annotated corpus of emirati Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Miyazaki, Japan.
  25. Salam Khalifa, Nasser Zalmout, and Nizar Habash. 2020. Morphological analysis and disambiguation for Gulf Arabic: The interplay between resources and methods. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 3895– 3904, Marseille, France. European Language Resources Association.
  26. Dan Kondratyuk. 2019. Cross-lingual lemmatization and morphology tagging with two-stage multilingual BERT fine-tuning. In Proceedings of the 16th Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 12–18, Florence, Italy. Association for Computational Linguistics. find
  27. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020. An empirical study of pre-trained transformers for Arabic information extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4727–4734, Online. Association for Computational Linguistics.
  28. Mohamed Maamouri, Ann Bies, Tim Buckwalter, and Wigdan Mekki. 2004. The Penn Arabic Treebank: Building a Large-Scale Annotated Arabic Corpus. In Proceedings of the International Conference on Arabic Language Resources and Tools, pages 102– 109, Cairo, Egypt. find
  29. Mohamed Maamouri, Ann Bies, Seth Kulick, Dalila Tabessi, and Sondos Krouna. 2012. Egyptian Arabic Treebank DF Parts 1-8 V2.0 - LDC catalog numbers LDC2012E93, LDC2012E98, LDC2012E89, LDC2012E99, LDC2012E107, LDC2012E125, LDC2013E12, LDC2013E21. find
  30. Yuval Marton, Nizar Habash, and Owen Rambow. 2013. Dependency parsing of modern standard Arabic with lexical and inflectional features. Computational Linguistics, 39(1):161–194. ↗
  31. Arya D. McCarthy, Ekaterina Vylomova, Shijie Wu, Chaitanya Malaviya, Lawrence Wolf-Sonkin, Garrett Nicolai, Christo Kirov, Miikka Silfverberg, Sabrina J. Mielke, Jeffrey Heinz, Ryan Cotterell, and Mans Hulden. 2019. find
  32. The SIGMORPHON 2019 shared task: Morphological analysis in context and cross-lingual transfer for inflection. In Proceedings of the 16th Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 229–244, Florence, Italy. Association for Computational Linguistics. find
  33. Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157. ↗
  34. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  35. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. MADAMIRA: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), pages 1094–1101, Reykjavik, Iceland.
  36. Fatiha Sadat and Nizar Habash. 2006. Combination of Arabic preprocessing schemes for statistical machine translation. In Proceedings of the International Conference on Computational Linguistics and the Conference of the Association for Computational Linguistics (COLING-ACL), pages 1–8, Sydney, Australia.
  37. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained Arabic dialect identification. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 1332–1344, Santa Fe, New Mexico, USA.
  38. Younes Samih, Mohamed Eldesouki, Mohammed Attia, Kareem Darwish, Ahmed Abdelali, Hamdy Mubarak, and Laura Kallmeyer. 2017. Learning from relatives: Unified dialectal Arabic segmentation. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 432–441, Vancouver, Canada. Association for Computational Linguistics.
  39. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Huggingface’s transformers: State-of-the-art natural language processing. ↗
  40. Nasser Zalmout. 2020. Morphological Tagging and Disambiguation in Dialectal Arabic Using Deep Learning Architectures. Ph.D. thesis, New York University. find
  41. Nasser Zalmout and Nizar Habash. 2017. Don’t throw those morphological analyzers away just yet: Neural morphological disambiguation for Arabic. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 704–713, Copenhagen, Denmark.
  42. Nasser Zalmout and Nizar Habash. 2019. Adversarial multitask learning for joint multi-feature and multi-dialect morphological modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1775–1786, Florence, Italy. Association for Computational Linguistics.
  43. Nasser Zalmout and Nizar Habash. 2020. Joint diacritization, lemmatization, normalization, and fine-grained morphological tagging. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8297–8307, Online. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/inoue-2022-morphosyntactic-tagging-pre-trained-language/