Elamur Research⌕
☰
home / papers / 2022 / Assessing the Linguistic Knowledge in Arabic Pre-trained Language Models Using Minimal Pairs

Assessing the Linguistic Knowledge in Arabic Pre-trained Language Models Using Minimal Pairs

Summary

The authors test how well Arabic pretrained BERT-based language models have actually learned Arabic grammar. To do this, they manually compiled the first Arabic dataset of minimal pairs (MP). In each pair, the correct sentence differs from the incorrect one by only one word, and this substitution violates a specific grammatical rule. The dataset covers nine phenomena of Modern Standard Arabic (MSA): agreement in verbal and nominal sentences, adjective–noun agreement, and the Idafa construction. Each model was tested by checking whether it assigns a higher probability to the correct sentence than to the incorrect one. Accuracy is the proportion of pairs in which the model chose the correct sentence. Fifteen models were tested, and the results proved modest. Overall accuracy ranges from 47% to 55%, i.e., barely above random guessing. CAMeL-CA and GigaBERT achieved the highest average score. The work is important for two reasons: it is the first systematic test of the linguistic knowledge of Arabic PLMs, and it provides an open resource. For Mova, it is a ready-made example of how to build a similar benchmark for a language with rich morphology and free word order.

Key points

  • Dataset. About 3000 minimal pairs: 1000 for verbal sentences, 1000 for nominal sentences, and 500 each for adjectives and Idafa. Correct and incorrect sentences are equal in number. The dataset was compiled by an Arabic language specialist with a master's degree and validated by three native speakers. The data are available on GitHub (wafa7d/AssessingArabicBERTs).
  • Models. 15 models were tested: six versions of AraBERT, ArabicBERT, GigaBERT, QARiB, ARBERT, MARBERT, and four variants of CAMeLBERT (MSA, CA, DA, Mix).
  • Overall result. The best average accuracy is 55.1%, achieved by CAMeL-CA and GigaBERT. CAMeL-CA was trained only on Classical Arabic, and GigaBERT on MSA.
  • Individual phenomena. CAMeL-CA reached 76.5% on verb–subject gender agreement. GigaBERT scored 76% on the rule for verb–subject number. MARBERT scored 75% on adjectives with "non-rational" (non-human) nouns. Within the paper itself, these figures contradict the claim that no phenomenon exceeded 60%.
  • Easiest and hardest. On average, the models handle best the rule under which a verb preceding its subject does not agree with it in number. They perform worst on subject–predicate number agreement in nominal sentences.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.17

@inproceedings{alrajhi2022assessing,
  title = {{Assessing the Linguistic Knowledge in Arabic Pre-trained Language Models Using Minimal Pairs}},
  author = {Alrajhi, Wafa Abdullah and Al-Khalifa, Hend and AlSalman, Abdulmalik},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.17},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.17}
}

References

8 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Aidan N. Gomez., Lukasz Kaiser. and Polosukhin, I., 2017. Attention is all you need. Advances in neural information processing systems, 30. find
  2. Radford, A., Narasimhan, K., Salimans, T. and Sutskever, I., 2018. Improving language understanding by generative pre-training. find
  3. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D. and Sutskever, I., 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8), p.9. find
  4. Brown, T., et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33, pp.1877-1901. find
  5. Devlin, J., Chang, M.W., Lee, K. and Toutanova, K., 2019, June. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 4171-4186). find
  6. Antoun, W., Baly, F. and Hajj, H., 2020, May. AraBERT: Transformer-based Model for Arabic Language Understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection (pp. 9-15). ↗ our card
  7. Safaya, A., Abdullatif, M. and Yuret, D., 2020. BERT-CNN for Offensive Speech Identification in Social Media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation", Barcelona (online)", International Committee for Computational Linguistics. find
  8. Lan W, Chen Y, Xu W, Ritter A. Gigabert: Zero-shot transfer learning from English to Arabic. In Proceedings of The 2020 Conference on Empirical Methods on Natural Language Processing (EMNLP) 2020. find

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alrajhi-2022-assessing-linguistic-knowledge-arabic-pre/