Elamur Research⌕
☰
home / papers / 2023 / Investigating Zero-shot Cross-lingual Language Understanding for Arabic

Investigating Zero-shot Cross-lingual Language Understanding for Arabic

Paper · 2023 · Zaid Alyafeai and Moataz Ahmed

Summary

The paper asks whether a multilingual model can understand Arabic after being fine-tuned only on other languages, and whether languages with similar morphology transfer better. The authors fine-tune mBERT on one source language at a time and then evaluate it on Arabic without any Arabic training examples. Three tasks are used: natural language inference (XNLI), question answering (xQuAD), and named entity recognition. Before the transfer experiments, they measure morphological richness without supervision. They count words and unique word types in the six-language UN parallel corpus and compare how many tokens different tokenizers produce. Russian, which, like Arabic, is fusional and morphologically rich, transfers best for question answering. For inference it performs about as well as English, close to the in-language Arabic baseline. Thai and Urdu transfer worst. NER transfer falls well short of the Arabic baseline and gives no clear pattern. The study suggests that typological closeness, not only shared script, shapes what multilingual models carry between languages.

Key points

  • Setup: the task stays fixed while the source language varies. mBERT is fine-tuned on language L1 and tested zero-shot on Arabic (L2) for XNLI, xQuAD and NER.
  • On the UN parallel corpus, Arabic has the most unique word types and Russian the second most. Russian reaches 96,292 types from 2,628,322 tokens, while English has 70,330 types from 2,963,479 tokens. Both have fewer running words than Spanish or French.
  • FARASA segmentation of Arabic shrinks the vocabulary sharply but makes sequences longer. The authors present this as a real trade-off for model training.
  • XNLI: English and Russian come near the Arabic baseline, and German is fairly close. Urdu and Thai score around 50% accuracy. Training used 50,000 samples, and the Arabic test set is machine-translated.
  • xQuAD: Russian has the best exact match and F1, followed by Romanian, with Thai last. On NER, which has more than 100 entity types, F1 drops strongly, and F1 and accuracy diverge a great deal.

Interesting ideas

  • I1. Comparing running-word counts with unique-type counts on a parallel corpus gives a cheap, model-free measure of how inflection-heavy a language is.
  • I2. Segmenting words into morphemes reduces vocabulary but lengthens sequences, so any segmentation choice should be judged on both sides of that trade-off.
  • I3. Holding the task fixed and varying only the training source turns zero-shot performance into a probe of how similar two data sources are.
  • I4. Reporting several metrics side by side exposes cases where a single score is misleading, as with the large F1 versus accuracy gap on NER.
  • I5. The authors treat a machine-translated test set as a possible reason why the scores look the way they do, not as neutral ground truth.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Running the same type/token count over our English lemmatizer output on parallel or paraphrased text would show how much sparsity the lemmas actually remove. The count could also flag domains where inflection still fragments the vocabulary.
  • I2 (tokens): When splitting clitics or contractions in the en tokenizer, we should track sentence length growth alongside vocabulary reduction. This would show the cost each split imposes on the downstream parser.
  • I3 (tree): Training gparse on one treebank or UD dialect and testing on another, task unchanged, would rank which treebanks are structurally closest to our target English. That ranking could guide the order and weighting of treebank mixing.
  • I4 (learn): Our evaluations of entity and role labelling should always report a frequency-insensitive metric next to plain accuracy. This keeps a majority "no entity" class from hiding real failures.
  • I5 (answer): When school tests or fable QA sets are machine-translated or LLM-generated, we should tag their provenance. Results on them should be checked against a small human-written control set before we draw conclusions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓King Fahd University of Petroleum and MineralsDhahran · Saudi Arabia54 works · 4,666 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksMultilingual & cross-lingual NLP · 882 works

Cite

Cite as Zaid Alyafeai and Moataz Ahmed. 2023. Investigating Zero-shot Cross-lingual Language Understanding for Arabic. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.26

@inproceedings{alyafeai2023investigating,
  title = {{Investigating Zero-shot Cross-lingual Language Understanding for Arabic}},
  author = {Zaid Alyafeai and Moataz Ahmed},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.26},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.26}
}

References

28 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Khadige Abboud, Olga Golovneva, and Christopher DiPersio. 2022. Cross-lingual transfer for low-resource arabic language understanding. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 225–237.
  2. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for arabic. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Demonstrations, pages 11–16.
  3. Ahmed Abdelali, Hamdy Mubarak, Shammur Absar Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, et al. 2023. Benchmarking arabic ai with large language models. arXiv preprint arXiv:2305.14982. ↗
  4. Mohamed Alkaoud and Mairaj Syed. 2020. On the importance of tokenization in arabic embedding models. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 119–129. ↗
  5. Badr AlKhamissi, Muhammad N ElNokrashy, and Mohamed Gabr. 2020. Deep diacritization: Efficient hierarchical recurrence for improved arabic diacritization. arXiv preprint arXiv:2011.00538. ↗
  6. Zaid Alyafeai, Maged S Alshaibani, Badr AlKhamissi, Hamzah Luqman, Ebrahim Alareqi, and Ali Fadel. 2023. Taqyim: Evaluating arabic nlp tasks using chatgpt models. arXiv preprint arXiv:2306.16322. ↗
  7. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  8. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019. On the cross-lingual transferability of monolingual representations. CoRR, abs/1910.11856. ↗
  9. Kaj Bostrom and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pretraining. arXiv preprint arXiv:2004.03720. ↗
  10. Jonathan H Clark, Dan Garrette, Iulia Turc, and John Wieting. 2021. Canine: Pre-training an efficient tokenization-free encoder for language representation. arXiv preprint arXiv:2103.06874. ↗
  11. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. ↗
  12. Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. arXiv preprint arXiv:1809.05053. ↗
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  14. Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natarajan. 2022. Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages. ↗
  15. Daniela Gerz, Ivan Vulić, Edoardo Ponti, Jason Naradowsky, Roi Reichart, and Anna Korhonen. 2018. Language modeling for morphologically rich languages: Character-aware modeling for word-level prediction. Transactions of the Association for Computational Linguistics, 6:451–465. ↗
  16. Valentin Hofmann, Janet B Pierrehumbert, and Hinrich Schütze. 2020. Generating derivational morphology with bert. arXiv preprint arXiv:2005.00672. ↗
  17. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pages 4411–4421. PMLR. find
  18. Karima Kadaoui, Samar M Magdy, Abdul Waheed, Md Tawkat Islam Khondaker, Ahmed Oumar El-Shangiti, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Tarjamat: Evaluation of bard and chatgpt on machine translation of ten arabic varieties. arXiv preprint arXiv:2308.03051. ↗
  19. Muhammad Khalifa, Muhammad Abdul-Mageed, and Khaled Shaalan. 2021. Self-training pre-trained language models for zero-and few-shot multi-dialectal arabic sequence labeling. arXiv preprint arXiv:2101.04758. ↗
  20. Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Gptaraeval: A comprehensive evaluation of chatgpt on arabic nlp. arXiv preprint arXiv:2305.14976. ↗
  21. David King, Andrea Sims, and Micha Elsner. 2020. Interpreting sequence-to-sequence models for russian inflectional morphology. In Proceedings of the Society for Computation in Linguistics 2020, pages 409–418. ↗
  22. Stav Klein and Reut Tsarfaty. 2020. Getting the## life out of living: How adequate are word-pieces for modelling complex morphology? In Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 204–209. find
  23. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. arXiv preprint arXiv:1804.10959. ↗
  24. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020. Gigabert: Zero-shot transfer learning from english to arabic. In Proceedings of The 2020 Conference on Empirical Methods on Natural Language Processing (EMNLP). find
  25. Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot cross-lingual transfer with multilingual transformers. arXiv preprint arXiv:2005.00633. (Long Papers), pages 1112–1122. Association for Computational Linguistics. ↗
  26. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934. ↗
  27. A Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  28. Mai Oudah, Amjad Almahairi, and Nizar Habash. 2019. The impact of preprocessing on arabic-english statistical and neural machine translation. arXiv preprint arXiv:1906.11751. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alyafeai-2023-investigating-zero-shot-cross-lingual/