Elamur Research⌕
☰
home / papers / 2022 / Cross-lingual transfer for low-resource Arabic language understanding

Cross-lingual transfer for low-resource Arabic language understanding

Paper · 2022 · Khadige Abboud, Olga Golovneva and Christopher DiPersio

Summary

The paper looks at how to build a natural language understanding (NLU) component for an Arabic voice assistant when little labeled Arabic data exists. The authors borrow knowledge from English and French, which have rich labeled datasets. They pretrained three BERT-style models on Wikipedia plus large commercial data: Arabic-only, Arabic-English and Arabic-English-French. Each model was then distilled into a smaller student that fits production memory and latency limits. Before pretraining, the Arabic text was preprocessed in two ways: diacritics were removed, and only affixes that carry meaning were split off. An off-the-shelf machine translator also converted English training utterances into Arabic. The models were tested on domain classification (DC) and joint intent classification and NER (IC-NER) in three settings: zero-shot, few-shot and pre-production. Both multilingual models beat the monolingual baseline, even though Arabic differs strongly from the source languages in script and morphology. The work matters because it shows that transfer between distant languages, helped by cheap translation, can speed up support for a new language.

Key points

  • The task inventory is large: 18 domains, 333 intents and 268 entity labels. This is far richer than the three-class NER used in earlier Arabic work such as ANERcorp-style setups.
  • In the few-shot setting, English data reduced SemER by 15.06% relative to the baseline, and French added roughly 5.7% more. The trilingual model reached up to 20.76% lower SemER on IC-NER and 30.21% lower DC error.
  • Adding machine-translated Arabic data improved results by 12.79% for the Bi model and 11.7% for the Tri model. This benefit dropped to a few percent once few-shot or full MTurk Arabic data was included.
  • In the pre-production setting without MT data, the multilingual models beat the monolingual one in 14 of 18 domains on DC and 13 of 18 on IC-NER.
  • Compared with distilmBERT, the in-house pretraining on unlabeled assistant utterances gave up to 25.1 SemER points improvement in zero-shot.

Interesting ideas

  • I1. Split affixes only when they change meaning (pronouns, quantities) and leave purely grammatical ones such as the article attached; this keeps the vocabulary compact while exposing the pieces that NER actually needs.
  • I2. Optional diacritics were stripped (except Shadda) after an internal study showed that inconsistent transcription made them harmful rather than helpful; the lesson is to measure whether an optional marker is applied reliably before trusting it as a feature.
  • I3. Error analysis showed that some failures came from a single Arabic token covering what English expresses with two ("turn on", "hang up"), which shifted the label-sequence distribution; the authors propose down-sampling the mismatched source data as a fix.
  • I4. The paper reports both micro averages (each utterance counts equally) and macro averages (each domain counts equally), and the macro view revealed that the trilingual model mainly helps small domains.
  • I5. Machine-translated training data works as a cheap bootstrap whose value fades as real target data accumulates, so its contribution should be tracked rather than assumed.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The English tokenizer could apply the same test to clitics like possessive 's or contractions, splitting only where the piece fills a role later in the pipeline.
  • I2 (learn): Before adding noisy optional signals (casing, punctuation in user text) as parser features, run an ablation to check whether they help or merely add variance.
  • I3 (frames): Phrasal verbs joined by compound:prt should be merged into one predicate before VerbNet role lookup, so that "hang up" maps to a single frame rather than a verb plus a stray particle.
  • I4 (learn): Evaluations on school tests or fable QA should also report per-category macro scores, so that gains on rare question types are not hidden by frequent ones.
  • I5 (learn): Silver data from witness parsers or LLM teachers could be re-weighted or retired as gold data grows, with the remaining marginal gain checked against a negative control.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢AmazonSeattle · United States112 works · 40,565 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksMultilingual & cross-lingual NLP · 828 worksLow-resource languages · 267 works

Cite

Cite as Khadige Abboud, Olga Golovneva and Christopher DiPersio. 2022. Cross-lingual transfer for low-resource Arabic language understanding. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.21

@inproceedings{abboud2022cross,
  title = {{Cross-lingual transfer for low-resource Arabic language understanding}},
  author = {Khadige Abboud, Olga Golovneva and Christopher DiPersio},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.21},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.21}
}

References

33 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for Arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16. ACL.
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105. ACL. ↗
  3. Mohammad Al-Smadi, Saad Al-Zboon, Yaser Jararweh, and Patrick Juola. 2020. Transfer learning for Arabic named entity recognition with deep neural networks. IEEE Access, 8:37736–37745. ↗
  4. Fahd Alotaibi and Mark Lee. 2014. A hybrid approach to features representation for fine-grained Arabic named entity recognition. In Proceedings of COL-ING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 984–995. Dublin City University and Association for Computational Linguistics.
  5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15. ↗
  6. Giusepppe Attardi. 2015. Wikiextractor. https://github.com/attardi/wikiextractor. ↗
  7. Yassine Benajiba and Rosso Paolo. 2008. Aner-corpdataset. https://camel.abudhabi.nyu.edu/ anercorp/. ↗
  8. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451. ACL. find
  9. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. Advances in Neural Information Processing Systems, 32:7059– 7069. find
  10. Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. Bertje: A dutch BERT model. arXiv preprint arXiv:1912.09582. ↗
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL HLT, pages 4171–4186.
  12. Ibrahim Abu El-Khair. 2016. 1.5 billion words Arabic corpus. arXiv preprint arXiv:1611.04033. ↗
  13. Ahmed El-Kishky, Xingyu Fu, Aseel Addawood, Nahil Sobh, Clare Voss, and Jiawei Han. 2019. Constrained sequence-to-sequence semitic root extraction for enriching word embeddings. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 88–96.
  14. Guillaume Lample et al. 2015. C++ implementation of Neural Machine Translation of Rare Words with Subword Units, with Python API. https://github.com/glample/fastBPE. ↗
  15. Jindřich Libovickỳ, Rudolf Rosa, and Alexander Fraser. 2019. How language-neutral is multilingual BERT? arXiv preprint arXiv:1911.03310. ↗
  16. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203–7219. ACL.
  17. Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 448–462. ACL.
  18. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996– 5001. ACL.
  19. Edoardo Maria Ponti, Julia Kreutzer, Ivan Vulić, and Siva Reddy. 2021. Modelling latent translations for cross-lingual transfer. arXiv preprint arXiv:2107.11353. ↗
  20. Chadi Helwe, Ghassan Dib, Mohsen Shamas, and Shady Elbassuoni. 2020. A semi-supervised BERT approach for Arabic named entity recognition. In Proceedings of the Fifth Arabic NLP Workshop, pages 49–57. ACL.
  21. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108. ↗
  22. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. ↗
  23. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the ACL, pages 1715–1725. ACL.
  24. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104. ACL.
  25. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 4163–4174.
  26. Andrew Johnson, Penny Karanasou, Judith Gaspers, and Dietrich Klakow. 2019. Cross-lingual transfer learning for Japanese named entity recognition. In Proceedings of NAACL HLT (Industry Papers), pages 182–189. find
  27. Kathrein Abu Kwaik, Motaz Saad, Stergios Chatzikyriakidis, and Simon Dobnika. 2018. A lexical distance study of Arabic dialects. Procedia computer science, 142:2–13. find
  28. Muayad Abdul-Halim Ahmad Shamsan and Abdulmajeed Attayib. 2015. Inflectional morphology in arabic and english: a contrastive study. International Journal of English Linguistics, 5(2):139. ↗
  29. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. Aravec: A set of Arabic word embedding models for use in Arabic NLP. Procedia Computer Science, 117:256–265. find
  30. Chengwei Su, Rahul Gupta, Shankar Ananthakrishnan, and Spyros Matsoukas. 2018. A re-ranker scheme for integrating large scale nlu models. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 670–676. IEEE. find
  31. Chao Wang, Judith Gaspers, Thi Ngoc Quynh Do, and Hui Jiang. 2021. Exploring cross-lingual transfer learning with unsupervised machine translation. In ACL-IJCNLP 2021, pages 2011–2020. ACL. find
  32. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844. ACL. ↗
  33. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. OSIAN: Open source international arabic news corpus-preparation and integration into the CLARIN-infrastructure. In Proceedings of the 4th Arabic NLP Workshop, pages 175– 182. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abboud-2022-cross-lingual-transfer-low-resource/