Elamur Research⌕
☰
home / papers / 2020 / Contextual Embeddings for Arabic-English Code-Switched Data

Contextual Embeddings for Arabic-English Code-Switched Data

Paper · 2020 · Caroline Sabty, Mohamed Islam and Slim Abdennadher

Summary

The paper tackles the shortage of resources for Arabic-English code-switched (CS) text, which is common on social media in Arab countries. Earlier work on this language pair produced only static embeddings, which cannot tell apart senses that depend on context. The authors build two pre-training corpora, CS TRAIN and a larger CS TRAIN ++. They combine three sources: real social-media CS data, monolingual Arabic news-wire text, and synthetic CS text. The synthetic part was made by machine-translating selected Arabic tokens into English, guided by linguistic constraints about where switches tend to happen.

On these corpora they train bilingual contextual models with FLAIR, BERT and ELECTRA. They also introduce KERMIT, a new model trained both as a generator and as a discriminator, each fed a different variant of the input. They test all models intrinsically with BERTScore cosine similarity and on three tasks: NER, sentiment analysis and question answering. KERMIT does best on the intrinsic test and on NER, and leads the transformer models on QA. A FLAIR plus FastText stack wins on sentiment analysis. All models are released as open source.

Key points

  • CS TRAIN has about 105M tokens (10M English, 95M Arabic, 7M sentences). CS TRAIN ++ has about 144M tokens (17M English, 127M Arabic, 9M sentences).
  • Around 30M CS tokens came from translating Modern Standard Arabic text with an NMT API. Translation started at trigger words that often come right before a switch point, such as the Arabic equivalents of "the", "in" and "and".
  • The baseline stacks Arabic pre-trained Pooled FLAIR with FastText, a combination that earlier beat the static Bi-CS embeddings on CS NER.
  • Sentiment analysis (CS-translated book reviews): FLAIR + FastText in the FLAIR framework reached 88.8% F1. In the HuggingFace fine-tuning setup, KERMIT ++ was the best model at 79.9%.
  • QA (CS-translated questions from an Arabic QA corpus): KERMIT ++ scored 39.9% F1 against 30% for AraBERT. AraBERT, trained only on monolingual Arabic, was beaten by the authors' models on all three tasks.

Interesting ideas

  • I1. Synthetic code-switched text was produced by translating only at linguistically motivated trigger positions, not at random, so scarce mixed data could be grown cheaply.
  • I2. In the QA set, only the questions were translated and the answers were left untouched, because rewriting answers could reorder words and break the gold spans.
  • I3. The intrinsic test checks two things: similar CS tokens should get high cosine similarity and unrelated tokens should get low similarity.
  • I4. A monolingual model (AraBERT) served as a reference point to show the gain from training on mixed data.
  • I5. Stacking character-aware contextual embeddings (FLAIR) with classical FastText vectors gave the best sentiment results, beating the larger transformers.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver training data for rare constructions could be built by rule-constrained rewriting of existing sentences, with the constraint rule stored alongside each example to keep it explainable.
  • I2 (answer): When we transform reading-comprehension texts for augmentation, gold answer spans should stay fixed or be re-checked, so the transformation never quietly corrupts the labels.
  • I3 (learn): Every similarity-based check should include unrelated pairs as a negative control, so that a model that rates everything as similar cannot pass.
  • I4 (learn): Evaluations of Mova components could always include a "trained on the wrong distribution" baseline, making clear how much the domain-matched data actually adds.
  • I5 (tokens): Character-level features could be combined with static subword vectors in the tagger as a cheap, robust signal for noisy or out-of-vocabulary tokens.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSemitic languages · 2,908 works

Cite

Cite as Caroline Sabty, Mohamed Islam and Slim Abdennadher. 2020. Contextual Embeddings for Arabic-English Code-Switched Data. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{sabty2020contextual,
  title = {{Contextual Embeddings for Arabic-English Code-Switched Data}},
  author = {Caroline Sabty, Mohamed Islam and Slim Abdennadher},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.20/}
}

References

35 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for arabic. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Demonstrations, pages 11–16.
  2. Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638–1649.
  3. Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019a. Flair: An easy-to-use framework for state-of-the-art nlp. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 54–59.
  4. Alan Akbik, Tanja Bergmann, and Roland Vollgraf. 2019b. Pooled contextualized embeddings for named entity recognition. In NAACL 2019, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics, page 724–728.
  5. Mohamed Aly and Amir Atiya. 2013. Labr: A large scale arabic book reviews dataset. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 494–498. ↗
  6. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  7. Utsab Barman, Amitava Das, Joachim Wagner, and Jennifer Foster. 2014. Code mixing: A challenge for language identification in the language of social media. In Proceedings of the first workshop on computational approaches to code switching, pages 13–23. ↗
  8. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. ↗
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  10. Manaal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 462–471.
  11. Mohammed M Fouad, Ahmed Mahany, Naif Aljohani, Rabeeh Ayaz Abbasi, and Saeed-Ul Hassan. 2020. Ar-wordvec: efficient word embedding models for arabic tweets. Soft Computing, 24(11):8061–8068. find
  12. Yingying Gao, Junlan Feng, Ying Liu, Leijing Hou, Xin Pan, and Yong Ma. 2019. Code-switching sentence generation by bert and generative adversarial networks. In INTERSPEECH, pages 3525–3529. find
  13. Stephan Gouws, Yoshua Bengio, and Greg Corrado. 2015. Bilbowa: Fast bilingual distributed representations without word alignments. In Proceedings of the 32nd International Conference on Machine Learning, pages 748–756. ↗
  14. Scott A Hale. 2014. Global connectivity and multilinguals in the twitter network. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 833–842. find
  15. Injy Hamed, Mohamed Elmahdy, and Slim Abdennadher. 2017. Building a first language model for code-switch arabic-english. Procedia Computer Science, 117:208–216. find
  16. Injy Hamed, Mohamed Elmahdy, and Slim Abdennadher. 2018. Collection and analysis of code-switch egyptian arabic-english speech corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  17. Injy Hamed, Moritz Zhu, Mohamed Elmahdy, Slim Abdennadher, and Ngoc Thang Vu. 2019. Code-switching language modeling with bilingual word embeddings: A case study for egyptian arabic-english. In International Conference on Speech and Computer, pages 160–170. Springer. ↗
  18. Karl Moritz Hermann and Phil Blunsom. 2014. Multilingual models for compositional distributed semantics. arXiv preprint arXiv:1404.4641. ↗
  19. Raki Lachraf, Youcef Ayachi, Ahmed Abdelali, Didier Schwab, et al. 2019. Arbengvec: Arabic-english cross-lingual word embedding model. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 40–48. find
  20. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Bilingual word representations with monolingual quality in mind. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 151–159.
  21. Mohamed Amine Menacer, David Langlois, Denis Jouvet, Dominique Fohr, Odile Mella, and Kamel Smaı̈li. 2019. Machine translation on a parallel code-switched corpus. In Canadian Conference on Artificial Intelligence, pages 426–432. Springer. find
  22. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. ↗
  23. Hussein Mozannar, Karl El Hajal, Elie Maamary, and Hazem Hajj. 2019. Neural arabic question answering. arXiv preprint arXiv:1906.05394. ↗
  24. Carol Myers-Scotton. 1997. Duelling languages: Grammatical structure in codeswitching. Oxford University Press. find
  25. Adithya Pratapa, Gayatri Bhat, Monojit Choudhury, Sunayana Sitaram, Sandipan Dandapat, and Kalika Bali. 2018a. Language modeling for code-mixing: The role of linguistic theory based synthetic data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1543–1553.
  26. Adithya Pratapa, Monojit Choudhury, and Sunayana Sitaram. 2018b. Word embeddings for code-mixed language processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3067–3072. find
  27. Caroline Sabty, Mohamed Elmahdy, and Slim Abdennadher. 2019a. Named entity recognition on arabic-english code-mixed data. In 2019 IEEE 13th International Conference on Semantic Computing (ICSC), pages 93–97. IEEE. find
  28. Caroline Sabty, Ahmed Sherif, Mohamed Elmahdy, and Slim Abdennadher. 2019b. Techniques for named entity recognition on arabic-english code-mixed data. International Journal of Transdisciplinary AI, 1(1):44–63. ↗
  29. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
  30. Shyam Upadhyay, Manaal Faruqui, Chris Dyer, and Dan Roth. 2016. Cross-lingual models of word embeddings: An empirical comparison. arXiv preprint arXiv:1604.00425. ↗
  31. Ivan Vulic and Marie-Francine Moens. 2015. Bilingual word embeddings from non-parallel document-aligned data applied to bilingual lexicon induction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL 2015), volume 2, pages 719–725. ACL; East Stroudsburg, PA. find
  32. Ivan Vulić and Marie-Francine Moens. 2016. Bilingual distributed word representations from document-aligned comparable data. Journal of Artificial Intelligence Research, 55:953–994. ↗
  33. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. ↗
  34. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. ↗
  35. Xinyuan Zhou, Emre Yılmaz, Yanhua Long, Yijie Li, and Haizhou Li. 2020. Multi-encoder-decoder transformer for code-switching speech recognition. arXiv preprint arXiv:2006.10414. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sabty-2020-contextual-embeddings-arabic-english-code/