Elamur Research⌕
☰
home / papers / 2024 / Computational Approaches to Arabic-English Code-Switching

Computational Approaches to Arabic-English Code-Switching

Summary

This 2021 PhD thesis from the German University in Cairo targets named entity recognition (NER) on text that mixes Arabic and English. Such mixing is very common on Egyptian social media and can happen even inside a single word. The work first sets Modern Standard Arabic NER baselines, starting with a feature-based CRF and moving to deep neural taggers with contextual embeddings.

It then builds what the author describes as the first Arabic-English code-switched corpus annotated for NER. The corpus draws on tweets, translated material and transcribed speech. Two strategies are used to improve taggers on this mixed data:

  • pretraining embeddings on code-switched text, including a new model called KERMIT;
  • augmenting the training data.

Finally, the thesis treats intra-word mixing as a language identification problem. A word is split into language-labelled segments, and each segment is also marked for whether it belongs to a named entity. According to the abstract, every proposed enhancement improved NER on code-switched input. This matters because no earlier work had covered these core tasks for this language pair.

Key points

  • For Arabic NER, the thesis compares a CRF tagger (baseline features plus embedding-derived features) with deep learning taggers. It includes hyper-parameter tuning and contextual embeddings.

  • The code-switched NER corpus combines three sources: a Twitter set, a translated set and a transcribed speech set. Experiments compare training on several monolingual sets against training on the code-switched set.

  • The code-switched embedding models studied are:

    • a baseline;
    • Contextual String Embeddings;
    • BERT;
    • ELECTRA;
    • the newly proposed KERMIT.

    All are judged both intrinsically and extrinsically, by their effect on NER.

  • The augmentation techniques tested are word embedding substitution, a modified Easy Data Augmentation (EDA) and back-translation.

  • For intra-word language identification, the baselines are Naïve Bayes and character BiLSTM models. The main approach is Segmental RNNs (SegRNN), with and without word embeddings, evaluated on a main set and on a coarse-grained NE setting. The available excerpt contains only the abstract and contents, so no scores are given here.

Interesting ideas

  • I1. The intra-word language tagger also flags named entities, so a proper name gets its own label rather than being forced into one language or the other; this separates "what language is this piece" from "is this a name".
  • I2. A segmental model labels whole spans inside a word, so a mixed token such as a foreign stem with a native affix is split and tagged in a single step rather than character by character.
  • I3. Before investing in a costly mixed dataset, the thesis tests whether combining several monolingual datasets can stand in for genuinely mixed annotated data.
  • I4. New embedding models are scored both intrinsically and by their downstream effect on the task, so a better-looking representation must also prove its worth in the tagger.
  • I5. Augmentation is assembled from three different mechanisms (embedding-neighbour substitution, an adapted EDA and back-translation), and each is compared on the same tagger.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): the tokenizer and tagger could emit a separate name/foreign-material flag alongside POS, so imported names and borrowings do not distort morphology and lemma guesses.
  • I2 (tokens): span-level segmentation of hybrid English forms (brand-name verbs, hyphenated coinages) would give the morphology layer explicit, inspectable stem/affix boundaries.
  • I3 (tree): the same comparison could check whether merging treebanks from different UD dialects or genres matches a small in-domain treebank, before annotating more.
  • I4 (learn): any new feature or teacher tier could face both a local probe and a downstream parse/answer score, with negative controls, before adoption.
  • I5 (learn): silver data could be expanded in similar ways, keeping only variants on which the witness parsers still agree, so labels stay trustworthy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Multilingual & cross-lingual NLP Semitic languages

Cite

Cite as Caroline Sabty. 2024. Computational Approaches to Arabic-English Code-Switching. arXiv:2410.13318.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.13318 · arXiv 2410.13318

@misc{sabty2024computational,
  title = {{Computational Approaches to Arabic-English Code-Switching}},
  author = {Caroline Sabty},
  year = {2024},
  doi = {10.48550/arxiv.2410.13318},
  eprint = {2410.13318},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.13318}
}

References

64 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [55] Chiu, J. P., and Nichols, E. Named entity recognition with bidirectional lstm-cnns. Transactions of the Association for Computational Linguistics 4 (2016), 357–370. our card
  2. [56] Chollet, F., et al. Keras. https://keras.io, 2015. ↗
  3. [57] Clark, K., Luong, M.-T., Le, Q. V., and Manning, C. D. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 (2020). ↗
  4. [58] Collobert, R., and Weston, J. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning (2008), pp. 160–167. find
  5. [59] Collobert, R., Weston, Kavukcuoglu, K., and Kuksa, P. Natural language processing (almost) from scratch. Journal of Machine Learning Research 12, Aug (2011), 2493–2537. our card
  6. [60] Cotterell, R., Müller, T., Fraser, A., and Schütze, H. Labeled morphological segmentation with semi-markov models. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning (2015), pp. 164–174. find
  7. [61] Cotterell, R., Renduchintala, A., Saphra, N., and Callison-Burch, C. An algerian arabic-french code-switched corpus. In Workshop on Free/Open-Source Arabic Corpora and Corpora Processing Tools Workshop Programme (2014), p. 34. find
  8. [62] Creutz, M., and Lagus, K. Unsupervised discovery of morphemes. In Proceedings of the ACL-02 Workshop on Morphological and Phonological Learning (July 2002), Association for Computational Linguistics, pp. 21– 30. find
  9. [63] Cui, X., Goel, V., and Kingsbury, B. Data Augmentation for Deep Neural Network Acoustic Modeling. IEEE/ACM Transactions on Audio, Speech, and Language Processing 23, 9 (2015), 1469–1477. find
  10. [64] Darwish, K., Habash, N., Abbas, M., Al-Khalifa, H., Al-Natsheh, H. T., El-Beltagy, S. R., Bouamor, H., Bouzoubaa, K., J., Bottou, L., Karlen, M., BIBLIOGRAPHY Cavalli-Sforza, V., El-Hajj, W., et al. A panoramic survey of natural language processing in the arab world. arXiv preprint arXiv:2011.12631 (2020). ↗
  11. [65] Das, A., and Gambäck, B. Identifying languages at the word level in code-mixed indian social media text. In Proceedings of the 11th International Conference on Natural Language Processing (2014), International Institute of Information Technology Goa, India. find
  12. [66] Derczynski, L., Nichols, E., van Erp, M., and Limsopatham, N. Results of the wnut2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text (2017), pp. 140–147. our card
  13. [67] Devarakonda, A., Naumov, M., and Garland, M. Adaptive batch sizes for training deep neural networks. arXiv preprint arXiv:1712.02029 (2017). ↗
  14. [68] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). ↗ our card
  15. [69] Diaz, G. I., Fokoue-Nkoutche, A., Nannicini, G., and Samulowitz, H. An effective algorithm for hyperparameter optimization of neural networks. IBM Journal of Research and Development 61, 4/5 (2017), 9–1. find
  16. [70] Doddington, G. R., Mitchell, A., Przybocki, M. A., Ramshaw, L. A., Strassel, S. M., and Weischedel, R. M. The automatic content extraction (ace) program-tasks, data, and evaluation. In Lrec (2004), Lisbon, pp. 837–840. our card
  17. [71] Dong, Z., Yao, Z., Cai, Y., Arfeen, D., Gholami, A., Mahoney, M. W., and Keutzer, K. Hawq-v2: Hessian aware trace-weighted quantization of neural networks. arXiv preprint arXiv:1911.03852 (2019). ↗
  18. [72] Dozat, T. Incorporating nesterov momentum into adam. In ICLR (2016), pp. 1–4. find
  19. Adabatch: BIBLIOGRAPHY [73] Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research 12, 7 (2011). find
  20. [74] Eberhand, D., Simons, G. F., and Fenning, C. Ethnologue: Languages of the world, 23rd edn. dallas, tx: Sil international, 2020. find
  21. [76] El Bazi, I., and Laachfoubi, N. Arabic named entity recognition using deep learning approach. International Journal of Electrical & Computer Engineering (2088-8708) 9, 3 (2019). find
  22. [77] El-Shishtawy, T., and El-Ghannam, F. An accurate arabic root-based lemmatizer for information retrieval purposes. arXiv:1203.3584 (2012). ↗ our card
  23. [78] Elfardy, H., Al-Badrashiny, M., and Diab, M. Code switch point detection in arabic. In International Conference on Application of Natural Language to Information Systems (2013), Springer, pp. 412–416. find
  24. [79] Elfardy, H., and Diab, M. Token level identification of linguistic code switching. In Proceedings of COLING 2012: Posters (2012), pp. 287–296. find
  25. [80] ElKateb, S., Black, W., Rodrı́guez, H., Alkhalifa, M., Vossen, P., Pease, A., and Fellbaum, C. Building a WordNet for Arabic. In LREC (2006), pp. 29–34. find
  26. [81] Elman, J. L. Finding structure in time. Cognitive science 14, 2 (1990), 179–211. find
  27. [82] Eskander, R., Al-Badrashiny, M., Habash, N., and Rambow, O. Foreign words and the automatic processing of arabic social media text written in roman script. In Proceedings of The First Workshop on Computational Approaches to Code Switching (2014), pp. 1–12. find
  28. [83] Farghaly, A., and Shaalan, K. Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing (TALIP) 8, 4 (2009), 1–22. arXiv preprint BIBLIOGRAPHY [84] Faruqui, M., and Dyer, C. Improving vector space word representations using multilingual correlation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics (2014), pp. 462–471. our card
  29. [85] Finkel, J. R., Grenager, T., and Manning, C. D. Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05) (2005), pp. 363–370. our card
  30. [86] Fouad, M. M., Mahany, A., Aljohani, N., Abbasi, R. A., and Hassan, S.-U. Arwordvec: Efficient word embedding models for arabic tweets. Soft Computing 24, 11 (2020), 8061–8068. find
  31. [87] Galliani, P., Dezfouli, A., Bonilla, E., and Quadrianto, N. Gray-box inference for structured gaussian process models. In Artificial Intelligence and Statistics (2017), PMLR, pp. 353–361. find
  32. [88] Gao, Y., Feng, J., Liu, Y., Hou, L., Pan, X., and Ma, Y. Code-switching sentence generation by bert and generative adversarial networks. In INTERSPEECH (2019), pp. 3525–3529. find
  33. [89] Geetha, P., Chandu, K., and Black, A. W. Tackling code-switched ner: Participation of cmu. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching (2018), pp. 126–131. find
  34. [90] Gers, F. A., Schmidhuber, J., and Cummins, F. Learning to forget: Continual prediction with lstm. Neural computation 12, 10 (2000), 2451– 2471. find
  35. [91] Glorot, X., Bordes, A., and Bengio, Y. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics (2011), JMLR Workshop and Conference Proceedings, pp. 315–323. find
  36. [92] Gordon, M. A., Duh, K., and Andrews, N. Compressing bert: Studying the effects of weight pruning on transfer learning. arXiv preprint arXiv:2002.08307 (2020). ↗ our card
  37. BIBLIOGRAPHY [93] Gouws, S., Bengio, Y., and Corrado, G. Bilbowa: Fast bilingual distributed representations without word alignments. In Proceedings of the 32nd International Conference on Machine Learning (2015), pp. 748–756. find
  38. [94] Graves, A., Fernández, S., Gomez, F., and Schmidhuber, J. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning (2006), pp. 369–376. find
  39. [95] Gridach, M. Character-aware neural networks for arabic named entity recognition for social media. In Proceedings of the 6th workshop on South and Southeast Asian natural language processing (WSSANLP2016) (2016), pp. 23–32. find
  40. [96] Grönroos, S.-A., Virpioja, S., Smit, P., and Kurimo, M. Morfessor flatcat: An hmm-based method for unsupervised and semi-supervised learning of morphology. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers (2014), pp. 1177–1185. find
  41. [97] Gudikandula, P. Recurrent neural networks and lstm explained. https://medium.com/@shubhidabral/bert-alert-part-1-4fc67809cad, 2019. ↗
  42. [98] Guo, J., Che, W., Wang, H., and Liu, T. Revisiting embedding features for simple semi-supervised learning. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) find
  43. [99] Gupta, D., Tripathi, S., Ekbal, A., and Bhattacharyya, P. A hybrid approach for entity extraction in code-mixed social media data. MONEY 25 (2016), 66. find
  44. [100] Gupta, P., Schütze, H., and Andrassy, B. Table filling multi-task recurrent neural network for joint entity and relation extraction. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (2016), pp. 2537–2547. find
  45. [101] Habibi, M., Weber, L., Neves, M., Wiegandt, D. L., and Leser, U. Deep learning with word embeddings improves biomedical named entity recognition. Bioinformatics 33, 14 (2017), i37–i48. find
  46. BIBLIOGRAPHY [102] Hailu, T. T., Yu, J., and Fantaye, T. G. Pre-trained Word Embedding based Parallel Text Augmentation Technique for Low-Resource NMT in Favor of Morphologically Rich Languages. In Proceedings of the 3rd International Conference on Computer Science and Application Engineering (2019), pp. 1–5. find
  47. [103] Hailu, T. T., Yu, J., Fantaye, T. G., et al. Intrinsic and Extrinsic Automatic Evaluation Strategies for Paraphrase Generation Systems. Journal of Computer and Communications 8, 02 (2020), 1. find
  48. [104] Hale, S. A. Global connectivity and multilinguals in the twitter network. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2014), pp. 833–842. find
  49. [105] Hamed, I., Elmahdy, M., and Abdennadher, S. Building a First Language Model for Code-switch Arabic-English. Procedia Comput. Sci. 117 (2017), 208–216. find
  50. [106] Hamed, I., Elmahdy, M., and Abdennadher, S. Collection and analysis of code-switch egyptian arabic-english speech corpus. In LREC (2018), pp. 208–216. find
  51. [107] Hamed, I., Elmahdy, M., and Abdennadher, S. Collection and analysis of code-switch egyptian arabic-english speech corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) (2018). find
  52. [108] Hamed, I., Zhu, M., Elmahdy, M., Abdennadher, S., and Vu, N. T. Code-switching language modeling with bilingual word embeddings: A case study for egyptian arabic-english. In International Conference on Speech and Computer (2019), Springer, pp. 160–170. find
  53. [109] Hammarström, H., and Borin, L. Unsupervised learning of morphology. Computational Linguistics 37, 2 (2011), 309–350. find
  54. [110] Hanisch, D., Fundel, K., Mevissen, H.-T., Zimmer, R., and Fluck, J. Prominer: rule-based protein and gene entity recognition. BMC bioinformatics 6, 1 (2005), 1–9. find
  55. BIBLIOGRAPHY [111] Helwe, C., Dib, G., Shamas, M., and Elbassuoni, S. A semi-supervised bert approach for arabic named entity recognition. In Proceedings of the Fifth Arabic Natural Language Processing Workshop (2020), pp. 49–57. find
  56. [112] Helwe, C., and Elbassuoni, S. Arabic named entity recognition via deep co-learning. Artificial Intelligence Review 52, 1 (2019), 197–215. find
  57. [113] Hermann, K. M., and Blunsom, P. Multilingual models for compositional distributed semantics. arXiv preprint arXiv:1404.4641 (2014). ↗ our card
  58. [114] Hernández-Garcı́a, A., and König, P. Data Augmentation Instead of Explicit Regularization. arXiv preprint arXiv:1806.03852 (2018). ↗
  59. [115] Hinton, G., Srivastava, N., and Swersky, K. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on 14, 8 (2012). find
  60. [116] Hochreiter, S., and Schmidhuber, J. Long short-term memory. Neural computation 9, 8 (1997), 1735–1780. find
  61. [117] Hossin, M., and Sulaiman, M. A review on evaluation metrics for data classification evaluations. International Journal of Data Mining & Knowledge Management Process 5, 2 (2015), 1. find
  62. [118] Hovy, E., Marcus, Weischedel, R. Ontonotes: the 90% solution. In Proceedings of the human language technology conference of the NAACL, Companion Volume: Short Papers (2006), pp. 57–60. our card
  63. [119] Huang, Z., Xu, W., and Yu, K. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991 (2015). ↗ our card
  64. [120] Hughes, B., Baldwin, T., Bird, S., Nicholson, J., and MacKinlay, A. Reconsidering language identification for written language resources. In International Conference on Language Resources and Evaluation (2006). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sabty-2024-computational-approaches-arabic-english-code/