Elamur Research⌕
☰
home / papers / 2024 / Cross-Lingual Transfer from Related Languages: Treating Low-Resource Maltese as Multilingual Code-Switching

Cross-Lingual Transfer from Related Languages: Treating Low-Resource Maltese as Multilingual Code-Switching

Paper · 2024 · Kurt Micallef, Nizar Habash, Claudia Borg et al.

Summary

Multilingual encoders transfer poorly to a language whose script differs from that of its related high-resource languages. Maltese is a difficult case. It is Semitic with an Arabic core, but it carries heavy Italian and English influence and is written in Latin letters. Converting all of it to Arabic script helps the Arabic-derived part but moves the rest away from Italian and English. The authors annotated 439 sentences (9,683 tokens) from the Maltese UD treebank with word-level origin labels and trained several etymology classifiers on them. The classifier's predictions decide, token by token, whether a word is transliterated into Arabic script, translated word-by-word into Arabic, Italian or English, or left unchanged. They fine-tuned mBERT and monolingual models (CAMeLBERT, ItalianBERT, BERT) on four downstream tasks, including POS tagging, dependency parsing and NER. Transliterating only Arabic-origin words beat both raw Maltese and blanket processing. Pairing it with translation of the other words worked best. With English translations, mBERT exceeded raw-text fine-tuning on every task. The broader lesson is that, for mixed languages, preprocessing should follow each word's origin.

Key points

  • New resource: etymology tags on 439 sentences / 9,683 tokens from the training set of the Maltese Universal Dependencies Treebank (Čéplö, 2018). The labels are Arabic, Non-Arabic, Mixed (with sub-types for verbs, plurals and univerbations), Code-Switching, Name (Arabic), Name (Non-Arabic) and Symbol. Code, annotations and classifiers are released at github.com/MLRS/malti.
  • Arabic-origin words make up 60% of tokens but only 47% of types. Non-Arabic words make up 16% of tokens and 27% of types. Dictionary counts give a different picture: 32.4% Arabic, 52.5% Italian/Sicilian and 6.1% English. The gap arises because Arabic-origin words, including function words, are frequent in running text.
  • Several classifiers were compared:
    • a training-free Translation heuristic based on edit distances between a token and its translations or transliteration;
    • an MLE baseline;
    • a CRF with feature groups for orthography, n-grams, closed-class words, translations and distances;
    • an Ensemble.
  • Transliteration builds on Micallef et al. (2023): non-deterministic character maps, ranking by a Tunisian word model, and closed-class mappings. This work adds mappings for digits, symbols and some missing letters. Translations come from Google Translate at word level and are stored once as a static table over unique tokens.
  • Writing names in Arabic script hurt NER, largely because Arabic script has no letter case. CAMeLBERT did best on POS, DP and NER with Arabic transliteration plus English translation. The authors attribute this to Latin-script text unintentionally present in its pre-training data.

Interesting ideas

  • I1. Route each token through its own normalization path depending on its predicted source language, instead of applying one transform to the whole sentence.
  • I2. A zero-training baseline can label words purely by edit distance to their translations: an unchanged translation means a name or code-switch, and otherwise the closest language wins.
  • I3. Translating word by word into a fixed type-level table preserves one-to-one token alignment, so existing token-level annotations such as POS tags and UD arcs remain valid. Sentence translation would need noisy aligners.
  • I4. The Mixed label separates a foreign stem from native inflection, for example an Italian verb stem with Arabic prefixes and suffixes. Stem origin and morphology are treated as independent properties.
  • I5. Some results were explained by checking what the representation threw away (case in Arabic script) and what the model had unexpectedly seen (contamination from other scripts).

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): loanwords, Latin phrases and foreign names in English text could be flagged first and then sent to separate lemmatization and tagging paths instead of being forced through English morphology rules.
  • I2 (learn): this kind of cheap, explainable rule could serve as an extra witness or sanity baseline next to trained taggers when building silver data.
  • I3 (tree): when transferring or converting treebanks, a type-level substitution table keeps the gold trees intact and avoids alignment errors.
  • I4 (tokens): the morphology analyzer could record stem origin separately from affix behavior, which helps with productive coinages such as English verbs formed from borrowed nouns.
  • I5 (learn): evaluations could include ablations such as lowercasing input as negative controls, to show how much a component depends on surface cues.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,226 citations🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates124 works · 4,709 citations🎓University of TübingenTübingen · Germany78 works · 41,280 citations🎓Carnegie Mellon University QatarDoha · Qatar48 works · 2,773 citations🎓University of MaltaMsida · Malta32 works · 895 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 3,066 worksMultilingual & cross-lingual NLP · 882 worksLow-resource languages · 287 works

Cite

Cite as Kurt Micallef, Nizar Habash, Claudia Borg, Fadhl Eryani and Houda Bouamor. 2024. Cross-Lingual Transfer from Related Languages: Treating Low-Resource Maltese as Multilingual Code-Switching. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.eacl-long.61

@inproceedings{micallef2024cross,
  title = {{Cross-Lingual Transfer from Related Languages: Treating Low-Resource Maltese as Multilingual Code-Switching}},
  author = {Kurt Micallef, Nizar Habash, Claudia Borg, Fadhl Eryani and Houda Bouamor},
  booktitle = {Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.eacl-long.61},
  url = {https://doi.org/10.18653/v1/2024.eacl-long.61}
}

References

32 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Joseph Aquilina. 1987. Maltese-English Dictionary Vol. I, A-L. Midsea Books, Valletta, Malta. find
  2. Joseph Aquilina. 1990. Maltese-English Dictionary Vol. II, M-Z. Midsea Books, Valletta, Malta. find
  3. Terra Blevins and Luke Zettlemoyer. 2022. Language contamination helps explains the cross-lingual capabilities of English pretrained models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3563–3574, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  4. Joseph M. Brincat. 2017. Maltese: blending Semitic, Romance and Germanic lexemes. Lexicographica, 33(2017):207–224. find
  5. Slavomír Čéplö. 2018. Constituent order in Maltese: A quantitative analysis. Ph.D. thesis, Charles University, Prague. find
  6. Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020. Parsing with multilingual BERT, a small corpus, and a small treebank. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1324–1334, Online. Association for Computational Linguistics.
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  8. Kareem Darwish. 2014. Arabizi detection and conversion to Arabic. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 217–224, Doha, Qatar. Association for Computational Linguistics.
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  10. Ramy Eskander, Mohamed Al-Badrashiny, Nizar Habash, and Owen Rambow. 2014. Foreign words and the automatic processing of Arabic social media text written in Roman script. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 1–12, Doha, Qatar. Association for Computational Linguistics.
  11. Albert Gatt and Slavomír Čéplö. 2013. Digital Corpora and Other Electronic Resources for Maltese. In Proceedings of the International Conference on Corpus Linguistics, pages 96–97. UCREL, Lancaster, UK. find
  12. Lucie Gianola, Ēriks Ajausks, Victoria Arranz, Chomicha Bendahman, Laurent Bié, Claudia Borg, Aleix Cerdà, Khalid Choukri, Montse Cuadros, Ona de Gibert, Hans Degroote, Elena Edelman, Thierry Etchegoyhen, Ángela Franco Torres, Mercedes García Hernandez, Aitor García Pablos, Albert Gatt, Cyril Grouin, Manuel Herranz, Alejandro Adolfo Kohan, Thomas Lavergne, Maite Melero, Patrick Paroubek, Mickaël Rigault, Mike Rosner, Roberts Rozis, Lonneke van der Plas, Rinalds Vı̄ksna, and Pierre Zweigenbaum. 2020. Automatic removal of identifying information in official eu languages for public administrations: The MAPA project. In Proceedings of the 33rd International Conference on Legal Knowledge and Information Systems (JURIX’20), pages 223–226. IOS Press. find
  13. Nizar Habash, Abdelhadi Soudi, and Tim Buckwalter. 2007. On Arabic Transliteration. In A. van den Bosch and A. Soudi, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods, pages 15–22. Springer, Netherlands. find
  14. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  15. Dan Kondratyuk and Milan Straka. 2019. 75 languages, 1 model: Parsing Universal Dependencies universally. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2779–2795, Hong Kong, China. Association for Computational Linguistics. ↗
  16. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, pages 282–289, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  17. Vladimir I. Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics – Doklady, 10:707–710. find
  18. Antonio Martínez-García, Toni Badia, and Jeremy Barnes. 2021. Evaluating morphological typology in zero-shot cross-lingual transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3136–3153, Online. Association for Computational Linguistics. find
  19. Stephen Mayhew, Tatiana Tsygankova, and Dan Roth. 2019. ner and pos when nothing is capitalized. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6256– 6261, Hong Kong, China. Association for Computational Linguistics. find
  20. Kurt Micallef, Fadhl Eryani, Nizar Habash, Houda Bouamor, and Claudia Borg. 2023. Exploring the impact of transliteration on NLP performance: Treating Maltese as an Arabic dialect. In Proceedings of the Workshop on Computation and Written Language (CAWL 2023), pages 22–32, Toronto, Canada. Association for Computational Linguistics.
  21. Kurt Micallef, Albert Gatt, Marc Tanti, Lonneke van der Plas, and Claudia Borg. 2022. Pre-training data quality and quantity for a low-resource language: New corpus and BERT models for Maltese. In Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing, pages 90–101, Hybrid. Association for Computational Linguistics.
  22. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991–16111, Toronto, Canada. Association for Computational Linguistics.
  23. Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 448–462, Online. Association for Computational Linguistics.
  24. Doreen Osmelak and Shuly Wintner. 2023. The denglisch corpus of German-English code-switching. In Proceedings of the 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pages 42–51, Dubrovnik, Croatia. Association for Computational Linguistics. find
  25. Kartikey Pant and Tanvi Dadu. 2020. Towards code-switched classification exploiting constituent language resources. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: Student Research Workshop, pages 37–43, Suzhou, China. Association for Computational Linguistics. find
  26. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673, Online. Association for Computational Linguistics.
  27. Fred Philippy, Siwen Guo, and Shohreh Haddadan. 2023. Identifying the correlation between language distance and cross-lingual transfer in a multilingual representation space. In Proceedings of the 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pages 22–29, Dubrovnik, Croatia. Association for Computational Linguistics. find
  28. Stefan Schweter. 2020. Italian BERT and ELECTRA models. find
  29. Ali Shazal, Aiza Usman, and Nizar Habash. 2020. A unified model for Arabizi detection and transliteration using sequence-to-sequence models. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 167–177, Barcelona, Spain (Online). Association for Computational Linguistics.
  30. Jörg Tiedemann and Lars Nygaard. 2004. The OPUS corpus - parallel and free: http://logos.uio.no/ opus. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), Lisbon, Portugal. European Language Resources Association (ELRA). ↗
  31. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844, Hong Kong, China. Association for Computational Linguistics.
  32. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120–130, Online. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/micallef-2024-cross-lingual-transfer-related-languages/