Elamur Research⌕
☰
home / papers / 2023 / Continual Learning Under Language Shift

Continual Learning Under Language Shift

Paper · 2023 · Evangelia Gogoulou, Timothee Lesort, Magnus Boman et al.

Summary

Retraining large language models from scratch each time new data appears is costly. This paper asks what happens when an existing model is instead updated with text in previously unseen languages. The authors train GPT-style models with 126M, 356M and 1.3B parameters, first on English. They then add Danish, Icelandic and Norwegian one at a time, producing 15 English-first sequences of two to four languages. Each language gets an equal amount of training data from the Nordic Pile, and all models share the GPT-SW3 tokenizer. The authors measure transfer to new languages (forward) and to earlier ones (backward) using test cross-entropy, and compare against monolingual and jointly trained baselines. Gains on newly added languages are consistently positive regardless of order. The effect on earlier languages varies: Icelandic is linked to strong forgetting, while Danish and Norwegian interact more mildly and can even benefit English. Bigger models reach lower loss but show the same transfer patterns. Accidental language contamination combined with syntactic similarity explains the results best, which can guide continual multilingual pre-training.

Key points

  • Training setup: Each continual stage runs 35K steps (roughly 36M sequences of 2048 tokens). Models use the 64K-token BPE GPT-SW3 tokenizer and are implemented in Nvidia NeMo. The joint en+da+is+no baseline is trained for 105K steps.
  • Data sources: The Pile, Wikipedia, mc4, OSCAR, Icelandic Gigaword, Danish Gigaword and NCC.
  • TDS metric: The authors propose token distribution similarity (TDS), the cosine similarity between token-frequency vectors of two corpora. It ranks da-no highest (0.92) and en-is lowest (0.35). However, it fails to predict the weak Danish–Norwegian backward transfer. Among the URIEL features, only syntactic similarity (SYN) fits the observed patterns.
  • Contamination analysis: fastText language identification shows the Icelandic corpus is almost purely Icelandic (99.32%), and other corpora contain negligible Icelandic. The Norwegian and Danish corpora include 3.16% and 1.32% English text, respectively.
  • Effect of scale: Compared with 126M, final average test loss drops by about 8% at 356M and 11% at 1.3B. Forgetting of earlier languages stays largely unchanged.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université de MontréalMontreal · Canada68 works · 79,382 citations🎓Uppsala UniversityUppsala · Sweden64 works · 13,164 citations🔬Mila - Quebec Artificial Intelligence InstituteMontreal · Canada54 works · 13,565 citations🎓KTH Royal Institute of TechnologyStockholm · Sweden18 works · 788 citations🎓Karolinska InstitutetStockholm · Sweden3 works · 134 citations🔬RISE Research Institutes of SwedenGothenburg · Sweden2 works · 33 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Evangelia Gogoulou, Timothee Lesort, Magnus Boman and Joakim Nivre. 2023. Continual Learning Under Language Shift. arXiv:2311.01200.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2311.01200 · arXiv 2311.01200

@misc{gogoulou2023continual,
  title = {{Continual Learning Under Language Shift}},
  author = {Evangelia Gogoulou, Timothee Lesort, Magnus Boman and Joakim Nivre},
  year = {2023},
  doi = {10.48550/arxiv.2311.01200},
  eprint = {2311.01200},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.01200}
}

References

4 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 1. Abnar, S., Dehghani, M., Neyshabur, B., Sedghi, H.: Exploring the limits of large scale pre-training. In: International Conference on Learning Representations (2021) 2. Aljundi, R., Belilovsky, E., Tuytelaars, T., Charlin, L., Caccia, M., Lin, M., Page-Caccia, L.: Online continual learning with m… find
  2. Continual Learning Under Language Shift 8. Coria, J.M., Veron, M., Ghannay, S., Bernard, G., Bredin, H., Galibert, O., Rosset, S.: Analyzing BERT cross-lingual transfer capabilities in continual sequence labeling. In: Proceedings of the First Workshop on Performance and Interpretability Evaluations… find
  3. Wang, J., Zettlemoyer, L., Kozareva, Z., Diab, M., Stoyanov, V., Li, X.: Few-shot learning with multilingual generative language models. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pp. 9019–9052 (2022) 25. Littell, P., Mortensen, D.R., Lin, K., Kairis,… find
  4. OpenAI (2018) 34. Ramasesh, V.V., Lewkowycz, A., Dyer, E.: Effect of scale on catastrophic forgetting in neural networks. In: International Conference on Learning Representations (2022) 35. Scao, T.L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A.S., Yvon, F., G… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gogoulou-2023-continual-learning-under-language-shift/