Elamur Research⌕
☰
home / papers / 2026 / Transferring Scientific English Pre-Trained Language Models to Multiple Languages Using Cross-Lingual Transfer

Transferring Scientific English Pre-Trained Language Models to Multiple Languages Using Cross-Lingual Transfer

Paper · 2026 · Nikolas Rauscher, Fabio Barth and Georg Rehm

Summary

Scientific writing is overwhelmingly in English, so even large languages such as German or Japanese have little scientific training text. As a result, scientific language models have remained largely English-only. The authors test whether domain knowledge learned in English can be moved into other languages rather than training multilingual scientific models from scratch.

They clean the English part of the SciLaD corpus (open-access papers converted to text with GROBID) using a DataTrove pipeline. They then continue pre-training a T5-base checkpoint with span corruption, producing EN-T5-Sci. Next, they apply WECHSEL: the Transformer weights are kept, a new SentencePiece tokenizer is trained per target language on SciLaD data, and its embeddings are initialized from English ones through aligned fastText vectors. The outcome is a set of six scientific T5 models, one each in Spanish, Portuguese, Polish, Russian, Japanese and German.

In zero-shot Global-MMLU tests, EN-T5-Sci beats its base model, especially in STEM and Social Sciences, and is on par with the far larger BLOOM on science categories. The transferred models mostly outperform monolingual general-domain baselines. The released pipeline offers a low-cost route that could also be reused for other languages, such as Ukrainian.

Key points

  • SciLaD language skew: English accounts for about 73.95% of the corpus (~11M documents). Japanese has 4.1%, Portuguese 3.5%, German 1.99%, and Polish under 1%.
  • Cleaning results: The DataTrove pipeline stripped roughly 184M citation artifacts while keeping 99.9% of documents. Mean document length fell by 22.0%, perplexity by 5.4%, and paragraph duplicates by 42.3%.
  • EN-T5-Sci training:
    • Model: T5-base (220M parameters) with 15% span masking and a mean span length of 3.
    • Data: 512-token windows with 50% overlap, giving about 261M training examples.
    • Compute: 487,500 steps on 4× H100 GPUs with a batch size of 384.
  • WECHSEL transfer setup: Each target language uses a 5 GB SciLaD sample. Tokenizers are SentencePiece BPE with 32k subwords (32,100 tokens including T5 extra IDs). Embeddings are initialized with k=10 nearest neighbours and temperature τ=0.1.
  • Availability: The pipeline and all models are public on GitHub (nikolas-rauscher/scientific-english-crosslingual-transfer) and Hugging Face. The authors note the lack of non-English scientific benchmarks as the main limitation, since Global-MMLU is machine-translated.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬German Research Centre for Artificial IntelligenceKaiserslautern · Germany79 works · 29,970 citations🎓Humboldt-Universität zu BerlinBerlin · Germany20 works · 41,895 citations🎓Technische Universität BerlinBerlin · Germany22 works · 957 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEnglish · 2,379 worksMultilingual & cross-lingual NLP · 882 works

Cite

Cite as Nikolas Rauscher, Fabio Barth and Georg Rehm. 2026. Transferring Scientific English Pre-Trained Language Models to Multiple Languages Using Cross-Lingual Transfer. Proceedings of Natural Scientific Language Processing (NSLP) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/43tu6dfduwvw

@inproceedings{rauscher2026transferring,
  title = {{Transferring Scientific English Pre-Trained Language Models to Multiple Languages Using Cross-Lingual Transfer}},
  author = {Nikolas Rauscher, Fabio Barth and Georg Rehm},
  booktitle = {Proceedings of Natural Scientific Language Processing (NSLP) @ LREC 2026},
  year = {2026},
  doi = {10.63317/43tu6dfduwvw},
  url = {https://doi.org/10.63317/43tu6dfduwvw}
}

References

24 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdulhady Abas Abdullah, Arkaitz Zubiaga, Seyedali Mirjalili, Amir H. Gandomi, Fatemeh Daneshfar, Mohammadsadra Amini, Alan Salam Mohammed, and Hadi Veisi. 2025. Evolution of meta’s llama models and parameter-efficient fine-tuning of large language models: a survey.
  2. Mehdi Ali, Michael Fromm, Klaudia Thellmann, et al. 2025. Teuken-7b-base & teuken-7b-instruct: Towards european llms. find
  3. Ramesh K. Bhukya, Anjali Chaturvedi, Hardik Bajaj, Udgam Shah, Sumit Singh, and Uma Shanker Tiwary. 2023. Efficiently transferring pre-trained language model roberta base english to hindi using wechsel. In 2023 26th Conference of the Oriental COCOSDA International Committee for the Co-ordination and… find
  4. Luca Foppiano, Sotaro Takeshita, Pedro Ortiz Suarez, Ekaterina Borisova, Raia Abu Ahmad, Malte Ostendorff, Fabio Barth, Julian Moreno-Schneider, and Georg Rehm. 2025. Scilad: A large-scale, transparent, reproducible dataset for natural scientific language processing. find
  5. Jason Alan Fries, Leon Weber, Natasha Seelam, et al. 2022. Bigbio: A framework for data-centric biomedical natural language processing. find
  6. Grobid. 2008–2026. Grobid. https://github.com/kermitt2/grobid. ↗
  7. Naimul Haque. 2025. Catastrophic forgetting in llms: A comparative analysis across language tasks. find
  8. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding.
  9. Ming Hu, Chenglong Ma, Wei Li, et al. 2025. A survey of scientific large language models: From data foundations to agent frontiers. find
  10. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, et al. 2024. Mixtral of experts. find
  11. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering.
  12. Jungseob Lee, Seongtae Hong, Hyeonseok Moon, and Heuiseok Lim. 2025. Cross-lingual optimization for language transfer in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15100–15119, Vienna, Austria. Asso… find
  13. Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969–4983, Online. Association for Computational Linguistics.
  14. Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. 2022. WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hum…
  15. Malte Ostendorff and Georg Rehm. 2023. Efficient language model training through cross-lingual and progressive transfer learning.
  16. Long N. Phan, James T. Anibal, Hieu Tran, Shaurya Chanana, Erol Bahadroglu, Alec Peltekian, and Grégoire Altan-Bonnet. 2021. Scifive: a text-to-text transformer model for biomedical literature. find
  17. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. Exploring the limits of transfer learning with a unified text-to-text transformer. find
  18. François Remy, Pieter Delobelle, Hayastan Avetisyan, Alfiya Khabibullina, Miryam de Lhoneux, and Thomas Demeester. 2024. Trans-tokenization and cross-lingual vocabulary transfers: Language adaptation of llms for low-resource nlp. find
  19. Tarek Saier, Johan Krause, and Michael Färber. 2023. unarxive 2022: All arxiv publications preprocessed for nlp, including structured full-text and citation network. In 2023 ACM/IEEE Joint Conference on Digital Libraries (JCDL), page 66–70. IEEE. find
  20. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. find
  21. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, et al. 2025. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation.
  22. C Tardy. 2004. The role of english in scientific communication: lingua franca or tyrannosaurus rex? Journal of English for Academic Purposes, 3(3):247–269. find
  23. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science.
  24. BigScience Workshop, :, Teven Le Scao, Angela Fan, Christopher Akiki, et al. 2023. Bloom: A 176b-parameter open-access multilingual language model. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rauscher-2026-transferring-scientific-english-pre-trained/