Elamur Research⌕
☰
home / papers / 2025 / Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian

Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian

Paper · 2025 · Michael Hoffmann, Jophin John, Stefan Schweter et al.

Summary

Most open LLMs are trained mainly on English, which leaves German weaker and dialects such as Bavarian close to unsupported. The authors start from Llama 3.1-8B and grow it to about 10B parameters with block expansion, which inserts new Transformer blocks initialised as identity mappings. They then continue pretraining on 164B tokens. English and German get equal shares, and a small Bavarian portion is added. Bavarian text was gathered from a few trusted sources and widened with a FastText language-ID filter. It was upsampled and introduced only near the end of training. The team also built a shared trilingual tokenizer and, by rendering German benchmarks in Bavarian, produced the first evaluation suite that covers all three languages. The base model ranks second among European models of similar size. After instruction tuning it leads all compared models on Bavarian, beats EuroLLM in English and ties it in German. The whole run used a single Cerebras CS-2 with metered energy use. This shows that a small team can add a low-resource variety to a strong existing model.

Key points

  • The training mix is 82B English tokens, 82B German tokens and only 80M Bavarian tokens. That leaves English at under half of the data.
  • Bavarian seed data came from the Boarische Wikipedia corpus (Wortschatz Uni Leipzig) and four OPUS datasets. A FastText classifier, trained on Boarisch Wikipedia against standard German, was used to find more Bavarian text.
  • The evaluation suite translates the German versions of HellaSwag, ARC, TruthfulQA and MMLU into Bavarian. This allows direct comparison across the three languages.
  • Llama-GENBA-10B-Instruct beats Apertus-8B-Instruct-2509 and gemma-2-9B-it on the Bavarian tasks. The base model sits in the Bavarian top five, but its German results are only mid-tier.
  • Pretraining took 66 days on one CS-2 at about 22.3 kW average draw, for roughly 35.23 MWh in total. Five core contributors built the model, and it is released under a restricted non-commercial license.

Interesting ideas

  • I1. Block expansion adds identity-initialised layers to a frozen base and trains only those new layers, so new language skill is gained with little forgetting of old skills.
  • I2. The scarce data is held back until about 90% of training is done and then upsampled, so the small corpus is not drowned out by bulk data early on.
  • I3. A small trusted seed corpus trains a cheap classifier that tells the rare variety from its standard neighbour, and that classifier then mines more data.
  • I4. Translating an existing benchmark into a closely related variety produces parallel test items, so any score gap reflects the language variety rather than the content.
  • I5. The authors log energy use and wall-clock time alongside accuracy, making cost part of the result.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could gain extra layers trained only on a new domain or UD dialect, such as tales or conversion targets, while the original treebank accuracy stays protected.
  • I2 (learn): a small hand-checked set, such as fable trees or repaired silver data, could be mixed in late and upweighted during parser or tagger training.
  • I3 (learn): a light classifier trained on a few gold examples of a register, such as archaic fable English, could filter web text into candidate silver data before the witness parsers vote.
  • I4 (answer): school tests and fable QA could be rewritten into simplified or dialect-flavoured English to measure how robust the readers are on identical questions.
  • I5 (learn): training and evaluation reports could record compute per stage so that LLM-teacher tiers and parser retraining can be weighed against their gains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 works

Cite

Cite as Michael Hoffmann, Jophin John, Stefan Schweter, Gokul Ramakrishnan, Hoi-Fong Mak, Alice Zhang, Dmitry Gaynullin and Nicolay J. Hammer. 2025. Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian. arXiv:2509.05668.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2509.05668 · arXiv 2509.05668

@misc{hoffmann2025llama,
  title = {{Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian}},
  author = {Michael Hoffmann, Jophin John, Stefan Schweter, Gokul Ramakrishnan, Hoi-Fong Mak, Alice Zhang, Dmitry Gaynullin and Nicolay J. Hammer},
  year = {2025},
  doi = {10.48550/arxiv.2509.05668},
  eprint = {2509.05668},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2509.05668}
}

References

42 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. ↗
  2. Mehdi Ali, Michael Fromm, Klaudia Thellmann, Jan Ebert, Alexander Arno Weber, Richard Rutmann, Charvi Jain, Max Lübbering, Daniel Steinigen, Johannes Leveling, et al. 2024a. Teuken-7b-base & teuken-7binstruct: Towards european llms. arXiv preprint arXiv:2410.03730. ↗
  3. Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, et al. 2024b. Tokenizer choice for llm training: Negligible or crucial? In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3907– 3924. ↗
  4. Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. 2023. Llemma: An open language model for mathematics. ↗
  5. Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. 2024. Cosmopedia. find
  6. Verena Blaschke, Barbara Kovačić, Siyao Peng, Hinrich Schütze, and Barbara Plank. 2024. Maibaam: A multi-dialectal bavarian universal dependency treebank. arXiv preprint arXiv:2403.10293. ↗
  7. Verena Blaschke, Miriam Winkler, Constantin Förster, Gabriele Wenger-Glemser, and Barbara Plank. 2025. A multi-dialectal dataset for german dialect asr and dialect-to-standard speech translation. arXiv preprint arXiv:2506.02894. ↗
  8. Branden Chan, Stefan Schweter, and Timo Möller. 2020. German’s next language model. arXiv preprint arXiv:2010.10906. ↗
  9. Monojit Choudhury, Shivam Chauhan, Rocktim Jyoti Das, Dhruv Sahnan, Xudong Han, Haonan Li, Aaryamonvikram Singh, Alok Anil Jadhav, Utkarsh Agarwal, Mukund Choudhary, et al. 2025. Llama-3-nanda-10b-chat: An open generative large language model for hindi. arXiv preprint arXiv:2504.06011. ↗
  10. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. ↗
  11. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. ↗
  12. Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen. 2023. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. arXiv e-prints, pages arXiv–2307.
  13. Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2021. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366–3385. find
  14. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  15. Eurostat. 2024. Electricity and heat statistics. Scientific Reports. find
  16. Zhaoye Fei, Yunfan Shao, Linyang Li, Zhiyuan Zeng, Conghui He, Hang Yan, Dahua Lin, and Xipeng Qiu. 2024. Query of cc: unearthing large scale domain-specific knowledge from public corpora. arXiv preprint arXiv:2401.14624. ↗
  17. Pablo Gamallo, Pablo Rodríguez, Daniel Santos, Susana Sotelo, Nuno Miquelina, Silvia Paniagua, Daniela Schmidt, Iria de Dios-Flores, Paulo Quaresma, Daniel Bardanca, et al. 2024. A galician-portuguese generative model. In EPIA Conference on Artificial Intelligence, pages 292–304. Springer. find
  18. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. The language model evaluation harness. find
  19. Gurpreet Gosal, Yishi Xu, Gokul Ramakrishnan, Rituraj Joshi, Avraham Sheinin, Biswajit Mishra, Natalia Vassilieva, Joel Hestness, Neha Sengupta, Sunil Kumar Sahu, et al. 2024. Bilingual adaptation of monolingual foundation models. arXiv preprint arXiv:2407.12869. ↗
  20. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. ↗
  21. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. ↗
  22. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759. ↗
  23. Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schütze. 2023. Glotlid: Language identification for low-resource languages. arXiv preprint arXiv:2310.16248. ↗
  24. Wen Lai, Mohsen Mesgar, and Alexander Fraser. 2024. Llms beyond english: Scaling the multilingual capability of llms with cross-lingual feedback. arXiv preprint arXiv:2406.01771. ↗
  25. Sean Lie. 2024. Inside the cerebras wafer-scale cluster. IEEE Micro, 44(3):49–57. ↗
  26. Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958. ↗
  27. Pedro Henrique Martins, João Alves, Patrick Fernandes, Nuno M Guerreiro, Ricardo Rei, Amin Farajian, Mateusz Klimaszewski, Duarte M Alves, José Pombal, Manuel Faysse, et al. 2025. Eurollm-9b: Technical report. arXiv preprint arXiv:2506.04079. ↗
  28. Jonas Pfeiffer, Naman Goyal, Xi Victoria Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre-training modular transformers. arXiv preprint arXiv:2205.06266. ↗
  29. Jan Pfister, Julia Wunderle, and Andreas Hotho. 2024. Ll\" ammlein: Compact and competitive german-only language models from scratch. arXiv preprint arXiv:2411.11171. ↗
  30. Parker Riley, Isaac Caswell, Markus Freitag, and David Grangier. 2019. Translationese as a language in" multilingual" nmt. arXiv preprint arXiv:1911.03823. ↗
  31. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99–106. ↗
  32. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, et al. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. ↗
  33. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. ↗
  34. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. ↗
  35. Ahmet Üstün, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction finetuned open-access multilingual language model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15894–15939, Bangkok, Thailand. Association for Computational Linguistics. ↗
  36. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  37. Zirui Wang, Zachary C Lipton, and Yulia Tsvetkov. 2020. On negative interference in multilingual models: Findings and a meta-learning treatment. arXiv preprint arXiv:2010.03017. ↗
  38. Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, et al. 2023. Polylm: An open source polyglot large language model. arXiv preprint arXiv:2307.06018. ↗
  39. BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. ↗
  40. Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ying Shan, and Ping Luo. 2024. Llama pro: Progressive llama with block expansion. arXiv preprint arXiv:2401.02415. ↗
  41. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. ↗
  42. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hoffmann-2025-llama-genba-10b-trilingual-large/