Elamur Research⌕
☰
home / papers / 2024 / Bilingual Adaptation of Monolingual Foundation Models

Bilingual Adaptation of Monolingual Foundation Models

Summary

English-dominant open LLMs such as Llama 2 handle other languages poorly. Training a bilingual model from scratch is costly and tends to give weaker English. Adapting an existing model brings its own problems: it can forget English, and its English-centric tokenizer splits foreign text into many pieces. The authors offer a step-by-step recipe, tested mainly on adapting Llama 2 to Arabic.

The recipe has four parts:

  • Grow the vocabulary with frequent Arabic tokens.
  • Initialize the new embeddings so they sit in a sensible place in the existing space.
  • Train only the embedding layers.
  • Continue pre-training the whole model on a mix of Arabic (AraV5) and English (the Pile) text.

They also run ablations on initialization methods, data mix ratios and learning rates. The adapted 7B, 13B and 70B models gain clearly in Arabic, and English stays roughly level. The 70B model slightly passes Jais 30b on the Arabic average. Llama 3 8B (Arabic) and Llama 2 13B (Hindi) runs suggest the recipe carries over to other models and languages. The work matters because it shows a reusable, comparatively cheap way to build bilingual models without starting over.

Key points

  • Tokenizer cost and vocabulary extension. On the same Arabic text, the base Llama 2 tokenizer needs up to about 4× as many tokens as the MLV2 bilingual tokenizer from Jais. Llama 2-extend100 adds 32000 Arabic tokens (vocabulary grows from 32000 to 64000). This cuts Arabic fertility by 72.17%, keeps English fertility unchanged, and beats the base tokenizer by ≈2% in validation loss when the number of Arabic tokens is held fixed.
  • Embedding initialization. The best method is Embedding Space Transformation: a linear map (W, b) is fitted to project Jais-30B embeddings into the Llama 2 embedding space, and new tokens are placed with it. This beat similarity-based averaging, which uses the top-5 neighbours found with text-embedding-3-large. Xavier, Kaiming and mean-vector initialization did poorly.
  • Embedding-only stage. The model is trained on 15B tokens (AraV5:Pile at 9:1) with everything frozen except the embedding and unembedding layers. This lowered Arabic loss by up to 2%.
  • Hyperparameters. For Llama 2, re-warming to the original peak learning rate of 3e-4 worked best, with a 4M-token batch. Llama 3 8B needed a lower learning rate (7.5e-5), a 6M-token batch, and only a 20% vocabulary extension.
  • Results.
    • Llama 2 70B adapted: Arabic overall average rises from 41.8% to 49.2% (Jais 30b: 49.1%); English goes from 60.2% to 60.9%.
    • Llama 3 8B: Arabic rises from 43.2% to 47.9%.
    • Llama 2 13B Hindi: XNLI-Hindi rises from 37.44% to 45.68%, while English average drops from 57.1% to 55.3%.

Interesting ideas

  • I1. When comparing two tokenizers, hold the amount of target-language training tokens fixed ("isotoken"); otherwise the tokenizer that splits more finely looks better only because the model saw more tokens. — For Mova (learn): Any comparison of parsers or taggers trained on differently segmented silver data should equalize training volume, so the evaluation reflects real quality and not data quantity.
  • I2. Fertility (tokens per word) is a cheap intrinsic measure that exposes how badly a segmenter fragments one kind of text compared with another, before any expensive training. — For Mova (tokens): Track fertility-like split rates of the tokenizer per genre (fables, maths problems, school texts) to catch domains where segmentation quietly degrades.
  • I3. Mix "replay" data from the original domain into continued training to keep old skills while learning new ones; here, adding more English even helped the new language. — For Mova (learn): When retraining the perceptron parser or gparse on new silver data, keep a share of the original gold treebank in each batch to guard against forgetting, and check it with old-domain test sets.
  • I4. New vocabulary items can be given starting vectors by fitting a simple linear map from another model's embedding space over the items both share, instead of random initialization. — For Mova (tree): When gparse gains new word pieces or a new embedding source, a fitted linear projection on the shared vocabulary could seed the new rows so training starts aligned.
  • I5. First train only the newly added embedding layers with the rest frozen, then unfreeze everything, so new parameters adjust before they can disturb what the network already knows. — For Mova (tree): A frozen-body warm-up stage could make domain or dialect adaptation of gparse safer, protecting accuracy (e.g. LAS) on the original treebanks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Language models & pretraining

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2407.12869 · arXiv 2407.12869

@misc{gosal2024bilingual,
  title = {{Bilingual Adaptation of Monolingual Foundation Models}},
  author = {Gosal, Gurpreet and Xu, Yishi and Ramakrishnan, Gokul and Joshi, Rituraj and Sheinin, Avraham and Chen, Zhiming and Mishra, Biswajit and Vassilieva, Natalia and Hestness, Joel and Sengupta, Neha and Sahu, Sunil Kumar and Jia, Bokang and Pandit, Onkar and Katipomu, Satheesh and Kamboj, Samta and Ghosh, Samujjwal and Pal, Rahul and Mullah, Parvez and Doraiswamy, Soundar and Chami, Mohamed El Karim and Nakov, Preslav},
  year = {2024},
  doi = {10.48550/arxiv.2407.12869},
  eprint = {2407.12869},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.12869}
}

References

21 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahuja, K., Diddee, H., Hada, R., Ochieng, M., Ramesh, K., Jain, P., Nambi, A., Ganu, T., Segal, S., Ahmed, M., Bali, K., and Sitaram, S. MEGA: Multilingual evaluation of generative AI. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the Csaki, Z., Pawakapan, P., Thakker, U., and Xu, Q. Effi-2023 Conference on Empirical Methods in Natural ciently adapting pretrained language models to new lan-Language Processing, pp. 4232–4267, Singapore, guages, 2023. December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.258. URL Da Dalt, S., Llop, J., Baucells, I., Pamies, M., Xu, Y., https://aclanthology.org/2023.emnlp-main.258. Gonzalez-Agirre, A., and Villegas, M. FLOR: On the ef-Ali, M., Fromm, M., Thellmann, K., Rutmann, R., Lübbering, M., Leveling, J., Klug, K., Ebert, J., Doll, N., Buschhoff, J. S., Jain, C., Weber, A. A., Jurkschat, L., Abdelwahab, H., John, C., Suarez, P. O., Ostendorff, M., Weinbach, S., Sifa, R., Kesselheim, S., and Flores-Herr, N. Tokenizer choice for llm training: Negligible or crucial?, 2024. fectiveness of language adaptation. In Calzolari, N., Kan, M.-Y., Hoste, V., Lenci, A., Sakti, S., and Xue, N. (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 7377– 7388, Torino, Italia, May 2024. ELRA and ICCL. URL https://aclanthology.org/2024.lrec-main.650. ↗ our card
  2. Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S., Jiang, A. Q., Deng, J., Biderman, S., and Welleck, S. Llemma: An open language model for mathematics, 2024. de Vries, W. and Nissim, M. As good as new. how to successfully recycle English GPT-2 to make models for other languages. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 836–846, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.findings-acl.74. URL https://aclanthology.org/2021.findings-acl.74. ↗ our card
  3. Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020. find our card
  4. Dey, N., Gosal, G., Zhiming, Chen, Khachane, H., Marshall, W., Pathria, R., Tom, M., and Hestness, J. Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster, 2023. find
  5. Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., Sallinen, A., Sakhaeirad, A., Swamy, V., Krawczuk, I., Bayazit, D., Marmet, A., Montariol, S., Hartley, M.-A., Jaggi, M., and Bosselut, A. Meditron-70b: Scaling medical pretraining for large language models, 2023. et. al., T. L. S. Bloom: A 176b-parameter open-access multilingual language model, 2023. find our card
  6. French, R. M. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):128–135, 1999. ISSN 1364-6613. doi: https://doi.org/10.1016/S1364-6613(99)01294-2. URL https://www.sciencedirect.com/science/article/pii/ ↗
  7. Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2924–2936, Minneapolis, Minnesota, June 2019. Association for Computa-Fujii, K., Nakamura, T., Loem, M., Iida, H., Ohi, M., Hattori, K., Shota, H., Mizuki, S., Yokota, R., and Okazaki, N. Continual pre-training for cross-lingual llm adaptation: Enhancing japanese language capabilities, 2024. find our card
  8. A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L. Training compute-optimal large language models, 2022. find
  9. Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., Xin, R., Nezhurina, M., Vasiljevic, I., Jitsev, J., Dimakis, A. G., Ilharco, G., Song, S., Kollar, T., Carmon, Y., Dave, A., Heckel, R., Muennighoff, N., and Schmidt, L. Language models scale reliably with over-training and on downstream tasks, 2024. find
  10. Huang, H., Yu, F., Zhu, J., Sun, X., Cheng, H., Song, D., Chen, Z., Alharthi, A., An, B., He, J., Liu, Z., Zhang, Z., Chen, J., Li, J., Wang, B., Zhang, L., Sun, R., Wan, X., Li, H., and Xu, J. Acegpt, localizing large language models in arabic, 2024. find our card
  11. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The pile: An 800gb dataset of diverse text for language modeling, 2020. find
  12. Isik, B., Ponomareva, N., Hazimeh, H., Paparas, D., Vassilvitskii, S., and Koyejo, S. Scaling laws for downstream task performance of large language models, 2024. find
  13. Glorot, X. and Bengio, Y. Understanding the difficulty Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., of training deep feedforward neural networks. In Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, Teh, Y. W. and Titterington, M. (eds.), Proceedings of G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-the Thirteenth International Conference on Artificial A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, Intelligence and Statistics, volume 9 of Proceedings of T., and Sayed, W. E. Mistral 7b, 2023. Machine Learning Research, pp. 249–256, Chia Laguna Resort, Sardinia, Italy, 13–15 May 2010. PMLR. URL Kim, D., Park, C., Kim, S., Lee, W., Song, W., Kim, Y., https://proceedings.mlr.press/v9/glorot10a.html. Kim, H., Kim, Y., Lee, H., Kim, J., Ahn, C., Yang, S., Lee, S., Park, H., Gim, G., Cha, M., Lee, H., and Guo, Y., Fu, J., Zhang, H., Zhao, D., and Kim, S. Solar 10.7b: Scaling large language models Shen, Y. Efficient continual pre-training with simple yet effective depth up-scaling, 2024. URL by mitigating the stability gap, 2024. URL https://arxiv.org/abs/2312.15166. https://arxiv.org/abs/2406.14833. ↗
  14. Gupta, K., Thérien, B., Ibrahim, A., Richter, M. L., Anthony, Q., Belilovsky, E., Rish, I., and Lesort, T. Continual pre-training of large language models: How to (re)warm your model?, 2023. find
  15. Kusupati, A., Bhatt, G., Rege, A., Wallingford, M., Sinha, A., Ramanujan, V., Howard-Snyder, W., Chen, K., Kakade, S., Jain, P., and Farhadi, A. Matryoshka representation learning, 2024. find
  16. Hardalov, M., Mihaylov, T., Zlatkova, D., Dinkov, Y., Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, Koychev, I., and Nakov, P. EXAMS: A multi-subject E. RACE: Large-scale ReAding comprehension high school examinations dataset for cross-lingual dataset from examinations. In Palmer, M., Hwa, and multilingual question answering. In Webber, B., R., and Riedel, S. (eds.), Proceedings of the 2017 Cohn, T., He, Y., and Liu, Y. (eds.), Proceedings of Conference on Empirical Methods in Natural Lan-the 2020 Conference on Empirical Methods in Natural guage Processing, pp. 785–794, Copenhagen, Den-Language Processing (EMNLP), pp. 5427–5444, Online, mark, September 2017. Association for Computational November 2020. Association for Computational Lin-Linguistics. doi: 10.18653/v1/D17-1082. URL guistics. doi: 10.18653/v1/2020.emnlp-main.438. URL https://aclanthology.org/D17-1082. https://aclanthology.org/2020.emnlp-main.438. Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into how models mimic human falsehoods, 2021. rectifiers: Surpassing human-level performance on ima-Loshchilov, I. and Hutter, F. Sgdr: stochastic gra-genet classification. In 2015 IEEE International Confer-dient descent with warm restarts, 2017. URL ence on Computer Vision (ICCV), pp. 1026–1034, 2015. https://doi.org/10.48550/arXiv.1608.03983. doi: 10.1109/ICCV.2015.123. ↗ our card
  17. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding, 2021. find our card
  18. Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., and Zhang, Y. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, 2024. find
  19. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, Marchisio, K., Lewis, P., Chen, Y., and Artetxe, M. Mini-model adaptation: Efficiently extending pretrained models to new languages via aligned shallow training. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Findings of the Association for Computational Lin-P., Baldwin, T., and Xing, E. Jais and jais-chat: Arabic-guistics: ACL 2023, pp. 5474–5490, Toronto, Canada, centric foundation and instruction-tuned open generative July 2023. Association for Computational Linguis-large language models, 2023a. tics. doi: 10.18653/v1/2023.findings-acl.338. URL Sengupta, N., Sahu, S. K., Jia, B., Katipomu, S., Li, H., https://aclanthology.org/2023.findings-acl.338. Koto, F., Marshall, W., Gosal, G., Liu, C., Chen, Z., Afzal, O. M., Kamboj, S., Pandit, O., Pal, R., Prad-Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a han, L., Mujahid, Z. M., Baali, M., Han, X., Bsharat, suit of armor conduct electricity? a new dataset for open S. M., Aji, A. F., Shen, Z., Liu, Z., Vassilieva, N., Hes-book question answering. In EMNLP, 2018. tness, J., Hock, A., Feldman, A., Lee, J., Jackson, A., Minixhofer, B., Paischer, F., and Rekabsaz, N. WECH-Ren, H. X., Nakov, P., Baldwin, T., and Xing, E. Jais SEL: Effective initialization of subword embeddings and jais-chat: Arabic-centric foundation and instruction-for cross-lingual transfer of monolingual language tuned open generative large language models, 2023b. models. In Carpuat, M., de Marneffe, M.-C., and Sennrich, R., Haddow, B., and Birch, A. Neural machine Meza Ruiz, I. V. (eds.), Proceedings of the 2022 translation of rare words with subword units. In Erk, Conference of the North American Chapter of the K. and Smith, N. A. (eds.), Proceedings of the 54th Association for Computational Linguistics: Human Annual Meeting of the Association for Computational Language Technologies, pp. 3992–4006, Seattle, United Linguistics (Volume 1: Long Papers), pp. 1715–1725, States, July 2022. Association for Computational Lin-Berlin, Germany, August 2016. Association for Compu-guistics. doi: 10.18653/v1/2022.naacl-main.293. URL tational Linguistics. doi: 10.18653/v1/P16-1162. URL https://aclanthology.org/2022.naacl-main.293. https://aclanthology.org/P16-1162. Nangia, N., Vania, C., Bhalerao, R., and Bowman, S. R. Tay, Y., Dehghani, M., Rao, J., Fedus, W., Abnar, S., CrowS-pairs: A challenge dataset for measuring social Chung, H. W., Narang, S., Yogatama, D., Vaswani, A., biases in masked language models. In Webber, B., and Metzler, D. Scale efficiently: Insights from pre-Cohn, T., He, Y., and Liu, Y. (eds.), Proceedings of training and fine-tuning transformers, 2022. the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1953–1967, Online, Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, November 2020. Association for Computational Lin-A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., guistics. doi: 10.18653/v1/2020.emnlp-main.154. URL Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, https://aclanthology.org/2020.emnlp-main.154. M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Petrov, A., Malfa, E. L., Torr, P. H. S., and Bibi, A. Lan-Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, guage model tokenizers introduce unfairness between M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, languages, 2023. A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizen-Y. WINOGRANDE: an adversarial winograd schema stein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., challenge at scale. CoRR, abs/1907.10641, 2019. URL Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., http://arxiv.org/abs/1907.10641. Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Sap, M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Y. Social IQa: Commonsense reasoning about social in-Llama 2: Open foundation and fine-tuned chat models, teractions. In Inui, K., Jiang, J., Ng, V., and Wan, X. 2023. (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the Wu, C., Gan, Y., Ge, Y., Lu, Z., Wang, J., Feng, Y., Shan, 9th International Joint Conference on Natural Language Y., and Luo, P. Llama pro: Progressive llama with block Processing (EMNLP-IJCNLP), pp. 4463–4473, Hong expansion, 2024. ↗
  20. Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1454. URL Yang, T., Li, C., Zhu, Y., Sun, Y., Huang, F., and Lin, https://aclanthology.org/D19-1454. C. Large language models: A survey. arXiv preprint arXiv:2402.06196, 2024. Sengupta, N., Sahu, S. K., Jia, B., Katipomu, S., Li, H., Koto, F., Afzal, O. M., Kamboj, S., Pandit, O., Pal, R., Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Pradhan, L., Mujahid, Z. M., Baali, M., Aji, A. F., Liu, Y. HellaSwag: Can a machine really finish your sen-Z., Hock, A., Feldman, A., Lee, J., Jackson, A., Nakov, tence? In Korhonen, A., Traum, D., and Màrquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791– 4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1472. URL https://aclanthology.org/P19-1472. cost of high fertility tokenizers as more tokens for the given text mean limited context for the model, more memory requirements. ↗ our card
  21. Zhao, J., Zhang, Z., Gao, L., Zhang, Q., Gui, T., and Huang, X. Llama beyond english: An empirical study on language capability transfer, 2024. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gosal-2024-bilingual-adaptation-monolingual-foundation-models/