Elamur Research⌕
☰
home / papers / 2026 / ELO: Efficient Layer-Specific Optimization for Continual Pretraining of Multilingual LLMs

ELO: Efficient Layer-Specific Optimization for Continual Pretraining of Multilingual LLMs

Paper · 2026 · Hangyeol Yoo, ChangSu Choi, Minjun Kim et al.

Summary

Continual pretraining (CP) of multilingual LLMs on a new language is costly and often weakens English ability. LoRA and layer-selective tuning reduce the number of trainable parameters, but every forward pass still runs through the full network, so they save little time. The authors propose ELO. It extracts only the first and last decoder layers, along with the token embedding and LM head, into a small standalone model. Only those two layers are trained, on 9 GB of target-language text mixed with English at a 1:9 ratio. The updated layers are then inserted back into the original model, and a brief full fine-tuning pass on about 1 GB of data re-aligns the parameters. Instruction-following is restored through a chat vector followed by SFT. The method was tested on Korean and Japanese with Llama 3.1-8B, Mistral-7B-v0.3 and Qwen2-7B. ELO trains up to 6.46× faster than FFT, scores up to 6.2% higher on qualitative benchmarks, and keeps English scores close to the base model. This makes language adaptation practical for teams with limited compute.

Key points

  • Benchmarks: English used MMLU and MT-Bench. Korean used KoBEST and LogicKor. Japanese used MARC-ja and MT-Bench(ja). For the qualitative tests, GPT-4 served as the evaluator, assigning ratings out of 10.
  • Llama 3.1, Korean: ELO needed 3.5 h of total training versus 19.8 h for FFT. Its LogicKor score was 7.76, compared with 7.31 for FFT and 6.03 for the base Instruct model. MMLU dropped only slightly, from 68.07 to 66.69.
  • Layer alignment is essential: without it, LogicKor fell to 4.5. With 1 GB of alignment data it reached 7.76, and the peak was 7.78 at 1.5 GB.
  • Speedup grows with data size: with 50 GB, ELO was 5.29× faster than LoRA and 6.04× faster than FFT. With 200 GB, the gaps widened to 10.72× and 12.23×.
  • Limitations: the alignment step is full fine-tuning, so peak GPU memory is not reduced. Results have not been verified on corpora larger than 1 TB. SFT used 31K bilingual (1:1) ShareGPT-style instructions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Korea Advanced Institute of Science and TechnologyDaejeon · South Korea56 works · 3,269 citations🎓Seoul National University of Science and TechnologySeoul · South Korea2 works

Also on the byline: Hanbat National University · LG (South Korea).

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksMultilingual & cross-lingual NLP · 882 works

Cite

Cite as Hangyeol Yoo, ChangSu Choi, Minjun Kim, Seohyun Song, SeungWoo Song, Inho Won, Jongyoul Park, Cheoneum Park and KyungTae Lim. 2026. ELO: Efficient Layer-Specific Optimization for Continual Pretraining of Multilingual LLMs. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-industry.55

@inproceedings{yoo2026elo,
  title = {{ELO: Efficient Layer-Specific Optimization for Continual Pretraining of Multilingual LLMs}},
  author = {Hangyeol Yoo, ChangSu Choi, Minjun Kim, Seohyun Song, SeungWoo Song, Inho Won, Jongyoul Park, Cheoneum Park and KyungTae Lim},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.55},
  url = {https://doi.org/10.18653/v1/2026.eacl-industry.55}
}

References

25 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Guangji Bai, Zheng Chai, Chen Ling, Shiyu Wang, Jiaying Lu, Nan Zhang, Tingwei Shi, Ziyang Yu, Meng-Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, and 1 others. 2023. Vicuna: An open-source chatbot impressing g… find
  2. ChangSu Choi, Yongbin Jeong, Seoyoon Park, Inho Won, HyeonSeok Lim, SangMin Kim, Yejee Kang, Chanhyuk Yoon, Jaewan Park, Yiseul Lee, HyeJin Lee, Younggyun Hahm, Hansaem Kim, and Kyung-Tae Lim. 2024. Optimizing language augmentation for multilingual large language models: A case study on Korean. In P… find
  3. Yiming Cui, Ziqing Yang, and Xin Yao. 2024. Efficient and effective text encoding for chinese llama and alpaca. Preprint, arXiv:2304.08177.
  4. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36.
  5. Peter Devine. 2024. Tagengo: A multilingual chat dataset. arXiv preprint arXiv:2405.12612. find
  6. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  7. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, and 1 others. 2021. A framework for few-shot language model evaluation. Version v0. 0.1. Sept, page 8. find
  8. Soufiane Hayou, Nikhil Ghosh, and Bin Yu. 2024. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354.
  9. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  10. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  11. Shih-Cheng Huang, Pin-Zu Li, Yu-Chi Hsu, Kuang-Ming Chen, Yu Tung Lin, Shih-Kai Hsiao, Richard Tzong-Han Tsai, and Hung yi Lee. 2024. Chat vector: A simple approach to equip llms with instruction following and model alignment in new languages. Preprint, arXiv:2310.04799. find
  12. Myeongjun Jang, Dohyung Kim, Deuk Sin Kwon, and Eric Davis. 2022. KoBEST: Korean balanced evaluation of significant tasks. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3697–3708, Gyeongju, Republic of Korea. International Committee on Computational Linguist… find
  13. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, and 1 others. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. find
  14. Seungduk Kim, Seungtaek Choi, and Myeongho Jeong. 2024. Efficient and effective vocabulary expansion towards multilingual large language models. arXiv preprint arXiv:2402.14714.
  15. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, and 3 others. 2023. A survey of large language models. Prepri…
  16. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 3…
  17. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. Lima: Less is more for alignment. Preprint, arXiv:2305.11206. find
  18. Kentaro Kurihara, Daisuke Kawahara, and Tomohide Shibata. 2022. Jglue: Japanese general language understanding evaluation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2957–2966.
  19. Vedang Lad, Wes Gurnee, and Max Tegmark. 2024. The remarkable robustness of llms: Stages of inference? Preprint, arXiv:2406.19384. — [arXiv:2406.19384](https://arxiv.org/abs/2406.19384) find
  20. Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. 2023. Relora: High-rank training through low-rank updates. In Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ NeurIPS 2023).
  21. Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. 2024. A comprehensive overview of large language models. Preprint, arXiv:2307.06435.
  22. OpenAI. 2023. Gpt-4 technical report. arXiv:2303.08774.
  23. Preprint, Jeonghwan Park. 2024. Logickor:korean language model multidisciplinary reasoning benchmark. find
  24. Stability-AI. 2024. Logickor:korean language model multidisciplinary reasoning benchmark. find
  25. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, and 1 others. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yoo-2026-elo-efficient-layer-specific-optimization/