Elamur Research⌕
☰
home / papers / 2026 / Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance

Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance

Summary

Financial NLP needs several skills at once: domain knowledge, numerical reasoning and multilingual handling. Training one model on all of them is expensive and prone to catastrophic forgetting. Model merging has mostly been studied for SFT models, so the authors test whether continual pretraining (CPT) experts, which drift much further from their base weights, can also be combined usefully. They start from Llama-3-8B and three public CPT checkpoints built on it, specialised in finance, mathematics and Japanese. They apply Task Arithmetic, TIES and DARE-TIES within a three-stage pipeline:

  1. Each expert is merged with the base.
  2. The two best results are combined.
  3. The third domain is added.

Evaluation uses a finance benchmark they assembled from public data. Merging an expert back with its base restores general ability lost during CPT. Pairing experts can exceed every constituent and sometimes produces new cross-domain skills. The work offers practical guidance for building multi-skill domain LLMs from existing open models without further training.

Key points

  • Models: The base is Llama-3-8B. The experts are finance-Llama3-8B (instruction-pretrain), MathCoder2-Llama-3-8B and Llama-3-Swallow-8B-v0.1. All three were trained by independent groups from the same checkpoint and used as downloaded.
  • Benchmark: 18 tasks drawn from 8 established financial datasets, designed to give a fine-grained skill profile rather than a single aggregate score.
  • Merging methods:
    • Task Arithmetic gives the highest peaks, best when roughly 10–30% of parameters come from the secondary model, but it is very sensitive to λ.
    • TIES is the most stable across settings.
    • DARE-TIES seldom helps.
  • Similarity and gains: Closer model pairs (lower L2 distance, higher cosine similarity) yield larger Macro-Gain (Spearman, p < 0.05). Emergence (Macro-OG) shows no such link (p > 0.60), though it tends to collapse when L2 > 120 or cosine < 0.978.
  • Cross-lingual effect and limits: Adding the Japanese expert even improved English tasks. The study's limits are that it covers only one architecture and only finance tasks, with no general-purpose evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3cyxer74uvyp

@inproceedings{ueda2026merging,
  title = {{Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance}},
  author = {Ueda, Kentaro and Portet, François and Suwa, Hirohiko and Yasumoto, Keiichi},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3cyxer74uvyp},
  url = {https://doi.org/10.63317/3cyxer74uvyp}
}

References

32 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI@Meta. 2024. Llama 3 model card. find
  2. Antoine Bigeard, Langston Nashold, Rayan Krishnan, and Shirley Wu. 2025. Finance agent benchmark: Benchmarking llms on real-world financial research tasks. find
  3. Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022. ConvFinQA: Exploring the chain of numerical reasoning in conversational finance question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6279–62… find our card
  4. Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. 2024. Instruction pre-training: Language models are supervised multitask learners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2529– 2550, Miami, Florida, USA. Associat… find
  5. Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. Continual pre-training for cross-lingual llm adaptation: Enhancing japanese language capabilities. In Proceedings of the First Conference on Langua… find our card
  6. Masanori Hirano and Kentaro Imajo. 2024. The construction of instruction-tuned llms for finance without instruction data using continual pre-training and model merging. arXiv preprint arXiv:2409.19854. — [arXiv:2409.19854](https://arxiv.org/abs/2409.19854) find
  7. Shih-Cheng Huang, Pin-Zu Li, Yu-chi Hsu, Kuang-Ming Chen, Yu Tung Lin, Shih-Kai Hsiao, Richard Tsai, and Hung-yi Lee. 2024. Chat vector: A simple approach to equip LLMs with instruction following and model alignment in new languages. In Proceedings of the 62nd Annual Meeting of the Association for C… find
  8. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. Editing models with task arithmetic. find our card
  9. Rasmus Jørgensen, Oliver Brandt, Mareike Hartmann, Xiang Dai, Christian Igel, and Desmond Elliott. 2023. MultiFin: A dataset for multilingual financial NLP. In Findings of the Association for Computational Linguistics: EACL 2023, pages 894–909, Dubrovnik, Croatia. Association for Computational Lingu… find
  10. Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. 2023. Large language models in finance: A survey. In Proceedings of the Fourth ACM International Conference on AI in Finance, ICAIF ’23, page 374–382, New York, NY, USA. Association for Computing Machinery. find
  11. Yunshui Li, Yiyuan Ma, Shen Yan, Chaoyi Zhang, Jing Liu, Jianqiao Lu, Ziwen Xu, Mengzhao Chen, Minrui Wang, Shiyi Zhan, et al. 2025. Model merging in pre-training of large language models. arXiv preprint arXiv:2505.12082. — [arXiv:2505.12082](https://arxiv.org/abs/2505.12082) find
  12. Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. 2024. Mathcoder2: Better math reasoning from continued pretraining on model-translated mathematical code. find
  13. Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. Www’18 open challenge: Financial opinion mining and question answering. In Companion Proceedings of the The Web Conference 2018, WWW ’18, page 1941–1942, Republic and Canton of G… find
  14. Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala. 2014. Good debt or bad debt: Detecting semantic orientations in economic texts. J. Assoc. Inf. Sci. Technol., 65(4):782–796. find
  15. Julio Cesar Salinas Alvarado, Karin Verspoor, and Timothy Baldwin. 2015. Domain adaption of named entity recognition to support credit risk assessment. In Proceedings of the Australasian Language Technology Association Workshop 2015, pages 84–90, Parramatta, Australia. find
  16. Ankur Sinha and Tanmay Khandait. 2021. Impact of news on the commodity market: Dataset and results. In Advances in Information and Communication: Proceedings of the 2021 Future of Information and Communication Conference (FICC), Volume 2, pages 589–601. Springer. find
  17. Shamane Siriwardhana, Mark McQuade, Thomas Gauthier, Lucas Atkins, Fernando Fernandes Neto, Luke Meyers, Anneketh Vij, Tyler Odenthal, Charles Goddard, Mary MacCarthy, et al. 2024. Domain adaptation of llama3-70b-instruct through continual pre-training and model merging: A comprehensive evaluation.… — [arXiv:2406.14971](https://arxiv.org/abs/2406.14971) find
  18. Pragya Srivastava, Manuj Malik, Vivek Gupta, Tanuja Ganu, and Dan Roth. 2024. Evaluating LLMs’ mathematical reasoning in financial document question answering. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3853–3878, Bangkok, Thailand. Association for Computational Li… find
  19. Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozière, Jacob Kahn, Daniel Li, Wen-tau Yih, Jason Weston, et al. 2024. Branch-train-mix: Mixing expert llms into a mixture-of-experts llm. arXiv preprint arXiv:2403.07816. — [arXiv:2403.07816](https://arxiv.org/abs/2403.07816) find
  20. Yi-Lin Sung, Linjie Li, Kevin Lin, Zhe Gan, Mohit Bansal, and Lijuan Wang. 2023. An empirical study of multimodal model merging. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1563–1575, Singapore. Association for Computational Linguistics. find
  21. Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh, and Philipp Koehn. 2019. Overcoming catastrophic forgetting during domain adaptation of neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find
  22. Cornelius Wolff and Madelon Hulsebos. 2025. How well do llms reason over tabular data, really? In Proceedings of the 4th Table Representation Learning Workshop, pages 241–250. find
  23. Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. Bloomberggpt: A large language model for finance. ArXiv, abs/2303.17564. find our card
  24. Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2023. Pixiu: A large language model, instruction data and evaluation benchmark for finance. arXiv preprint arXiv:2306.05443. — [arXiv:2306.05443](https://arxiv.org/abs/2306.05443) find
  25. Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2023. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36:7093–7115. find our card
  26. Prateek Yadav, Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, and Tsendsuren Munkhdalai. 2025. What matters for model merging at scale? Transactions on Machine Learning Research. find
  27. Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. 2024. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.07666. find
  28. Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. 2023. Fingpt: Open-source financial large language models. FinLLM Symposium at IJCAI 2023. find our card
  29. Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: absorbing abilities from homologous models as a free lunch. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. find our card
  30. Yilun Zhao, Hongjun Liu, Yitao Long, Rui Zhang, Chen Zhao, and Arman Cohan. 2024a. Financemath: Knowledge-intensive math reasoning in finance domains. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12841–12858, Bangkok, Thail… find
  31. Yilun Zhao, Yitao Long, Hongjun Liu, Ryo Kamoi, Linyong Nan, Lyuhao Chen, Yixin Liu, Xiangru Tang, Rui Zhang, and Arman Cohan. 2024b. DocMath-eval: Evaluating math reasoning capabilities of LLMs in understanding long and specialized documents. In Proceedings of the 62nd Annual Meeting of the Associa… find
  32. Yuyan Zhou, Liang Song, Bingning Wang, and Weipeng Chen. 2024. MetaGPT: Merging large language models using model exclusive task arithmetic. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1711–1724, Miami, Florida, USA. Association for Computational… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ueda-2026-merging-continual-pretraining-models-domain/