Elamur Research⌕
☰
home / papers / 2026 / Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameters

Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameters

Paper · 2026 · Zhiyu Xu, Lean Wang, Yuanxin Liu et al.

Summary

Vision-language models (VLMs) handle general multimodal tasks well, but they are slow to acquire new specialized skills. Fine-tuning requires curated image–text data and heavy compute. The authors study a cheaper route: merging a domain-expert, text-only LLM into the language backbone of a VLM while leaving the vision encoder untouched. They call this "cross-modal skill injection." Unlike merging peer LLMs, it is asymmetric, and its goal is to make abilities emerge that work on visual input.

The study covers three dimensions:

  • Scenarios: language ability, mathematical reasoning and instruction following, measured on six benchmarks.
  • Methods: nine merging techniques, grouped as classic, data-aware or tuning-free.
  • Hyperparameters: a comparison of several black-box optimizers for tuning.

Injection transfers cross-lingual and instruction-following skills well, but mathematical reasoning transfers poorly. Simple classic methods, especially Task Arithmetic and DARE, beat more elaborate alternatives, and data-aware methods add little. The work gives practitioners guidance on when merging is worthwhile, which method suits their resources, and how to tune it efficiently.

Key points

  • The nine methods fall into three families: classic (TA, TIES, DARE), data-aware (Fisher Merging, RegMean) and tuning-free (WUDI, TSV, MetaGPT, NaN). The tuning-free NaN ranks second overall, which makes it a low-cost way to explore.
  • Visual math is evaluated with MathVista (math subset) and MathVerse. Gains here are only minor, while language and instruction-following scenarios show major gains.
  • The hyperparameter space is small (2–3 parameters, 40–60 evaluations) and has a few local optima.
    • GP-BO is the most effective optimizer.
    • Pattern Search is competitive but can get stuck in local optima, and Powell's method is slightly weaker.
    • CMA-ES performs poorly because the evaluation budget is too small for it.
  • Tuning difficulty varies by method: Task Arithmetic is easiest, DARE is next, and TIES is hardest. Instruction-following tasks are the toughest for local optimizers.
  • The scope is limited to vision–language models; audio and video are left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China315 works · 43,267 citations🏢TencentShenzhen · China163 works · 21,032 citations🎓University of Hong KongPok Fu Lam · China118 works · 13,496 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 811 works

Cite

Cite as Zhiyu Xu, Lean Wang, Yuanxin Liu, Lei Li, Hao Zhou, Fandong Meng, Jie Zhou and Xu Sun. 2026. Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameters. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.571

@inproceedings{xu2026investigating,
  title = {{Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameters}},
  author = {Zhiyu Xu, Lean Wang, Yuanxin Liu, Lei Li, Hao Zhou, Fandong Meng, Jie Zhou and Xu Sun},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.571},
  url = {https://doi.org/10.18653/v1/2026.acl-long.571}
}

References

55 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, and 8… — Flamingo: a Visual Language Model for Few-Shot Learning
  2. Shiqi Chen, Jinghan Zhang, Tongyao Zhu, Wei Liu, Siyang Gao, Miao Xiong, Manling Li, and Junxian He. 2025a. Bring reason to vision: Understanding perception and reasoning through model merging. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Mach… find
  3. Zhipeng Chen, Kun Zhou, Liang Song, Wayne Xin Zhao, Bingning Wang, Weipeng Chen, and Ji-Rong Wen. 2025b. Extracting and combining abilities for building multi-lingual ability-enhanced large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pa… find
  4. Runxi Cheng, Feng Xiong, Yongxian Wei, Wanyun Zhu, and Chun Yuan. 2025. Whoever started the interference should end it: Guiding data-free model merging via task vectors. In Forty-second International Conference on Machine Learning. find
  5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  6. Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm. find
  7. Yiming Cui, Ziqing Yang, and Xin Yao. 2024. Efficient and effective text encoding for chinese llama and alpaca. Preprint, arXiv:2304.08177.
  8. Yang Dai, Jianxiang An, Tianwei Lin, Hongyang He, Hongzhe Huang, Wenqiao Zhang, Zheqi Lv, Siliang Tang, and Yueting Zhuang. 2025. Graft: Integrating the domain knowledge via efficient parameter synergy for mllms. ArXiv, abs/2506.23940. — [arXiv:2506.23940](https://arxiv.org/abs/2506.23940) find
  9. MohammadReza Davari and Eugene Belilovsky. 2024. Model breadcrumbs: Scalable upcycling of finetuned foundation models via sparse task vectors merging. In ICML 2024 Workshop on Foundation Models in the Wild. find
  10. Pala Tej Deep, Rishabh Bhardwaj, and Soujanya Poria. 2024. Della-merging: Reducing interference in model merging through magnitude-based sampling. Preprint, arXiv:2406.11617. — [arXiv:2406.11617](https://arxiv.org/abs/2406.11617) find
  11. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient finetuning of quantized LLMs. In Thirty-seventh Conference on Neural Information Processing Systems.
  12. Yiyang Du, Xiaochen Wang, Chi Chen, Jiabo Ye, Yiru Wang, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Zhifang Sui, Maosong Sun, and Yang Liu. 2025. Adamms: Model merging for heterogeneous multimodal large language models with unsupervised coefficient optimization. CoRR, abs/2503.23733. fujiki. 2023. japa… — [arXiv:2503.23733](https://arxiv.org/abs/2503.23733) find
  13. Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodolà. 2024. Task singular vectors: Reducing task interference in model merging. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18695–18705. find
  14. Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024. Arcee’s MergeKit: A toolkit for merging large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing:…
  15. Jiawei Guo, Tianyu Zheng, Yizhi Li, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Graham Neubig, Wenhu Chen, and Xiang Yue. 2025. MAmmoTH-VL: Eliciting multimodal reasoning with instruction tuning at scale. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume…
  16. Nikolaus Hansen and Andreas Ostermeier. 2001. Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation, 9:159–195. find
  17. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  18. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations.
  19. Lingjie Jiang, Shaohan Huang, Xun Wu, Yixia Li, Guanhua Chen, Dongdong Zhang, and Furu Wei. 2026. Viscodex: Unified multimodal code generation via merging vision and coding models. In The Fourteenth International Conference on Learning Representations. find
  20. Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. 2023. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations.
  21. Donald R Jones, Matthias Schonlau, and William J Welch. 1998. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455–492. find
  22. Tamara G Kolda, Robert Michael Lewis, and Virginia Torczon. 2003. Optimization by direct search: New perspectives on some classical and modern methods. SIAM review, 45(3):385–482. find
  23. Hugo Laurencon, Leo Tronchon, Matthieu Cord, and Victor Sanh. 2024. What matters when building vision-language models? In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  24. Bo Li, Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu. 2024. Lmms-eval: Accelerating the development of large multimoal models. find
  25. Chen-An Li, Tzu-Han Lin, Yun-Nung Chen, and Hung-yi Lee. 2025a. Transferring textual preferences to vision-language understanding through model merging. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 923–943, Vienna, Austria… find
  26. Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2023a. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. In Thirty-seventh Conference on Neural Information Processing Systems Datasets…
  27. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b. Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.
  28. Yuan Li, Zhengzhong Liu, and Eric P. Xing. 2025b. Data mixing optimization for supervised fine-tuning of large language models. In Forty-second International Conference on Machine Learning. find
  29. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. arXiv preprint arXiv:2305.20050.
  30. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. Improved baselines with visual instruction tuning.
  31. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024a. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306.
  32. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024b. Llava-next: Improved reasoning, ocr, and world knowledge. find
  33. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems.
  34. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024a. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations…
  35. Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. 2024b. Wildvision: Evaluating vision-language models in the wild with human preferences. Preprint, arXiv:2406.11069. — [arXiv:2406.11069](https://arxiv.org/abs/2406.11069) find
  36. Michael Matena and Colin Raffel. 2022. Merging models with fisher-weighted averaging. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc.
  37. Shota Onohara, Atsuyuki Miyai, Yuki Imajuku, Kazuki Egashira, Jeonghun Baek, Xiang Yue, Graham Neubig, and Kiyoharu Aizawa. 2025. JMMMU: A Japanese massive multi-discipline multimodal understanding benchmark for culture-aware evaluation. In Proceedings of the 2025 Conference of the Nations of the Am… find
  38. Michael JD Powell. 1964. An efficient method for finding the minimum of a function of several variables without calculating derivatives. The computer journal, 7(2):155–162. find
  39. Yusu Qian, Hanrong Ye, Jean-Philippe Fauconnier, Peter Grasch, Yinfei Yang, and Zhe Gan. 2025. MIAbench: Towards better instruction following evaluation of multimodal LLMs. In The Thirteenth International Conference on Learning Representations. find
  40. Huaizhi Qu, Xinyu Zhao, Jie Peng, Kwonjoon Lee, Behzad Dariush, and Tianlong Chen. 2025. Uq-merge: Uncertainty guided multimodal large language model merging. In Findings of the Association for Computational Linguistics: ACL 2025, pages 1401–1417, Vienna, Austria. Association for Computational Lingu… find
  41. Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. 2024. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4663–4680, Miami, Florid…
  42. Chongjie Si, Kangtao Lv, Jingjing Jiang, Yadao Wang, Yongwei Wang, Xiaokang Yang, Wenbo Su, Bo Zheng, and Wei Shen. 2025. Nan: A training-free solution to coefficient estimation in model merging. Preprint, arXiv:2505.16148. — [arXiv:2505.16148](https://arxiv.org/abs/2505.16148) find
  43. I.M Sobol. 1967. On the distribution of points in a cube and the approximate evaluation of integrals. USSR Computational Mathematics and Mathematical Physics, 7(4):86–112. find
  44. Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. 2021. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’2…
  45. Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022. Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5227–5237. find
  46. Virginia Torczon. 1997. On the convergence of pattern search algorithms. SIAM Journal on optimization, 7(1):1–25. find
  47. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Ke-Yang Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the…
  48. Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. 2023. TIES-merging: Resolving interference when merging models. In Thirty-seventh Conference on Neural Information Processing Systems.
  49. Junyao Yang, Jianwei Wang, Huiping Zhuang, Cen Chen, and Ziqian Zeng. 2025. Rcp-merging: Merging long chain-of-thought models with domain-specific models by considering reasoning capability as prior. ArXiv, abs/2508.03140. — [arXiv:2508.03140](https://arxiv.org/abs/2508.03140) find
  50. Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: absorbing abilities from homologous models as a free lunch. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.
  51. Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng, Chunpu Xu, Shuyue Guo, Haoran Zhang, Xingwei Qu, Junjie Wang, Ruibin Yuan, Yizhi Li, Zekun Wang, Yudong Liu, Yu-Hsuan Tsai, Fengji Zhang, and 3 others. 2024a. Cmmmu: A chinese massive multi-discipline mult… — [arXiv:2401.11944](https://arxiv.org/abs/2401.11944) find
  52. Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. 2024b. Lmms-eval: Reality check on the evaluation of large multimodal models. Preprint, arXiv:2407.12772.
  53. Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, Peng Gao, and Hongsheng Li. 2024c. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In Computer Vision – ECCV 2024: 18th European Conference,…
  54. Yuyan Zhou, Liang Song, Bingning Wang, and Weipeng Chen. 2024. MetaGPT: Merging large language models using model exclusive task arithmetic. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1711– 1724, Miami, Florida, USA. Association for Computational… find
  55. Didi Zhu, Yibing Song, Tao Shen, Ziyu Zhao, Jinluan Yang, Min Zhang, and Chao Wu. 2025. REMEDY: Recipe merging dynamics in large vision-language models. In The Thirteenth International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2026-investigating-cross-modal-skill-injection/