Elamur Research⌕
☰
home / papers / 2026 / 高效提升多模态大语言模型推理能力的级联强化学习策略

高效提升多模态大语言模型推理能力的级联强化学习策略

Summary

Reinforcement learning is now a standard step for improving the reasoning of multimodal large language models, but each family of methods has a weakness. Online methods such as GRPO and GSPO reach better final accuracy, but they are expensive because the policy must generate rollouts during training, and they converge slowly. Offline preference methods such as DPO and MPO are cheaper but stop at a lower ceiling. The authors ran controlled experiments on the M3CoT dataset and propose CascadeRL, which runs the two in sequence on the same training questions. First, MPO uses pre-collected trajectories, sampled from a smaller model, to quickly prune low-quality regions of the output distribution. Then GSPO refines the model with its own rollouts. CascadeRL matches or beats pure online RL at about half the training cost. Applied to the eight instruction-tuned InternVL3.5 models, it raises average accuracy on seven multimodal reasoning benchmarks by close to 8%. An analysis of log-probabilities on positive and negative samples explains why it works: RL suppresses wrong answers, while SFT unintentionally makes them more likely.

Key points

  • Experimental setup: The analysis uses the M3CoT train and validation splits, which contain 7,861 training items. A limited-data regime is simulated by training on only half of them.
  • Findings on data size: With limited data, offline RL converges faster and gains more than online RL. With the full data, online RL reaches a higher ceiling. Offline-then-online training on the same questions rivals two rounds of online RL.
  • Offline stage: MPO combines a DPO preference loss, a BCO quality loss and an LM generation loss.
  • Online stage: GSPO is used without a KL term to a reference model. It samples 16 responses per question, and the reward gives 0.1 for correct format and 0.9 for a correct answer. GSPO was chosen for its stability on MoE models.
  • Main results: InternVL3.5-8B-Instruct gains +6.7% and InternVL3.5-241B-A28B-Instruct gains +6.5% in average accuracy across the seven benchmarks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Natural language processing

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.16383/j.aas.c250515

@inproceedings{2026,
  title = {{高效提升多模态大语言模型推理能力的级联强化学习策略}},
  author = {王玮赟 and 蒲恒骏 and 景凌林 and 乔宇},
  booktitle = {自动化学报 (Acta Automatica Sinica)},
  year = {2026},
  doi = {10.16383/j.aas.c250515},
  url = {https://doi.org/10.16383/j.aas.c250515}
}

References

69 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shao Z H, Wang P Y, Zhu Q H, Xu R X, Song J X, Bi X, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv: 2402.03300, 2024. find our card
  2. Zheng C J, Liu S X, Li M Z, Chen X H, Yu B W, Gao C, et al. Group sequence policy optimization. arXiv preprint arXiv: 2507.18071, 2025. find our card
  3. Chen Q G, Qin L B, Zhang J, Chen Z, Xu X, Che W X. M3CoT: A novel benchmark for multi-domain multi-step multimodal chain-of-thought. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. find
  4. Bangkok, Thailand: ACL, 2024. 8199−8221 find
  5. Wang W Y, Gao Z W, Gu L X, Pu H J, Cui L, Wei X G, et al. InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv: 2508. 18265, 2025. find our card
  6. Touvron H, Lavril T, Izacard G, Martinet X, Lachaux M A, Lacroix T, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv: 2302.13971, 2023. find
  7. Cai Z, Cao M S, Chen H J, Chen K, Chen K Y, Chen X, et al. Intern-LM2 technical report. arXiv preprint arXiv: 2403.17297, 2024. find our card
  8. Yang A, Li A F, Yang B S, Zhang B C, Hui B Y, Zheng B, et al. Qwen3 technical report. arXiv preprint arXiv: 2505.09388, 2025. find our card
  9. OpenAI. Introducing gpt-oss [Online], available: https://openai.com/index/introducing-gpt-oss/, August 5, 2025 ↗
  10. Ouyang L, Wu J, Jiang X, Almeida D, Wainwright C L, Mishkin P, et al. Training language models to follow instructions with human feedback. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. New Orleans, USA: Curran Associates Inc., 2022. find
  11. Xu Zheng-Fei, Xin Xin. An empirical study of Chinese entity linking based on large language model. Acta Automatica Sinica, 2025, 51(2): 327−342 (徐正斐, 辛欣. 基于大语言模型的中文实体链接实证研究. 自动化 学报, 2025, 51(2): 327−342) find
  12. Qin Long, Wu Wan-Sen, Liu Dan, Hu Yue, Yin Quan-Jun, Yang Dong-Sheng, et al. Autonomous planning and processing framework for complex tasks based on large language models. Acta Automatica Sinica, 2024, 50(4): 862−872 (秦龙, 武万森, 刘丹, 胡越, 尹全军, 阳东升, 等. 基于大语言模型 的复杂任务自主规划处理框架. 自动化学报, 2024, 50(4): 862− 872) find
  13. Chen Chu-Yan, Liu Ye-Xu, Jia Wei-Chen, He Yu-Tong, Yuan Kun, Wang Li-Wei. An effective algorithm for LLMs training with one-bit communication compression. Acta Automatica Sinica, DOI: 10.16383/j.aas.c250087 (陈楚岩, 刘烨谞, 贾维宸, 何雨桐, 袁坤, 王立威. 一种基于单比特 通信压缩的大模型训练方法研究. 自动化学报, DOI: 10.16383/j.aas.c250087) Yan… — [doi:10.16383/j.aas.c250087](https://doi.org/10.16383/j.aas.c250087) find
  14. Li Z J, Zhang J W, Wang Y, Du M F, Liu Q W, Wang D Y, et al. From multi-modal pre-training to multi-modal large language models: An overview of architectures, training. In: Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 2: Frontier Forum). Taiyuan, China: Ch… find
  15. Zheng Yi-Ning, Yu Zhen, Li Bu-Fan, Yang Jie, Yin Lin-Qi, Yin Zhang-Yue, et al. Survey of tool use in large language models. Acta Automatica Sinica, 2025, 51(11): 2371−2386 (郑逸宁, 余镇, 李不凡, 杨捷, 殷林琪, 印张悦, 等. 大语言模型的 工具使用综述. 自动化学报, 2025, 51(11): 2371−2386) find
  16. Schuhmann C, Beaumont R, Vencu R, Gordon C, Wightman R, Cherti M, et al. LAION-5B: An open large-scale dataset for training next generation image-text models. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. New Orleans, USA: Curran Associates Inc., 2022… find our card
  17. Li Q Y, Chen Z, Wang W Y, Wang W H, Ye S L, Jin Z J, et al. OmniCorpus: An unified multimodal corpus of 10 billion-level images interleaved with text. arXiv preprint arXiv: 2406.08418, 2024. — [arXiv:2406.08418](https://arxiv.org/abs/2406.08418) find
  18. Laurençon H, Saulnier L, Tronchon L, Bekman S, Singh A, Lozhkov A, et al. OBELICS: An open web-scale filtered dataset of interleaved image-text documents. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. New Orleans, USA: Curran Associates Inc., 2023. Ar… find our card
  19. Liu Y Z, Cao Y, Gao Z W, Wang W Y, Chen Z, Wang W H, et al. MMInstruct: A high-quality multi-modal instruction tuning dataset with extensive diversity. arXiv preprint arXiv: 2407. 15838, 2024. find
  20. Wang W Y, Shi M, Li Q Y, Wang W H, Huang Z H, Xing L J, et al. The all-seeing project: Towards panoptic visual recognition and understanding of the open world. In: Proceedings of the 12th International Conference on Learning Representations. Vienna, Austria: OpenReview.net, 2024. find
  21. Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707. 06347, 2017. find
  22. Zheng R, Dou S H, Gao S Y, Hua Y, Shen W, Wang B H, et al. Secrets of RLHF in large language models part I: PPO. arXiv preprint arXiv: 2307.04964, 2023. find our card
  23. Bradley R A, Terry M E. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 1952, 39(3−4): 324−345 find
  24. Jung S, Han G, Nam D W, On K W. Binary classifier optimization for large language model alignment. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. Vienna, Austria: ACL, 2025. 1858−1872 find
  25. Hong W Y, Yu W M, Gu X T, Wang G, Gan G B, Tang H M, et al. GLM-4.5V and GLM-4.1V-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv: 2507.01006, 2025. find
  26. Keye T K, Yang B, Wen B, Liu C Y, Chu C L, Song C R, et al. Kwai keye-VL technical report. arXiv preprint arXiv: 2507. 01949, 2025. find
  27. Bengio Y, Louradour J, Collobert R, Weston J. Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. Montreal, Canada: ACM, 2009. 41−48 find our card
  28. Wang X, Chen Y D, Zhu W W. A survey on curriculum learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(9): 4555−4576 find
  29. Xi Z H, Chen W X, Hong B Y, Jin S J, Zheng R, He W, et al. Training large language models for reasoning through reverse curriculum reinforcement learning. In: Proceedings of the 41st International Conference on Machine Learning. Vienna, Austria: JMLR.org, 2024. Article No. 2217 find
  30. Wu J X, Zhang D M, Zhong S L, Qiao H. Trajectory-based split hindsight reverse curriculum learning. In: Proceedings of the 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). find
  31. Prague, Czech Republic: IEEE, 2021. 3971−3978 find
  32. Qu Y X, Zhang T J, Garg N M, Kumar A. RECURSIVE IN-TROSPECTION: Teaching language model agents how to self-improve. In: Proceedings of the 38th International Conference on Neural Information Processing Systems. Vancouver, Canada: Curran Associates Inc., 2024. Article No. 1754 find
  33. Wang Z T, Cui G F, Li Y J, Wan K, Zhao W T. DUMP: Automated distribution-level curriculum learning for RL-based LLM post-training. arXiv preprint arXiv: 2504.09710, 2025. — [arXiv:2504.09710](https://arxiv.org/abs/2504.09710) find
  34. Wen L, Cai Y K, Xiao F R, He X, An Q, Duan Z Y, et al. Light-R1: Curriculum SFT, DPO and RL for long COT from scratch and beyond. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track). Vienna, Austria: ACL, 2025. 318−327 find
  35. Bi X, Chen D L, Chen G T, Chen S H, Dai D M, Deng C Q, et al. DeepSeek LLM: Scaling open-source language models with longtermism. arXiv preprint arXiv: 2401.02954, 2024. find
  36. Sun C N, Jung G Y, Tran T X, Pompili D. Cascade reinforcement learning with state space factorization for O-RAN-based traffic steering. In: Proceedings of the 21st Annual IEEE International Conference on Sensing, Communication, and Networking (SECON). Phoenix, USA: IEEE, 2024. 1−9 find
  37. Castillo G A, Weng B W, Zhang W, Hereid A. Reinforcement learning-based cascade motion policy design for robust 3D bipedal locomotion. IEEE Access, 2022, 10: 20135−20148 find
  38. Xu J, Qiao J B, Sun Q L, Shen K Y. A deep reinforcement learning framework for cascade reservoir operations under runoff uncertainty. Water, 2025, 17(15): Article No. 2324 find
  39. Pal A, Karkhanis D, Dooley S, Roberts M, Naidu S, White C. Smaug: Fixing failure modes of preference optimisation with DPO-positive. arXiv preprint arXiv: 2402.13228, 2024. find
  40. Bai J Z, Bai S, Yang S S, Wang S J, Tan S N, Wang P, et al. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv: 2308.12966, 2023. find our card
  41. Yue X, Ni Y S, Zhang K, Zheng T Y, Liu R Q, Zhang G, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. arXiv preprint arXiv: 2311.16502, 2023. find our card
  42. Zhang R R, Jiang D Z, Zhang Y C, Lin H K, Guo Z Y, Qiu P S, et al. MathVerse: Does your multi-modal LLM truly see the diagrams in visual math problems? arXiv preprint arXiv: 2403. 14624, 2024. find our card
  43. Yue Z H, Lin Z R, Song Y F, Wang W K, Ren S H, Gu S H, et al. Mimo-VL technical report. arXiv preprint arXiv: 2506.03569, 2025. find
  44. Shen W, Pei J B, Peng Y, Song X C, Liu Y, Peng J, et al. Skywork-R1V3 technical report. arXiv preprint arXiv: 2507.06167, 2025. find
  45. Qwen Team. QVQ: To see the world with wisdom [Online], available: https://qwen.ai/blog?id=qvq-72b-preview, December 25, 2024 ↗
  46. Qwen Team. Qwen2.5: A party of foundation models! [Online], available: https://qwenlm.github.io/blog/qwen2.5/, September 19, 2024 ↗
  47. Mesnard T, Hardin C, Dadashi R, Bhupatiraju S, Pathak S, Sifre L, et al. Gemma: Open models based on Gemini research and technology. arXiv preprint arXiv: 2403.08295, 2024. find
  48. StepFun Research Team. Step-1v: A hundred billion parameter multimodal large model [Online], available: https://platform.stepfun.com, March 23, 2024 ↗
  49. OpenAI. Gpt-5 system card [Online], available: https://cdn.openai.com/pdf/8124a3ceab78-4f06-96eb-49ea29ffb52f/gpt5-system-card-aug7.pdf, August 7, 2025 ↗
  50. Anthropic. The Claude 3 model family: Opus, sonnet, haiku [Online], available: https://www-cdn.anthropic.com/de8ba9b01 c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3. pdf, March 4, 2024 ↗
  51. Pichai S, Hassabis D, Kavukcuoglu K. Introducing Gemini 2.0: Our new AI model for the agentic era [Online], https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/, December 11, 2024 ↗
  52. Guo D, Wu F M, Zhu F D, Leng F X, Shi G, Chen H B, et al. Seed1.5-VL technical report. arXiv preprint arXiv: 2505.07062, 2025. find
  53. Lu P, Bansal H, Xia T, Liu J C, Li C Y, Hajishirzi H, et al. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv: 2310.02255, 2023. find our card
  54. Wang K, Pan J T, Shi W K, Lu Z M, Zhan M J, Li H S. Measuring multimodal mathematical reasoning with MATH-vision dataset. arXiv preprint arXiv: 2402.14804, 2024. find our card
  55. Zou C K, Guo X G, Yang R, Zhang J Y, Hu B, Zhang H. DynaMath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. arXiv preprint arXiv: 2411.00836, 2024. find
  56. Qiao R Q, Tan Q N, Dong G T, Wu M H, Sun C, Song X S, et al. We-Math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv: 2407.01284, 2024. find
  57. Xiao Y J, Sun E, Liu T Y, Wang W. LogicVista: Multimodal LLM logical reasoning benchmark in visual contexts. arXiv preprint arXiv: 2407.04973, 2024. — [arXiv:2407.04973](https://arxiv.org/abs/2407.04973) find
  58. OpenCompass Contributors. OpenCompass: A universal evaluation platform for foundation models [Online], available: https://github.com/open-compass/opencompass, July 4, 2023 ↗
  59. Zhu J G, Wang W Y, Chen Z, Liu Z Y, Ye S L, Gu L X, et al. InternVL3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv: 2504. 10479, 2025. find our card
  60. Lu S Y, Li Y, Chen Q G, Xu Z, Luo W H, Zhang K F, et al. Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint arXiv: 2405.20797, 2024. — [arXiv:2405.20797](https://arxiv.org/abs/2405.20797) find
  61. Bai S, Chen K Q, Liu X J, Wang J L, Ge W B, Song S B, et al. Qwen2.5-VL technical report. arXiv preprint arXiv: 2502.13923, 2025. find our card
  62. Yao Y, Yu T Y, Zhang A, Wang C Y, Cui J B, Zhu H J, et al. MiniCPM-V: A GPT-4V level MLLM on your phone. arXiv preprint arXiv: 2408.01800, 2024. find our card
  63. Chen Z, Wang W Y, Cao Y, Liu Y Z, Gao Z W, Cui E F, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv: 2412.05271, 2024. find our card
  64. Team G, Kamath A, Ferret J, Pathak S, Vieillard N, Merhej R, et al. Gemma 3 technical report. arXiv preprint arXiv: 2503. 19786, 2025. find
  65. Team K, Du A G, Yin B H, Xing B W, Qu B W, Wang B W, et al. Kimi-Vl technical report. arXiv preprint arXiv: 2504.07491, 2025. find
  66. StepFun Team. Step3: Cost-effective multimodal intelligence [Online], available: https://stepfun.ai/research/en/step3, July 25, 2025 ↗
  67. 王玮赟 复旦大学博士研究生. 2022 年获得哈尔滨工业大学计算机科学与 技术专业学士学位. 主要研究方向为 多模态大语言模型. E-mail: wywang22@m.fudan.edu.cn (WANG Wei-Yun Ph.D. candidate at Fudan University. He received his bachelor degree in computer science and technology from Harbin Institute of Technology in 2022. His main research interest is multimodal… find
  68. 蒲恒骏 上海人工智能实验室见习研 究员. 2024 年获得清华大学软件工程 专业学士学位. 主要研究方向为多模 态大语言模型. E-mail: puhengjun@pjlab.org.cn (PU Heng-Jun Research intern at Shanghai Artificial Intelligence Laboratory. He received his bachelor degree in software engineering from Tsinghua University in 2024. find
  69. His main research interest is multimodal large language models.) 景凌林 上海人工智能实验室青年研 究员. 2025 年获得英国拉夫堡大学计 算机科学与技术专业博士学位. 主要 研究方向为视频理解, 多模态大语言 模型. E-mail: jinglinglin@pjlab.org.cn (JING Ling-Lin Young researcher at Shanghai Artificial Intelligence Laboratory. He received his Ph.D. degree in computer sci… find

↑ back to top

This page in Markdown →