Elamur Research⌕
☰
home / papers / 2025 / LIMR: Less is More for RL Scaling

LIMR: Less is More for RL Scaling

Paper · 2025 · Xuefeng Li, Haoyang Zou and Pengfei Liu

Summary

Reinforcement learning (RL) has become central to improving LLM reasoning, as in o1, Deepseek R1 and Kimi1.5. Yet it is still unclear how much RL training data is needed, and which kind. The authors test whether enlarging the RL dataset is really what drives gains. They work in a "zero" setting: PPO is applied directly to base Qwen2.5-Math-7B, with no distillation or SFT warm-up. Their method, Learning Impact Measurement (LIM), records each problem's reward across training epochs. It then scores how closely that trajectory follows the model's average reward curve and keeps problems scoring above θ = 0.6. Applied to MATH-FULL (8,523 problems, levels 3–5), this yields LIMR, a 1,389-problem subset. RL on LIMR matches or slightly exceeds full-dataset training on AIME24, MATH500 and AMC23, and it clearly beats an equal-sized random subset. At 7B scale, it also strongly outperforms SFT on the LIMO and s1 data. The implication is that automated sample selection may matter more than data volume and can cut RL compute. Code, data and models are released.

Key points

  • Training setup: PPO in OpenRLHF, rollout batch 1,024, 8 samples per prompt at temperature 1.2, KL coefficient 0.01. The rule-based reward gives +1 for a correct answer, −0.5 for a wrong but well-formatted one, and −1 for a format error.
  • Main results (avg. over AIME24/MATH500/AMC23): base model 40.5, FULL 57.0, RAND 49.4, LINEAR 54.9, LIMR 58.1. On AIME24, LIMR equals FULL (32.5). On MATH500 (78.0 vs 76.6) and AMC23 (63.8 vs 61.9), LIMR edges ahead.
  • RL vs. SFT at 7B: SFT on s1 (1,000 samples) and LIMO (817 samples) reaches only 15.8 on AIME24, 55.8 and 65.0 on MATH500, and averages of 38.0 and 45.7. LIMR scores 32.5 on AIME24 and 78.0 on MATH500, with an average of 58.1.
  • LINEAR baseline: this alternative favours samples that improve steadily, using θ = 0.7. The paper reports its subset size inconsistently: 1,189 in the text and 1,138 in Table 2.
  • Open resources: the LIM implementation, training and evaluation code, the LIMR dataset and the trained models are available at github.com/GAIR-NLP/LIMR.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

Cite

Cite as Xuefeng Li, Haoyang Zou and Pengfei Liu. 2025. LIMR: Less is More for RL Scaling. arXiv:2502.11886.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.11886 · arXiv 2502.11886

@misc{li2025limr,
  title = {{LIMR: Less is More for RL Scaling}},
  author = {Xuefeng Li, Haoyang Zou and Pengfei Liu},
  year = {2025},
  doi = {10.48550/arxiv.2502.11886},
  eprint = {2502.11886},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.11886}
}

References

16 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. 2025. Process reinforcement th…
  2. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  3. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.
  4. Zhenyu Hou, Xin Lv, Rui Lu, Jiajie Zhang, Yujiang Li, Zijun Yao, Juanzi Li, Jie Tang, and Yuxiao Dong. 2025. Advancing language model reasoning through reinforcement learning and inference scaling. arXiv preprint arXiv:2501.11651.
  5. Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. 2024. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.11143.
  6. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
  7. Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. 2025. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. https://pretty-radio-b75.notion.site/DeepScaleR-Surpassing… ↗
  8. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling.
  9. OpenAI. 2024. Learning to reason with llms, september 2024. find
  10. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. find
  11. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Z… — Kimi k1.5: Scaling Reinforcement Learning with LLMs
  12. NovaSky Team. 2025a. Sky-t1: Train your own o1 preview model within 450. ai.github.io/posts/sky-t1. Accessed: 2025-01-09. find
  13. RUCAIBox STILL Team. 2025b. Still-3-1.5b-preview: Enhancing slow thinking abilities of small models through reinforcement learning. find
  14. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement.
  15. Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. 2025. Limo: Less is more for reasoning.
  16. Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 2025. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. https://hkust-nlp.notion.site/simplerl-reason. Notion Blog. https://novasky- ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2025-limr-less-more-rl-scaling/