Elamur Research⌕
☰
home / papers / 2025 / Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RL

Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RL

Paper · 2025 · Che Liu, Haozhe Wang, Jiazhen Pan et al.

Summary

Medical LLMs usually learn to reason through supervised fine-tuning on chain-of-thought (CoT) traces, often distilled from proprietary systems such as GPT-4o. Producing these traces is expensive and ties model development to external models. This paper asks whether reinforcement learning alone, with a trivially simple reward, can produce reasoning instead. The authors train Llama3.1-8B-Instruct and Llama3.1-70B-Instruct with GRPO. The model receives a reward of 1 only when its final boxed multiple-choice answer is correct, and no SFT stage, preference data or learned reward model is used. The training data come from public QA sets, MedQA and MedMCQA. A data-centric study examines how quantity, diversity, informativeness and question difficulty affect results. The resulting models, called AlphaMed, set new state-of-the-art scores on six medical QA benchmarks and beat SFT+RL pipelines like HuatuoGPT-o1 and m1. The 70B version also outperforms GPT-4o, Claude-3.5-Sonnet and DeepSeek-V3-671B on harder sets. The work suggests that carefully curated multiple-choice data can replace costly distilled rationales. It also notes that current benchmarks only partly capture reasoning gains.

Key points

  • Training setup: full-parameter GRPO via the verl framework for 300 steps. Each batch of 512 consists of 64 questions with 8 sampled answers per question, and the reward is a binary check of the \boxed{} answer. The 8B model was trained on 8 A800 GPUs and the 70B model on 64.
  • Training set: 19,178 QA pairs, made up of all of MedQA plus 1,600 MedMCQA items per difficulty level. PubMedQA was dropped because it carried little information and hurt performance.
  • Evaluation: six benchmarks, namely MedQA, MedMCQA, PubMedQA, MMLU-ProM, GPQA-M and MedXpert. The last four are treated as out-of-domain, and MedXpert is labelled "hard+".
  • Scores: AlphaMed(8B) reaches 76.19 on MedQA and 22.14 on MedXpert. AlphaMed(70B) reaches 87.52 on MedQA and 32.56 on MedXpert.
  • Data findings:
    • Informativeness of the training data drives gains.
    • Easy data already triggers reasoning, and hard data mainly helps on MedXpert.
    • Mixing difficulty levels gives the best generalization.
    • Trends that differ across benchmarks point to weaknesses in current evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China167 works · 20,987 citations🎓Hong Kong University of Science and TechnologyHong Kong · China142 works · 25,136 citations🎓The Ohio State UniversityColumbus · United States82 works · 18,502 citations🎓Imperial College LondonLondon · United Kingdom48 works · 12,181 citations🎓Technical University of MunichMunich · Germany55 works · 48,474 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksEfficient models & compression · 242 works

Cite

Cite as Che Liu, Haozhe Wang, Jiazhen Pan, Zhongwei Wan, Yong Dai, Fangzhen Lin, Wenjia Bai, Daniel Rueckert and Rossella Arcucci. 2025. Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RL. arXiv:2505.17952.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.17952 · arXiv 2505.17952

@misc{liu2025beyond,
  title = {{Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RL}},
  author = {Che Liu, Haozhe Wang, Jiazhen Pan, Zhongwei Wan, Yong Dai, Fangzhen Lin, Wenjia Bai, Daniel Rueckert and Rossella Arcucci},
  year = {2025},
  doi = {10.48550/arxiv.2505.17952},
  eprint = {2505.17952},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.17952}
}

References

52 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Qin, X. Li, H. Zou, Y. Liu, S. Xia, Z. Huang, Y. Ye, W. Yuan, H. Liu, Y. Li et al., “O1 replication journey: A strategic progress report–part 1,” arXiv preprint arXiv:2410.18982, 2024. find
  2. Z. Zeng, Q. Cheng, Z. Yin, B. Wang, S. Li, Y. Zhou, Q. Guo, X. Huang, and X. Qiu, “Scaling of search and learning: A roadmap to reproduce o1 from reinforcement learning perspective,” arXiv preprint arXiv:2412.14135, 2024. find
  3. J. Wang, M. Fang, Z. Wan, M. Wen, J. Zhu, A. Liu, Z. Gong, Y. Song, L. Chen, L. M. Ni et al., “Openr: An open source framework for advanced reasoning with large language models,” arXiv preprint arXiv:2410.09671, 2024. find
  4. M. Y. Guan, M. Joglekar, E. Wallace, S. Jain, B. Barak, A. Heylar, R. Dias, A. Vallone, H. Ren, J. Wei, H. W. Chung, S. Toyer, J. Heidecke, A. Beutel, and A. Glaese, “Deliberative alignment: Reasoning enables safer language models,” OpenAI Blog, 2024. [Online]. Available: https://openai.com/index/de… ↗
  5. K. Saab, T. Tu, W.-H. Weng, R. Tanno, D. Stutz, E. Wulczyn, F. Zhang, T. Strother, C. Park, E. Vedadi et al., “Capabilities of gemini models in medicine,” arXiv preprint arXiv:2404.18416, 2024. find
  6. J. Chen, C. Gui, A. Gao, K. Ji, X. Wang, X. Wan, and B. Wang, “Cod, towards an interpretable medical agent using chain of diagnosis,” arXiv preprint arXiv:2407.13301, 2024. find
  7. V. L. Patel, J. F. Arocha, and J. Zhang, “Thinking and reasoning in medicine,” The Cambridge handbook of thinking and reasoning, vol. 14, pp. 727–750, 2005. find
  8. S. Xu, Y. Zhou, Z. Liu, Z. Wu, T. Zhong, H. Zhao, Y. Li, H. Jiang, Y. Pan, J. Chen et al., “Towards next-generation medical agent: How o1 is reshaping decision-making in medical scenarios,” arXiv preprint arXiv:2411.14461, 2024. — [arXiv:2411.14461](https://arxiv.org/abs/2411.14461) find
  9. M.-H. Temsah, A. Jamal, K. Alhasan, A. A. Temsah, and K. H. Malki, “Openai o1-preview vs. chatgpt in healthcare: A new frontier in medical ai reasoning,” Cureus, vol. 16, no. 10, p. e70640, 2024. find
  10. Y. Xie, J. Wu, H. Tu, S. Yang, B. Zhao, Y. Zong, Q. Jin, C. Xie, and Y. Zhou, “A preliminary study of o1 in medicine: Are we closer to an ai doctor?” arXiv preprint arXiv:2409.15277, 2024. — [arXiv:2409.15277](https://arxiv.org/abs/2409.15277) find
  11. J. Chen, X. Wang, K. Ji, A. Gao, F. Jiang, S. Chen, H. Zhang, D. Song, W. Xie, C. Kong et al., “Huatuogpt-ii, one-stage training for medical adaption of llms,” arXiv preprint arXiv:2311.09774, 2023. — [arXiv:2311.09774](https://arxiv.org/abs/2311.09774) find
  12. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 24 824–24 837, 2022.
  13. H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, J. Wei et al., “Scaling instruction-finetuned language models,” arXiv preprint arXiv:2210.11416, 2022. find
  14. E. Zelikman, Y. Wu, J. Mu, and N. D. Goodman, “Star: Bootstrapping reasoning with reasoning,” Advances in Neural Information Processing Systems, vol. 35, pp. 15 476–15 488, 2022.
  15. T. Chu, Y. Zhai, J. Yang, S. Tong, S. Xie, D. Schuurmans, Q. V. Le, S. Levine, and Y. Ma, “Sft memorizes, rl generalizes: A comparative study of foundation model post-training,” arXiv preprint arXiv:2501.17161, 2025.
  16. H. Wang, C. Qu, Z. Huang, W. Chu, F. Lin, and W. Chen, “Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning,” arXiv preprint arXiv:2504.08837, 2025. find
  17. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,” Advances in Neural Information Processing Systems, vol. 35, pp. 27 730–27 744, 2022.
  18. H. Wang, L. Li, C. Qu, F. Zhu, W. Xu, W. Chu, and F. Lin, “Learning autonomous code integration for math language models,” arXiv preprint arXiv:2502.00691, 2025. find
  19. R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” Advances in Neural Information Processing Systems, vol. 36, 2023.
  20. A. Ura, “Openbiollm-70b: Advancing open-source biomedical llms with direct preference optimization,” Hugging Face Blog, 2024, available at https://huggingface.co/blog/aaditya/openbiollm. ↗
  21. D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948, 2025.
  22. H. Zeng, D. Jiang, H. Wang, P. Nie, X. Chen, and W. Chen, “Acecoder: Acing coder rl via automated test-case synthesis,” arXiv preprint arXiv:2502.01718, 2025. find
  23. J. Pan, C. Liu, J. Wu, F. Liu, J. Zhu, H. B. Li, C. Chen, C. Ouyang, and D. Rueckert, “Medvlm-r1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning,” arXiv preprint arXiv:2502.19634, 2025. — [arXiv:2502.19634](https://arxiv.org/abs/2502.19634) find
  24. H. Zhang, J. Chen, F. Jiang, F. Yu, Z. Chen, J. Li, G. Chen, X. Wu, Z. Zhang, Q. Xiao et al., “Huatuogpt, towards taming language model to be a doctor,” arXiv preprint arXiv:2305.15075, 2023.
  25. Y. Labrak, A. Bazoge, E. Morin, P.-A. Gourraud, M. Rouvier, and R. Dufour, “Biomistral: A collection of open-source pretrained large language models for medical domains,” arXiv preprint arXiv:2402.10373, 2024.
  26. Z. Ke, Y. Shao, H. Lin, T. Konishi, G. Kim, and B. Liu, “Continual pre-training of language models,” arXiv preprint arXiv:2302.03241, 2023. find
  27. K. Zhang, S. Zeng, E. Hua, N. Ding, Z.-R. Chen, Z. Ma, H. Li, G. Cui, B. Qi, X. Zhu et al., “Ultramedical: Building specialized generalists in biomedicine,” Advances in Neural Information Processing Systems, vol. 37, pp. 26 045–26 081, 2024. find
  28. J. Chen, Z. Cai, K. Ji, X. Wang, W. Liu, R. Wang, J. Hou, and B. Wang, “Huatuogpt-o1: Towards medical complex reasoning with llms,” arXiv preprint arXiv:2412.18925, 2024. find
  29. X. Huang, J. Wu, H. Liu, X. Tang, and Y. Zhou, “m1: Unleash the potential of test-time scaling for medical reasoning with large language models,” arXiv preprint arXiv:2504.00869, 2025. — [arXiv:2504.00869](https://arxiv.org/abs/2504.00869) find
  30. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017. find
  31. G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu, “Hybridflow: A flexible and efficient rlhf framework,” arXiv preprint arXiv:2409.19256, 2024.
  32. D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, and P. Szolovits, “What disease does this patient have? a large-scale open domain question answering dataset from medical exams,” Applied Sciences, vol. 11, no. 14, p. 6421, 2021.
  33. A. Pal, L. K. Umapathi, and M. Sankarasubbu, “Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering,” in Conference on health, inference, and learning. PMLR, 2022, pp. 248–260. — MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering
  34. Q. Jin, B. Dhingra, Z. Liu, W. W. Cohen, and X. Lu, “Pubmedqa: A dataset for biomedical research question answering,” arXiv preprint arXiv:1909.06146, 2019.
  35. Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang et al., “Mmlu-pro: A more robust and challenging multi-task language understanding benchmark,” in The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
  36. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman, “Gpqa: A graduate-level google-proof q&a benchmark,” in First Conference on Language Modeling, 2024.
  37. Y. Zuo, S. Qu, Y. Li, Z. Chen, X. Zhu, E. Hua, K. Zhang, N. Ding, and B. Zhou, “Medxpertqa: Benchmarking expert-level medical reasoning and understanding,” arXiv preprint arXiv:2501.18362, 2025. find
  38. X. Tang, D. Shao, J. Sohn, J. Chen, J. Zhang, J. Xiang, F. Wu, Y. Zhao, C. Wu, W. Shi et al., “Medagents-bench: Benchmarking thinking models and agent frameworks for complex medical reasoning,” arXiv preprint arXiv:2503.07459, 2025. — [arXiv:2503.07459](https://arxiv.org/abs/2503.07459) find
  39. Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang et al., “Mmlu-pro: A more robust and challenging multi-task language understanding benchmark,” arXiv preprint arXiv:2406.01574, 2024.
  40. M. S. A. Pal and M. Sankarasubbu, “Openbiollms: Advancing open-source large language models for healthcare and life sciences,” 2024. find
  41. P. Qiu, C. Wu, X. Zhang, W. Lin, H. Wang, Y. Zhang, Y. Wang, and W. Xie, “Towards building multilingual language model for medicine,” Nature Communications, vol. 15, no. 1, p. 8384, 2024.
  42. C. Christophe, P. K. Kanithi, P. Munjal, T. Raha, N. Hayat, R. Rajan, A. Al-Mahrooqi, A. Gupta, M. U. Salman, G. Gosal et al., “Med42–evaluating fine-tuning strategies for medical llms: Full-parameter vs. parameter-efficient approaches,” arXiv preprint arXiv:2404.14779, 2024. — [arXiv:2404.14779](https://arxiv.org/abs/2404.14779) find
  43. D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, and P. Szolovits, “What disease does this patient have? a large-scale open domain question answering dataset from medical exams,” Applied Sciences, vol. 11, no. 14, p. 6421, 2021.
  44. A. Pal, L. K. Umapathi, and M. Sankarasubbu, “Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering,” in Conference on Health, Inference, and Learning. PMLR, 2022, pp. 248–260. — MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering
  45. A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024.
  46. J. Chen, Z. Cai, K. Ji, X. Wang, W. Liu, R. Wang, J. Hou, and B. Wang, “Huatuogpt-o1, towards medical complex reasoning with llms,” arXiv preprint arXiv:2412.18925, 2024. find
  47. A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al., “Gpt-4o system card,” arXiv preprint arXiv:2410.21276, 2024.
  48. Q. Team, “Qwq: Reflect deeply on the boundaries of the unknown,” November 2024. [Online]. Available: https://qwenlm.github.io/blog/qwq-32b-preview/ ↗
  49. Anthropic, “The claude 3 model family: Opus, sonnet, haiku,” https://www-cdn.anthropic.com/ de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf, 2024. ↗
  50. A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseekv3 technical report,” arXiv preprint arXiv:2412.19437, 2024. find
  51. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman, “Gpqa: A graduate-level google-proof q&a benchmark,” arXiv preprint arXiv:2311.12022, 2023.
  52. Q. Team, “Qwen2.5: A party of foundation models,” September 2024. [Online]. Available: https://qwenlm.github.io/blog/qwen2.5/Limitations and Future Work ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2025-distillation-pushing-limits-medical-llm/