Elamur Research⌕
☰
home / papers / 2025 / ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning

ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning

Paper · 2025 · Yu Sun, Xingyu Qian, Weiwen Xu et al.

Summary

Reasoning-oriented LLMs do well in math and code, but medical QA depends on broad, accurate domain knowledge and lacks large, carefully checked reasoning data. Earlier sets are small and usually distilled from a single teacher. The authors build ReasonMed from about 195k training questions taken from MedQA, MedMCQA, PubMedQA and medical MMLU subsets. Three models (Qwen2.5-72B, DeepSeek-R1-Distill-Llama-70B and HuatuoGPT-o1-70B) sample chains of thought at several temperatures, producing roughly 1.75M paths. A verifier checks each path for correctness and medical factuality. Questions are then routed by how many paths pass:

  • Easy: the two best-ranked paths are kept.
  • Medium: an Error Refiner based on GPT-4o-mini repairs paths using the verifier's feedback.
  • Hard: GPT-o1 writes new reasoning.

The final set has 370k examples, each pairing detailed reasoning with a short summary. Fine-tuning experiments show that the combined format works best, while summary-only training is cheaper and still competitive. ReasonMed-7B and ReasonMed-14B beat similar-sized and some far larger models. The work offers a reusable recipe for building domain-specific reasoning corpora.

Key points

  • Scale: about 1.75M generated paths, of which 1.29M passed validation, distilled to 370k samples. The authors describe this as the largest open medical reasoning dataset, roughly ten times bigger than prior sets.
  • Training formats compared:
    • CoTMed: reasoning only, about 555 output tokens.
    • ResponseMed: summary only, about 225 tokens.
    • ReasonMed: hybrid, about 626 tokens; it overtakes CoTMed after 3 epochs (overall accuracy 69.63%).
  • ReasonMed-7B scores 82.0% on PubMedQA, exceeding LLaMA3.1-70B by 4.6 points. It also beats the best prior sub-10B model by 4.17%.
  • ReasonMed-14B reaches 72.8% across nine medical benchmarks. This beats Qwen2.5-14B (69.0%) and Qwen2.5-32B (72.6%) and nearly matches LLaMA3.1-70B (72.9%).
  • Cost and validation:
    • The EMD pipeline is about 3.6× cheaper than distilling everything with GPT-o1.
    • Best decoding temperature is 0.5–0.7.
    • A physician review of 100 samples agreed with the automatic scores but noted weak citations and limited clinical detail.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Lanzhou UniversityLanzhou · China5 works · 24 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Yu Sun, Xingyu Qian, Weiwen Xu, Hao Zhang, Chenghao Xiao, Long Li, Deli Zhao, Wenbing Huang, Tingyang Xu, Qifeng Bai and Yu Rong. 2025. ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.1344

@inproceedings{sun2025reasonmed,
  title = {{ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning}},
  author = {Yu Sun, Xingyu Qian, Weiwen Xu, Hao Zhang, Chenghao Xiao, Long Li, Deli Zhao, Wenbing Huang, Tingyang Xu, Qifeng Bai and Yu Rong},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1344},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1344}
}

References

35 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. Large language models for mathematical reasoning: Progresses and challenges. Preprint, arXiv:2402.00157.
  2. Guizhen Chen, Weiwen Xu, Hao Zhang, Hou Pong Chan, Chaoqun Liu, Lidong Bing, Deli Zhao, Anh Tuan Luu, and Yu Rong. 2025. FineReason: Evaluating and improving LLMs’ deliberate reasoning through reflective puzzle solving. In Proceedings of the 63rd Annual Meeting of the Association for Computational L…
  3. Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. 2024. Huatuogpt-o1, towards medical complex reasoning with llms. Preprint, arXiv:2412.18925. find
  4. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  5. Abhishek Dutta and Yen-Che Hsiao. 2024. Adaptive reasoning and acting in medical language agents. arXiv preprint arXiv:2410.10020. — [arXiv:2410.10020](https://arxiv.org/abs/2410.10020) find
  6. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… find
  7. Hongchao Gu, Dexun Li, Kuicai Dong, Hao Zhang, Hang Lv, Hao Wang, Defu Lian, Yong Liu, and Enhong Chen. 2025a. RAPID: Efficient retrieval-augmented long text generation with writing planning and information discovery. In Findings of the Association for Computational Linguistics: ACL 2025, pages 1674… find
  8. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025b. A survey on llm-as-a-judge. Preprint, arXiv:2411.15594. find
  9. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR).
  10. Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, and 1 others. 2023. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 3(4):6.
  11. Dong Huang, Jie M. Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. 2024. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. Preprint, arXiv:2312.13010. find
  12. Liu Huanshuo, Hao Zhang, Zhijiang Guo, Jing Wang, Kuicai Dong, Xiangyang Li, Yi Quan Lee, Cong Zhang, and Yong Liu. 2025. CtrlA: Adaptive retrieval-augmented generation via inherent control. In Findings of the Association for Computational Linguistics: ACL 2025, pages 12592–12618, Vienna, Austria. A… find
  13. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. arXiv preprint arXiv:2009.13081.
  14. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce…
  15. Jack Krolik, Herprit Mahal, Feroz Ahmad, Gaurav Trivedi, and Bahador Saket. 2024. Towards leveraging large language models for automated medical q&a evaluation. Preprint, arXiv:2409.01941. — [arXiv:2409.01941](https://arxiv.org/abs/2409.01941) find
  16. Xiang Li, Duyi Pan, Hongru Xiao, Jiale Han, Jing Tang, Jiabao Ma, Wei Wang, and Bo Cheng. 2025a. Dialogueagents: A hybrid agent-based speech synthesis framework for multi-party dialogue. Preprint, arXiv:2504.14482. — [arXiv:2504.14482](https://arxiv.org/abs/2504.14482) find
  17. Zhaodonghui Li, Haitao Yuan, Jiachen Shi, Hao Zhang, Yu Rong, and Gao Cong. 2025b. Amaze: A multi-agent zero-shot index advisor for relational databases. ArXiv, abs/2508.16044. — [arXiv:2508.16044](https://arxiv.org/abs/2508.16044) find
  18. Hanmeng Liu, Zhizhang Fu, Mengru Ding, Ruoxi Ning, Chaoli Zhang, Xiaozhang Liu, and Yue Zhang. 2025. Logical reasoning in large language models: A survey. Preprint, arXiv:2502.09100.
  19. Valentin Liévin, Christoffer Egeberg Hother, Andreas Geert Motzfeldt, and Ole Winther. 2023. Can large language models reason about medical questions? Preprint, arXiv:2207.08143. optimizer for relational databases? arXiv preprint arXiv:2502.05562. find
  20. Yi-Da Tang, Er-Dan Dong, and Wen Gao. 2024. Llms in medicine: The need for advanced evaluation systems for disruptive technologies. The Innovation, 5(3). find
  21. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  22. Qwen Team. 2025. Qwq-32b: Embracing the power of reinforcement learning. find
  23. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903.
  24. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. Preprint, arXiv:2501.19393.
  25. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
  26. OpenAI, :, Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, and 7 others. 2025. Competitive p… find
  27. Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, and 1 others. 2025. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. arXiv preprint arXiv:2506.07044. — [arXiv:2506.07044](https://arxiv.org/abs/2506.07044) find
  28. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pages 24…
  29. Bo Pang, Hanze Dong, Jiacheng Xu, Silvio Savarese, Yingbo Zhou, and Caiming Xiong. 2025. Bolt: Bootstrap long chain-of-thought in language models without distillation. Preprint, arXiv:2502.03860. — [arXiv:2502.03860](https://arxiv.org/abs/2502.03860) find
  30. Zehan Qi, Rongwu Xu, Zhijiang Guo, Cunxiang Wang, Hao Zhang, and Wei Xu. 2024. long 2 rag: Evaluating long-context & long-form retrieval-augmented generation with key point recall. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4852–4872, Miami, Florida, USA. Associa… find
  31. Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, and Pengfei Liu. 2024. O1 replication journey: A strategic progress report – part 1. Preprint, arXiv:2410.18982. find
  32. Jie Tan, Kangfei Zhao, Rui Li, Jeffrey Xu Yu, Chengzhi Piao, Hong Cheng, Helen Meng, Deli Zhao, and Yu Rong. 2025. Can large language models be query Zhongshen Zeng, Yinhong Liu, Yingjia Wan, Jingyao Li, Pengguang Chen, Jianbo Dai, Yuxuan Yao, Rongwu Xu, Zehan Qi, Wanru Zhao, Linling Shen, Jianqiao… find
  33. T. Zhao, S. Wang, C. Ouyang, M. Chen, C. Liu, J. Zhang, L. Yu, F. Wang, Y. Xie, J. Li, F. Wang, S. Grunwald, B. M. Wong, F. Zhang, Z. Qian, Y. Xu, C. Yu, W. Han, T. Sun, and 32 others. 2024. Artificial intelligence for geoscience: Progress, challenges, and perspectives. Innovation (Camb), 5(5):10069… find
  34. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 400–410, Bangkok, Thail…
  35. Kaiwen Zuo, Yirui Jiang, Fan Mo, and Pietro Lio. 2025. Kg4diagnosis: A hierarchical multi-agent llm framework with knowledge graph enhancement for medical diagnosis. In AAAI Bridge Program on AI for Medicine and Healthcare, pages 195–204. PMLR. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2025-reasonmed-370k-multi-agent-generated/