Elamur Research⌕
☰
home / papers / 2025 / Training AI Co-Scientists Using Rubric Rewards

Training AI Co-Scientists Using Rubric Rewards

Paper · 2025 · Shashwat Goel, Rishi Hazra, Dulhan Jayalath et al.

Summary

Language models struggle to draft research plans for open-ended scientific goals. They often miss explicit constraints and unstated requirements, and validating a plan through real experiments is slow, costly, or, as in medicine, impossible. The authors turn published papers into a training signal. Llama-4-Maverick extracts candidate research goals, goal-specific rubric items and reference solutions from each paper, and Claude-4-Sonnet selects the best candidate per paper. A policy is then trained with reinforcement learning (GRPO). Its reward comes from a frozen copy of the starting model, which grades plans using the rubrics as privileged information plus seven general guidelines. This removes the need for human labels or a stronger external judge. In a 225-hour study, ML experts preferred plans from the fine-tuned Qwen3-30B-A3B over the base model for 70% of goals. Frontier-model juries measured 12–22% relative gains on medical and recent arXiv goals, along with cross-domain transfer. The result points to a cheap, general recipe for co-scientist models where execution feedback is unavailable.

Key points

  • Released dataset: ResearchPlanGen (huggingface.co/datasets/facebook/research-plan-gen). It is built from NeurIPS/ICLR ML papers, PubMed medical papers and arXiv preprints from Aug–Sept 2025.
  • Rubric construction and scoring:
    • Each goal starts with 15 draft rubric items, filtered down to 10.
    • A plan's score is the share of items for which the grader reports no violated guideline.
    • Experts judged 84% of the extracted rubric items to be valid.
  • Experts preferred the fine-tuned model's plans for 70% of goals (p < 0.01). Ablations show that both the goal-specific rubrics and the general guidelines are necessary for the gains.
  • After training, the 30B model performs on par with Grok-4-Thinking but still falls short of GPT-5-Thinking.
  • Limitation: trained plans became more complicated. The authors attribute this to the self-grader judging cost and effort efficiency poorly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

Cite

Cite as Shashwat Goel, Rishi Hazra, Dulhan Jayalath, Timon Willi, Parag Jain, William F. Shen, Ilias Leontiadis, Francesco Barbieri, Yoram Bachrach, Jonas Geiping and Chenxi Whitehouse. 2025. Training AI Co-Scientists Using Rubric Rewards. arXiv:2512.23707.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2512.23707 · arXiv 2512.23707

@misc{goel2025training,
  title = {{Training AI Co-Scientists Using Rubric Rewards}},
  author = {Shashwat Goel, Rishi Hazra, Dulhan Jayalath, Timon Willi, Parag Jain, William F. Shen, Ilias Leontiadis, Francesco Barbieri, Yoram Bachrach, Jonas Geiping and Chenxi Whitehouse},
  year = {2025},
  doi = {10.48550/arxiv.2512.23707},
  eprint = {2512.23707},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2512.23707}
}

References

47 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dhruv Agarwal, Bodhisattwa Prasad Majumder, Reece Adamson, Megha Chakravorty, Satvika Reddy Gavireddy, Aditya Parashar, Harshit Surana, Bhavana Dalvi Mishra, Andrew McCallum, Ashish Sabharwal, et al. Open-ended Scientific Discovery via Bayesian Surprise. arXiv preprint arXiv:2507.00310, 2025. https:… — [arXiv:2507.00310](https://arxiv.org/abs/2507.00310) find
  2. Anthropic. System Card: Claude Opus 4 & Claude Sonnet 4. Technical report, Anthropic, May 2025. https://www-cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad13df32ed995.pdf. Accessed: 2025-12-14. ↗
  3. Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. HealthBench: Evaluating Large Language Models Towards Improved Human Health, 2025. https://…
  4. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora De… — On the Opportunities and Risks of Foundation Models
  5. Jacob Cohen. A Coefficient of Agreement for Nominal Scales. Educational and psychological measurement, 20(1):37–46, 1960. find
  6. Snezana Djurisic, Ana Rath, Sabrina Gaber, Silvio Garattini, Vittorio Bertele, Sandra-Nadia Ngwabyt, Virginie Hivert, Edmund AM Neugebauer, Martine Laville, Michael Hiesmayr, et al. Barriers to the Conduct of Randomised Clinical Trials within All Disease Areas. Trials, 18(1):360, 2017. find
  7. Bradley Efron and Robert J. Tibshirani. An Introduction to the Bootstrap. Chapman and Hall/CRC, New York, 1994. ISBN 978-0412042317. find
  8. Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023. https://proceedings.mlr.press/v202/gao23h.html. ↗
  9. Gemini Team. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities, 2025. https://arxiv.org/abs/2507.06261. ↗
  10. Gemma Team. Gemma 3 Technical Report. arXiv preprint arXiv:2503.19786, 2025. find
  11. Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al. Towards an AI Co-scientist. arXiv preprint arXiv:2502.18864, 2025. https://arxiv.org/abs/2502.18864. ↗
  12. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783, 2024. https://arxiv.org/abs/2407.21783. ↗
  13. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains, 2025. https://arxiv.org/abs/2507.17746. ↗
  14. Tarun Gupta and Danish Pruthi. All That Glitters is Not Novel: Plagiarism in AI Generated Research. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.acl-long.1249. https://aclanthology.org/2025.acl-long. 1249/. ↗
  15. Jeremy Howard and Sebastian Ruder. Universal Language Model Fine-tuning for Text Classification, 2018. https://arxiv.org/abs/1801.06146. ↗
  16. John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with AlphaFold. nature, 596(7873):583–589, 2021.
  17. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025. https://arxiv.org/abs/2507.20534. ↗
  18. Solomon Kullback and Richard A. Leibler. On Information and Sufficiency. The Annals of Mathematical Statistics, 22 (1):79–86, 1951. doi: 10.1214/aoms/1177729694. — [doi:10.1214/aoms/1177729694](https://doi.org/10.1214/aoms/1177729694) find
  19. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. arXiv preprint arXiv:2408.06292, 2024. https://arxiv.org/abs/2408. 06292. ↗
  20. Linyue Ma, Yilong Xu, Xiang Long, and Zhi Zheng. An Efficient Rubric-based Generative Verifier for Search-Augmented LLMs. arXiv preprint arXiv:2510.14660, 2025. https://arxiv.org/abs/2510.14660. — [arXiv:2510.14660](https://arxiv.org/abs/2510.14660) ↗
  21. Shambhavi Mishra, Gaurav Sahu, Marco Pedersoli, Laurent Charlin, Jose Dolz, and Christopher Pal. AInstein: Assessing the Feasibility of AI-Generated Approaches to Research Problems, 2025. https://arxiv.org/abs/2510.05432. — [arXiv:2510.05432](https://arxiv.org/abs/2510.05432) ↗
  22. Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, et al. MLGym: A New Framework and Benchmark for Advancing AI Research Agents. arXiv preprint arXiv:2502.14499, 2025. https://arxiv.org… ↗
  23. Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Ba… — AlphaEvolve: A coding agent for scientific and algorithmic discovery
  24. OpenAI. GPT-5 System Card. — OpenAI GPT-5 System Card https://openai.com/index/gpt-5-system-card/, August 2025. ↗
  25. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training La… — Training language models to follow instructions with human feedback
  26. Robert Root-Bernstein and Michele Root-Bernstein. Artistic Scientists and Scientific Artists: The Link Between Polymathy and Creativity. 2004. https://psycnet.apa.org/record/2004-13113-008. ↗
  27. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024. https://arxiv.org/abs/2402.03300. ↗
  28. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers, 2024. https://arxiv.org/abs/2409.04109. ↗
  29. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. PaperBench: Evaluating AI’s Ability to Replicate AI Research, 2025. https://arxiv.org/abs/2504.0184… ↗
  30. Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and J. Andrew Bagnell. All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning, 2025. https://arxiv.org/abs/2503.01067. — [arXiv:2503.01067](https://arxiv.org/abs/2503.01067) ↗
  31. Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models, 2024. https://arxiv.org/abs/2404.18796. ↗
  32. Jason Weston and Jakob Foerster. AI & Human Co-Improvement for Safer Co-Superintelligence, 2025. https://arxiv.org/abs/2512.05356. — [arXiv:2512.05356](https://arxiv.org/abs/2512.05356) ↗
  33. Chenxi Whitehouse, Sebastian Ruder, Tony Lin, Oksana Kurylo, Haruka Takagi, Janice Lam, Nicolò Busetto, Denise Diaz, and Francisco Guzmán. MENLO: From Preferences to Proficiency–Evaluating and Modeling Native-like Quality Across 47 Languages. arXiv preprint arXiv:2509.26601, 2025. https://arxiv.org/… — [arXiv:2509.26601](https://arxiv.org/abs/2509.26601) ↗
  34. Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, et al. RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human Experts. arXiv preprint arXiv:2411.15114, 2024. https://arxiv.org/abs/2… ↗
  35. Shirley Wu, Michel Galley, Baolin Peng, Hao Cheng, Gavin Li, Yao Dou, Weixin Cai, James Zou, Jure Leskovec, and Jianfeng Gao. CollabLLM: From Passive Responders to Active Collaborators, 2025a. https://arxiv.org/abs/2502. 00640. ↗
  36. Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification, 2025b. https://arxiv.org/abs/2508.05629. xAI. Grok 4 Model Card. Model card, xAI,… — [arXiv:2508.05629](https://arxiv.org/abs/2508.05629) ↗
  37. Tianze Xu, Pengrui Lu, Lyumanshan Ye, Xiangkun Hu, and Pengfei Liu. ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific Inquiry, 2025. https://arxiv.org/abs/2507.16280. — [arXiv:2507.16280](https://arxiv.org/abs/2507.16280) ↗
  38. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… — Qwen3 Technical Report
  39. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge. arXiv preprint arXiv:2410.02736, 2024. https://arxiv.org/abs/2410.02736. ↗
  40. Zhiling Ye, Yun Yue, Haowen Wang, Xudong Han, Jiadi Jiang, Cheng Wei, Lei Fan, Jiaxin Liang, Shuowen Zhang, Ji Li, Chunxiao Guo, Jian Wang, Peng Wei, and Jinjie Gu. Self-Rewarding Rubric-Based Reinforcement Learning for Open-Ended Reasoning, 2025. https://arxiv.org/abs/2509.25534. — [arXiv:2509.25534](https://arxiv.org/abs/2509.25534) ↗
  41. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How Transferable are Features in Deep Neural Networks?, 2014. https://arxiv.org/abs/1411.1792. — [arXiv:1411.1792](https://arxiv.org/abs/1411.1792) ↗
  42. Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Ilia Kulikov, Kyunghyun Cho, Dong Wang, Yuandong Tian, Jason E Weston, et al. NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions. arXiv preprint arXiv:2502.13124, 2025. https://arxiv.org/abs/2502.13124. ↗
  43. Xiang Yue, Tianyu Zheng, Ge Zhang, and Wenhu Chen. MAmmoTH2: Scaling Instructions from the Web. Advances in Neural Information Processing Systems, 37:90629–90660, 2024. https://proceedings.neurips.cc/paper_files/paper/ 2024/hash/a4ca07aa108036f80cbb5b82285fd4b1-Abstract-Conference.html. ↗
  44. Kang Zhang, Hong-Yu Zhou, Daniel T Baptista-Hon, Yuanxu Gao, Xiaohong Liu, Eric Oermann, Sheng Xu, Shengwei Jin, Jian Zhang, Zhuo Sun, et al. Concepts and Applications of Digital Twins in Healthcare and Medicine. Patterns, 5(8), 2024. find
  45. Yilun Zhao, Kaiyan Zhang, Tiansheng Hu, Sihong Wu, Ronan Le Bras, Taira Anderson, Jonathan Bragg, Joseph Chee Chang, Jesse Dodge, Matt Latzke, et al. SciArena: An Open Evaluation Platform for Foundation Models in Scientific Literature Tasks. arXiv preprint arXiv:2507.01001, 2025. https://arxiv.org/a… ↗
  46. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less Is More for Alignment, 2023. https://arxiv.org/abs/2305.11206. ↗
  47. Jin Peng Zhou, Séb Arnold, Nan Ding, Kilian Q Weinberger, Nan Hua, and Fei Sha. Graders Should Cheat: Privileged Information Enables Expert-Level Automated Evaluations. ACL, 2025. doi: 10.18653/v1/2025.emnlp-main.838. https://aclanthology.org/2025.emnlp-main.838/. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/goel-2025-training-ai-co-scientists/