Elamur Research⌕
☰
home / papers / 2025 / DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research

DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research

Paper · 2025 · Rulin Shao, Akari Asai, Shannon Zejiang Shen et al.

Summary

Open deep research (DR) agents are usually trained with reinforcement learning on short questions whose answers can be checked automatically. That kind of reward does not transfer to open-ended, long-form reports, where quality is loosely defined and checking claims requires current external evidence.

The authors propose RLER, which uses per-question rubrics as rewards and updates them during training. Each prompt starts with persistent rubrics derived from web search results. At every step, a rubric LM inspects the current rollouts and their search traces. It then adds positive criteria, which capture newly discovered useful knowledge, and negative ones, which target reward hacking such as verbatim copying of retrieved text. A buffer keeps only the rubrics whose scores vary most across rollouts.

Starting from Qwen3-8B, the authors run SFT on 16K trajectories, then RL with RLER plus format, search and citation rewards. The result, DR Tulu-8B, beats open agents and rivals proprietary systems on four long-form benchmarks at a tiny fraction of the cost. The model, data, rubrics, code and dr-agent-lib are all released.

Key points

  • Benchmarks and baselines: evaluation covers SQAv2 (AstaBench-ScholarQA-CS2), HealthBench, ResearchQA and DeepResearchBench.
    • DR Tulu-8B averages 15.6% above Tongyi DR and 0.7% above OpenAI DR.
    • Its margin over open 8–32B models ranges from 4.8 to 41.8 points.
  • Cost: on SQAv2, one query costs about USD 0.0019, compared with roughly USD 1.8 for OpenAI DR (about 1000× cheaper).
  • Effect of RLER: it adds 6.4–16.0 points over the SFT-only model. Substituting Qwen3-8B as judge and rubric generator still yields a gain of more than 4 points, about 1 point below the GPT-4.1 setup.
  • Tool use: the model learns to choose tools per task. It uses paper search about 90% of the time on SQAv2 and web search/browsing about 55% of the time on DeepResearchBench.
  • New dataset: GeneticDiseasesQA is a clinical benchmark on whether genetic variants qualify for therapy. DR Tulu-8B is competitive with proprietary agents on it, while other open agents cannot handle the task.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

Cite

Cite as Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G. Finlayson, David Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca Soldaini, Faeze Brahman, Wen-tau Yih, Tongshuang Wu, Luke Zettlemoyer, Yoon Kim, Hannaneh Hajishirzi and Pang Wei Koh. 2025. DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research. arXiv:2511.19399.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2511.19399 · arXiv 2511.19399

@misc{shao2025dr,
  title = {{DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research}},
  author = {Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G. Finlayson, David Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca Soldaini, Faeze Brahman, Wen-tau Yih, Tongshuang Wu, Luke Zettlemoyer, Yoon Kim, Hannaneh Hajishirzi and Pang Wei Koh},
  year = {2025},
  doi = {10.48550/arxiv.2511.19399},
  eprint = {2511.19399},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2511.19399}
}

References

46 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fan, R.-Z., Wang, Z., and Liu, P. Megascience: Pushing the frontiers of post-training datasets for science reasoning. arXiv preprint arXiv:2507.16812, 2025. — [arXiv:2507.16812](https://arxiv.org/abs/2507.16812) find
  2. Arora, R. K., Wei, J., Hicks, R. S., Bowman, P., Quiñonero-Candela, J., Tsimpourlas, F., Sharman, M., Shah, M., Vallone, A., Beutel, A., et al. Healthbench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775, 2025.
  3. Gao, J., Fu, W., Xie, M., Xu, S., He, C., Mei, Z., Zhu, B., and Wu, Y. Beyond ten turns: Unlocking long-horizon agentic search with large-scale asynchronous rl. arXiv preprint arXiv:2508.07976, 2025. — [arXiv:2508.07976](https://arxiv.org/abs/2508.07976) find
  4. Asai, A., He, J., Shao, R., Shi, W., Singh, A., Chang, J. C., Lo, K., Soldaini, L., Feldman, S., D’arcy, M., et al. Openscholar: Synthesizing scientific literature with retrieval-augmented lms. arXiv preprint arXiv:2411.14199, 2024.
  5. Gunjal, A., Wang, A., Lau, E., Nath, V., He, Y., Liu, B., and Hendryx, S. Rubrics as rewards: Reinforcement learning beyond verifiable domains. arXiv preprint arXiv:2507.17746, 2025.
  6. Bragg, J., D’Arcy, M., Balepur, N., Bareket, D., Dalvi, B., Feldman, S., Haddad, D., Hwang, J. D., Jansen, P., Kishore, V., et al. Astabench: Rigorous benchmarking of ai agents with a scientific research suite. arXiv preprint arXiv:2510.21652, 2025. find
  7. Guo, J., Chi, Z., Dong, L., Dong, Q., Wu, X., Huang, S., and Wei, F. Reward reasoning model. arXiv preprint arXiv:2505.14674, 2025. — [arXiv:2505.14674](https://arxiv.org/abs/2505.14674) find
  8. Cao, T., Bhandari, N., Yerukola, A., Asai, A., and Sap, M. Out of style: Rag’s fragility to linguistic variation. arXiv preprint arXiv:2504.08231, 2025. find
  9. Ho, X., Duong Nguyen, A.-K., Sugawara, S., and Aizawa, A. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Scott, D., Bel, N., and Zong, C. (eds.), Proceedings of the 28th International Conference on Computational Linguistics, pp. 6609–6625, Barcelona, Spain (O… — [doi:10.18653/v1/2020.coling-main](https://doi.org/10.18653/v1/2020.coling-main)
  10. Liu, J., Li, Y., Zhang, C., Li, J., Chen, A., Ji, K., Cheng, W., Wu, Z., Du, C., Xu, Q., et al. Webexplorer: Explore and evolve for training long-horizon web agents. arXiv preprint arXiv:2509.06501, 2025. — [arXiv:2509.06501](https://arxiv.org/abs/2509.06501) find
  11. Liu, N., Zhang, T., and Liang, P. Evaluating verifiability in generative search engines. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7001–7025, Singapore, December 2023a. Association for Computational Linguistics. doi: 10.…
  12. Jayalath, D., Goel, S., Foster, T., Jain, P., Gururangan, S., Zhang, C., Goyal, A., and Schelten, A. Compute as teacher: Turning inference compute into reference-free supervision. arXiv preprint arXiv:2509.14234, 2025. — [arXiv:2509.14234](https://arxiv.org/abs/2509.14234) find
  13. Jiang, D., Lu, Y., Li, Z., Lyu, Z., Nie, P., Wang, H., Su, A., Chen, H., Zou, K., Du, C., et al. Verltool: Towards holistic agentic reinforcement learning with tool use. arXiv preprint arXiv:2509.01055, 2025. — [arXiv:2509.01055](https://arxiv.org/abs/2509.01055) find
  14. Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634, 2023b.
  15. Jin, B., Zeng, H., Yue, Z., Yoon, J., Arik, S., Wang, D., Zamani, H., and Han, J. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. In COLM, 2025.
  16. Mallen, A., Asai, A., Zhong, V., Das, R., Hajishirzi, H., and Khashabi, D. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. arXiv preprint, 2022. find
  17. Krishna, K., Roy, A., and Iyyer, M. Hurdles to progress in long-form question answering. In Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y. (eds.), Proceedings of the 2021 Conference of the North American Chapter…
  18. Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., and Hajishirzi, H. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for C… find
  19. Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N., Lyu, X., Gu, Y., Malik, S., Graf, V., Hwang, J. D., Yang, J., Bras, R. L., Tafjord, O., Wilhelm, C., Soldaini, L., Smith, N. A., Wang, Y., Dasigi, P., and Hajishirzi, H. Tulu 3: Pushing… — Tulu 3: Pushing Frontiers in Open Language Model Post-Training
  20. Miroyan, M., Wu, T.-H., King, L., Li, T., Pan, J., Hu, X., Chiang, W.-L., Angelopoulos, A. N., Darrell, T., Norouzi, N., et al. Search arena: Analyzing search-augmented llms. arXiv preprint arXiv:2506.05334, 2025. find
  21. Nguyen, X.-P., Pandit, S., Reddy, R. G., Xu, A., Savarese, S., Xiong, C., and Joty, S. Sfr-deepresearch: Towards effective reinforcement learning for autonomously reasoning single agents. arXiv preprint arXiv:2509.06283, 2025. — [arXiv:2509.06283](https://arxiv.org/abs/2509.06283) find
  22. Li, H., Dong, Q., Chen, J., Su, H., Zhou, Y., Ai, Q., Ye, Z., and Liu, Y. Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579, 2024.
  23. Li, X., Jin, J., Dong, G., Qian, H., Zhu, Y., Wu, Y., Wen, J.-R., and Dou, Z. Webthinker: Empowering large reasoning models with deep research capability. arXiv preprint arXiv:2504.21776, 2025a.
  24. Noukhovitch, M., Huang, S., Xhonneux, S., Hosseini, A., Agarwal, R., and Courville, A. Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models, October 2024. URL http://arxiv.org/ abs/2410.18252. — [arXiv:2410.18252](https://arxiv.org/abs/2410.18252) ↗
  25. Li, Z., Guan, X., Zhang, B., Huang, S., Zhou, H., Lai, S., Yan, M., Jiang, Y., Xie, P., Huang, F., et al. Webweaver: Structuring web-scale evidence with dynamic OpenAI. Deep research system card, 2025. URL https://openai.com/index/deepresearch-system-card/. Accessed: 2025-10-21. ↗
  26. Prabhakar, A., Ram, R., Chen, Z., Savarese, S., Wang, F., Xiong, C., Wang, H., and Yao, W. Enterprise deep research: Steerable multi-agent deep research for enterprise analytics. arXiv preprint arXiv:2510.17797, 2025. — [arXiv:2510.17797](https://arxiv.org/abs/2510.17797) find
  27. Wei, J., Karina, N., Chung, H. W., Jiao, Y. J., Papay, S., Glaese, A., Schulman, J., and Fedus, W. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368, 2024.
  28. Rezaei, M., Vacareanu, R., Wang, Z., Wang, C., He, Y., and Akyürek, A. F. Online rubrics elicitation from pairwise comparisons. arXiv preprint arXiv:2510.07284, 2025. — [arXiv:2510.07284](https://arxiv.org/abs/2510.07284) find
  29. Wei, J., Sun, Z., Papay, S., McKinney, S., Han, J., Fulford, I., Chung, H. W., Passos, A. T., Fedus, W., and Glaese, A. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516, 2025.
  30. Shao, R., Li, S. S., Xin, R., Geng, S., Wang, Y., Oh, S., Du, S. S., Lambert, N., Min, S., Krishna, R., et al. Spurious rewards: Rethinking training signals in rlvr. arXiv preprint arXiv:2506.10947, 2025.
  31. Wu, J., Yin, W., Jiang, Y., Wang, Z., Xi, Z., Fang, R., Zhang, L., He, Y., Zhou, D., Xie, P., et al. Webwalker: Benchmarking llms in web traversal. arXiv preprint arXiv:2501.07572, 2025a.
  32. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
  33. Wu, M., Zhang, G., Min, S., Levine, S., and Kumar, A. Rlac: Reinforcement learning with adversarial critic for free-form generation tasks. arXiv preprint arXiv:2511.01758, 2025b. — [arXiv:2511.01758](https://arxiv.org/abs/2511.01758) find
  34. Sharma, M., Zhang, C. B. C., Bandi, C., Wang, C., Aich, A., Nghiem, H., Rabbani, T., Htet, Y., Jang, B., Basu, S., et al. Researchrubrics: A benchmark of prompts and rubrics for evaluating deep research agents. arXiv preprint arXiv:2511.07685, 2025. — [arXiv:2511.07685](https://arxiv.org/abs/2511.07685) find
  35. Xu, F., Song, Y., Iyyer, M., and Choi, E. A critical evaluation of evaluations for long-form question answering. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3225–3245, Tor…
  36. Shi, D., Cao, J., Chen, Q., Sun, W., Li, W., Lu, H., Dong, F., Qin, T., Zhu, K., Liu, M., et al. Taskcraft: Automated generation of agentic tasks. arXiv preprint arXiv:2506.10055, 2025. — [arXiv:2506.10055](https://arxiv.org/abs/2506.10055) find
  37. Singh, A., Chang, J. C., Haddad, D., Naik, A., Hwang, J. D., Kinney, R., Weld, D. S., Downey, D., and Feldman, S. Ai2 scholar QA: Organized literature synthesis with attribution. In Mishra, P., Muresan, S., and Yu, T. (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computationa… find
  38. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
  39. Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., and Manning, C. D. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural L…
  40. Team, T. D., Li, B., Zhang, B., Zhang, D., Huang, F., Li, G., Chen, G., Yin, H., Wu, J., Zhou, J., et al. Tongyi deepresearch technical report. arXiv preprint arXiv:2510.24701, 2025. — [arXiv:2510.24701](https://arxiv.org/abs/2510.24701) find
  41. Yifei, L. S., Chang, A., Malaviya, C., and Yatskar, M. Researchqa: Evaluating scholarly question answering at scale across 75 fields with survey-mined questions and rubrics. arXiv preprint arXiv:2509.00496, 2025. find
  42. Viswanathan, V., Sun, Y., Ma, S., Kong, X., Cao, M., Neubig, G., and Wu, T. Checklists are better than reward models for aligning language models. arXiv preprint arXiv:2507.18624, 2025.
  43. Yu, Q., Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., Dai, W., Fan, T., Liu, G., Liu, L., Liu, X., Lin, H., Lin, Z., Ma, B., Sheng, G., Tong, Y., Zhang, C., Zhang, M., Zhang, W., Zhu, H., Zhu, J., Chen, J., Chen, J., Wang, C., Yu, H., Song, Y., Wei, X., Zhou, H., Liu, J., Ma, W.-Y., Zhang, Y.-Q.,… — [arXiv:2504.15219](https://arxiv.org/abs/2504.15219) find
  44. M. Dapo: An open-source llm reinforcement learning system at scale, 2025. URL https://arxiv.org/ abs/2503.14476. ↗
  45. Zeng, Z., Yu, J., Gao, T., Meng, Y., Goyal, T., and Chen, D. Evaluating large language models at evaluating instruction following. In International Conference on Learning Representations (ICLR), 2024. find
  46. Zeng, Z., Ivison, H., Wang, Y., Yuan, L., Li, S. S., Ye, Z., Li, S., He, J., Zhou, R., Chen, T., Zhao, C., Tsvetkov, Y., Du, S. S., Jaques, N., Peng, H., Koh, P. W., and Hajishirzi, H. Rlve: Scaling up reinforcement learning for language models with adaptive verifiable environments. arXiv preprint 2… — [arXiv:2511.07317](https://arxiv.org/abs/2511.07317) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shao-2025-dr-tulu-reinforcement-learning-evolving/