Elamur Research⌕
☰
home / papers / 2026 / Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation

Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation

Paper · 2026 · Srujan P Mule, Aniketh Garikaparthi and Manasi Patwardhan

Summary

AI research agents can produce hundreds of candidate ideas, but testing each one experimentally is impractical. Current filters judge ideas by subjective qualities such as novelty, which may not predict whether a method actually works. The authors treat the task as forecasting which of two ideas wins empirically: given a research objective tied to a particular benchmark and a pair of candidate methods, the model must predict which one will achieve the higher score. They scraped NLP leaderboards from PapersWithCode, traced each entry back to the paper that originally proposed the method, and built 11,488 labelled idea pairs covering 724 benchmarks.

They fine-tuned 8B models in two ways:

  • Direct SFT: outputs only the winner label and reaches 77.1% accuracy, well above GPT-5 at 61.1%.
  • Reasoning variants: SFT on reasoning traces followed by RL with verifiable rewards (DAPO, DrGRPO) reaches about 71% while producing readable justifications.

The models stay largely stable under paraphrasing and changes in idea order, and they transfer to an external test set. This suggests small, cheap models could act as objective verifiers or reward models in automated science pipelines.

Key points

  • Dataset construction:
    • The pipeline began with 1,918 NLP leaderboards and 5,713 result-reporting papers.
    • Gemini-2.5-pro identified the papers that originally introduced each method; experts manually checked the low-confidence cases. This added 908 original papers.
    • Papers were converted to Markdown with s2orc-doc2json.
  • Accuracy:
    • Untuned Qwen3-8B reaches only 20.13%; the abstract cites about 30% for off-the-shelf 8B models generally.
    • Direct SFT reaches 77.1%, and the best reasoning model reaches 71.35%, compared with 61.10% for GPT-5.
  • Effect of RL on reasoning models:
    • SFT on reasoning traces alone performs poorly: 37.51% with paper-grounded traces and 25.54% with synthetic traces.
    • RL recovers accuracy to about 71%.
    • Some RL variants showed degenerate behaviour, such as skipping reasoning entirely or giving shallow "more recent" arguments. A minimum-length penalty was added to counter this.
  • Robustness: Paraphrasing caused only small changes in accuracy. Trained models gave the same answer regardless of idea order in over 85% of pairs. A recency bias appeared on the easiest subset.
  • Generalisation and limitations:
    • The models were evaluated zero-shot on Wen et al.'s (2025) independent test set and on a cross-domain time-split set.
    • The training data covers NLP only and depends on PapersWithCode, which has since shut down.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Tata Consultancy ServicesPune · India8 works · 35 citations

Also on the byline: Indian Institute of Science Education and Research Pune.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Srujan P Mule, Aniketh Garikaparthi and Manasi Patwardhan. 2026. Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1918

@inproceedings{mule2026teaching,
  title = {{Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation}},
  author = {Srujan P Mule, Aniketh Garikaparthi and Manasi Patwardhan},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1918},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1918}
}

References

48 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI@Meta. 2024. The llama 3 herd of models. ArXiv, abs/2407.21783.
  2. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in ai safety. Preprint, arXiv:1606.06565.
  3. Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy. 2025. Chain-of-thought reasoning in the wild is not always faithful. Preprint, arXiv:2503.08679. find
  4. Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. 2025. Researchagent: Iterative research idea generation over scientific literature with large language models. Preprint, arXiv:2404.07738. https://hyper.ai/en/sota/category/natural-language-processing ↗
  5. Nikhil Chandak, Shashwat Goel, Ameya Prabhu, Moritz Hardt, and Jonas Geiping. 2025. Scaling open-ended reasoning to predict the future. Preprint, arXiv:2512.25070. — [arXiv:2512.25070](https://arxiv.org/abs/2512.25070) find
  6. Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru WANG, Yu Zhang, Denghui Zhang, Tong Zhang, Hanghang Tong, and Heng Ji. 2026. RM-r1: Reward modeling as reasoning. In The Fourteenth International Conference on Learning Representations.
  7. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  8. Aniketh Garikaparthi, Manasi Patwardhan, Lovekesh Vig, and Arman Cohan. 2025. Iris: Interactive research ideation system for accelerating scientific discovery. Preprint, arXiv:2504.16728. find
  9. Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language… find
  10. Shashwat Goel, Rishi Hazra, Dulhan Jayalath, Timon Willi, Parag Jain, William F. Shen, Ilias Leontiadis, Francesco Barbieri, Yoram Bachrach, Jonas Geiping, and Chenxi Whitehouse. 2025. Training ai co-scientists using rubric rewards. Preprint, arXiv:2512.23707. find
  11. Mourad Gridach, Jay Nanavati, Khaldoun Zine El Abidine, Lenon Mendes, and Christina Mack. 2025. Agentic ai for scientific discovery: A survey of progress, challenges, and future directions. Preprint, arXiv:2503.08979. — [arXiv:2503.08979](https://arxiv.org/abs/2503.08979) find
  12. Xuemei Gu and Mario Krenn. 2025. Forecasting high-impact research topics via machine learning on evolving knowledge graphs. Machine Learning: Science and Technology, 6(2):025041. find
  13. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. 2025. Rubrics as rewards: Reinforcement learning beyond verifiable domains. Preprint, arXiv:2507.17746. find
  14. Alexander Gurung and Mirella Lapata. 2025. Learning to reason for long-form story generation. Preprint, arXiv:2503.22828. — [arXiv:2503.22828](https://arxiv.org/abs/2503.22828) find
  15. Danny Halawi, Fred Zhang, Chen Yueh-Han, and Jacob Steinhardt. 2024. Approaching human-level forecasting with language models. Preprint, arXiv:2402.18563. find
  16. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.
  17. Charles O’Neill, Tirthankar Ghosal, Roberta Răileanu, Mike Walmsley, Thang Bui, Kevin Schawinski, and Ioana Ciucă. 2025. Sparks of science: Hypothesis generation using structured paper data. Preprint, arXiv:2504.12976.
  18. Xiang Hu, Hongyu Fu, Jinge Wang, Yifeng Wang, Zhikun Li, Renjun Xu, Yu Lu, Yaochu Jin, Lili Pan, and Zhenzhong Lan. 2024. Nova: An iterative planning and search approach to enhance novelty and diversity of llm generated ideas. Preprint, arXiv:2410.14255. find
  19. OpenAI. 2025. Gpt-5 system card. Accessed: 2026-01-05. find
  20. Ruipeng Jia, Yunyi Yang, Yongbo Gai, Kai Luo, Shihao Huang, Jianhe Lin, Xiaoxi Jiang, and Guanjun Jiang. 2025. Writing-zero: Bridge the gap between non-verifiable tasks and verifiable rewards. Preprint, arXiv:2506.00103. — [arXiv:2506.00103](https://arxiv.org/abs/2506.00103) find
  21. Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip E. Tetlock. 2025. Forecastbench: A dynamic benchmark of ai forecasting capabilities. Preprint, arXiv:2409.19839. — [arXiv:2409.19839](https://arxiv.org/abs/2409.19839) find
  22. Esther Landhuis. 2016. Scientific literature: Information overload. Nature, 535:457 – 458. find
  23. Sang-Woo Lee, Sohee Yang, Donghyun Kwak, and Noah Y. Siegel. 2025. Advancing event forecasting through massive training of large language models: Challenges, solutions, and broader impacts. Preprint, arXiv:2507.19477. — [arXiv:2507.19477](https://arxiv.org/abs/2507.19477) find
  24. Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, Deli Zhao, Yu Rong, Tian Feng, and Lidong Bing. 2024. Chain of ideas: Revolutionizing research via novel idea development with llm agents. Preprint, arXiv:2410.13185. find
  25. Vardhan Palod, Karthik Valmeekam, Kaya Stechly, and Subbarao Kambhampati. 2025. Performative thinking? the brittle correlation between cot length and problem complexity. Preprint, arXiv:2509.07339. — [arXiv:2509.07339](https://arxiv.org/abs/2509.07339) find
  26. Jungsoo Park, Ethan Mendes, Gabriel Stanovsky, and Alan Ritter. 2025. Look before you leap: Estimating llm benchmark scores from descriptions. Preprint, arXiv:2509.20645. — [arXiv:2509.20645](https://arxiv.org/abs/2509.20645) find
  27. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Andreas Müller, Joel Nothman, Gilles Louppe, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perr… find
  28. Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. 2024. Mutual reasoning makes smaller llms stronger problem-solvers. Preprint, arXiv:2408.06195. find
  29. Soumya Rani Samineni, Durgesh Kalwar, Vardaan Gangal, Siddhant Bhambri, and Subbarao Kambhampati. 2025. Local coherence or global validity? investigating rlvr traces in math domains. Preprint, arXiv:2510.18176. — [arXiv:2510.18176](https://arxiv.org/abs/2510.18176) find
  30. Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. 2025a. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. Preprint, arXiv:2505.24864. find
  31. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
  32. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025b. Understanding r1-zero-like training: A critical perspective. Preprint, arXiv:2503.20783.
  33. Chenglei Si, Tatsunori Hashimoto, and Diyi Yang. 2025. The ideation-execution gap: Execution outcomes of llm-generated versus human research ideas. Preprint, arXiv:2506.20803. find
  34. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. The ai scientist: Towards fully automated open-ended scientific discovery. Preprint, arXiv:2408.06292.
  35. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2024. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. Preprint, arXiv:2409.04109.
  36. Hanane Nour Moussa, Patrick Queiroz Da Silva, Daniel Adu-Ampratwum, Alyson East, Zitong Lu, Nikki Puccetti, Mingyi Xue, Huan Sun, Bodhisattwa Prasad Majumder, and Sachin Kumar. 2025. Scholareval: Research idea evaluation grounded in literature. Preprint, arXiv:2510.16234. — [arXiv:2510.16234](https://arxiv.org/abs/2510.16234) find
  37. Michael D. Skarlinski, Sam Cox, Jon M. Laurent, James D. Braza, Michaela Hinks, Michael J. Hammerling, Manvitha Ponnapati, Samuel G. Rodriques, and Andrew D. White. 2024. Language agents achieve superhuman synthesis of scientific knowledge. Preprint, arXiv:2409.13740. find
  38. Haoyang Su, Renqi Chen, Shixiang Tang, Zhenfei Yin, Xinzhe Zheng, Jinzhe Li, Biqing Qi, Qi Wu, Hui Li, Wanli Ouyang, Philip Torr, Bowen Zhou, and Nanqing Dong. 2025. Many heads are better than one: Improved scientific idea generation by a llm-based multi-agent system. Preprint, arXiv:2410.09403.
  39. Minjun Zhu, Qiujie Xie, Yixuan Weng, Jian Wu, Zhen Lin, Linyi Yang, and Yue Zhang. 2025. Ai scientists fail without strong implementation capability. Preprint, arXiv:2506.01372. find
  40. Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA. find
  41. Xuli Tang, Xin Li, Ying Ding, Min Song, and Yi Bu. 2020. The pace of artificial intelligence innovations: Speed, talent, and trial-and-error. Journal of Informetrics, 14(4):101094. find
  42. Keisuke Ueda, Wataru Hirota, Takuto Asakura, Takahiro Omi, Kosuke Takahashi, Kosuke Arima, and Tatsuya Ishigaki. 2025. Exploring design of multi-agent llm dialogues for research ideation. Preprint, arXiv:2507.08350. — [arXiv:2507.08350](https://arxiv.org/abs/2507.08350) find
  43. Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2024. SciMON: Scientific inspiration machines optimized for novelty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 279–299, Bangkok, Thailand. Association for Computational L… find
  44. Jiaxin Wen, Chenglei Si, Yueh han Chen, He He, and Shi Feng. 2025. Predicting empirical ai research outcomes with language models. Preprint, arXiv:2506.00794. — [arXiv:2506.00794](https://arxiv.org/abs/2506.00794) find
  45. Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. 2025. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. Preprint, arXiv:2504.08066.
  46. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  47. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, and 16 others. 2025. Dapo: An open-source llm reinforcement learning s…
  48. Kevin Zhou, Adam Dejl, Gabriel Freedman, Lihu Chen, Antonio Rago, and Francesca Toni. 2025. Evaluating uncertainty quantification methods in argumentative large language models. Preprint, arXiv:2510.02339. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mule-2026-teaching-language-models-forecast-research/