Elamur Research⌕
☰
home / papers / 2026 / Agentic Rubrics as Contextual Verifiers for SWE Agents

Agentic Rubrics as Contextual Verifiers for SWE Agents

Paper · 2026 · Mohit Raghavendra, Anisha Gunjal, Bing Liu et al.

Summary

Choosing the best patch from many SWE-agent attempts requires a good verifier. Running tests is reliable but costly, since each repository needs its own environment setup. Cheaper methods that skip execution, such as patch classifiers or similarity heuristics, tend to ignore codebase context and are hard to interpret. The authors propose Agentic Rubrics to address this. A rubric agent built on the SWE-Agent scaffold explores the repository together with the issue text and produces a weighted checklist (rubrics.yaml) whose criteria fall into four categories: Runtime, Integrity, Spec Alignment, and File Change. An LLM judge then marks each candidate patch against every criterion as pass or fail, and the weighted results are combined into one score, with no code execution. The rubric is written once per task and reused for all candidates.

The method was tested with Best@16 selection on SWE-Bench Verified. It beat both test-based and execution-free baselines, and its rubric scores agreed with ground-truth tests. The rubrics also caught problems the tests missed, such as unnecessary edits. This makes the approach a scalable and interpretable verification signal, and the authors see it as a possible future reward source for RL.

Key points

  • Best@16 results on 500 SWE-Bench Verified tasks: 54.2% with Qwen3-Coder-30B-A3B rollouts and 40.6% with Qwen3-32B rollouts. The strongest baseline, Patch Classifier, scored 50.2% and 37.1%, so the gain is at least 3.5 points. Oracle Pass@16 reaches 65.6% and 51.4%.
  • Rubric structure: each criterion carries a weight from 1 to 3 (nice-to-have, important, must-have). The final score is the weighted share of satisfied criteria, ranging from 0 to 1.
  • Value of repository access: when Sonnet-4.5 writes rubrics from the issue text alone, Best@16 falls by 4.0 points on Qwen3-32B and by 1.4 points on Qwen3-Coder-30B-A3B.
  • Judge model choice: this has only a minor effect. GPT-5-mini scores 52.6 and GPT-5 with high reasoning effort scores 55.0.
  • Distillation: the ability to generate rubrics can be transferred to smaller open-weight models. These distilled models outperform both a patch-classifier verifier and the base models before fine-tuning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksInformation Systems · 1,437 worksSoftware · 92 works

Cite

Cite as Mohit Raghavendra, Anisha Gunjal, Bing Liu and Yunzhong He. 2026. Agentic Rubrics as Contextual Verifiers for SWE Agents. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.697

@inproceedings{raghavendra2026agentic,
  title = {{Agentic Rubrics as Contextual Verifiers for SWE Agents}},
  author = {Mohit Raghavendra, Anisha Gunjal, Bing Liu and Yunzhong He},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.697},
  url = {https://doi.org/10.18653/v1/2026.acl-long.697}
}

References

29 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Afra Feyza Akyürek, Advait Gosai, Chen Bo Calvin Zhang, Vipul Gupta, Jaehwan Jeong, Anisha Gunjal, Tahseen Rabbani, Maria Mazzone, David Randolph, Mohammad Mahmoudi Meymand, Gurshaan Chattha, Paula Rodriguez, Diego Mares, Pavit Singh, Michael Liu, Subodh Chawla, Pete Cline, Lucy Ogaz, Ernesto Hernan… — PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning
  2. Rahul K Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. 2025. Healthbench: Evaluating large language models towards improved human health. arXiv pre…
  3. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. 2024. Large language monkeys: Scaling inference compute with repeated sampling. Preprint, arXiv:2407.21787.
  4. Giuseppe Crupi, Rosalia Tufano, Alejandro Velasco, Antonio Mastropaolo, Denys Poshyvanyk, and Gabriele Bavota. 2025. On the effectiveness of llm-as-a-judge for code generation and summarization. Preprint, arXiv:2507.16587. — [arXiv:2507.16587](https://arxiv.org/abs/2507.16587) find
  5. Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. 2025. SWE-Bench Pro: Can… — [arXiv:2509.16941](https://arxiv.org/abs/2509.16941) find
  6. Ryan Ehrlich, Bradley Brown, Jordan Juravsky, Ronald Clark, Christopher Ré, and Azalia Mirhoseini. 2025. Codemonkeys: Scaling test-time compute for software engineering. Preprint, arXiv:2501.14723. — [arXiv:2501.14723](https://arxiv.org/abs/2501.14723) find
  7. Shashwat Goel, Rishi Hazra, Dulhan Jayalath, Timon Willi, Parag Jain, William F. Shen, Ilias Leontiadis, Francesco Barbieri, Yoram Bachrach, Jonas Geiping, and Chenxi Whitehouse. 2025. Training ai co-scientists using rubric rewards. Preprint, arXiv:2512.23707. find
  8. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. 2025. Rubrics as rewards: Reinforcement learning beyond verifiable domains. arXiv preprint arXiv:2507.17746. find
  9. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, and 1 others. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. — Qwen2.5-Coder Technical Report
  10. Naman Jain, Jaskirat Singh, Manish Shetty, Liang Zheng, Koushik Sen, and Ion Stoica. 2025. R2e-gym: Procedural environments and hybrid verifiers for scaling open-weights swe agents. arXiv preprint arXiv:2504.07164. — [arXiv:2504.07164](https://arxiv.org/abs/2504.07164) find
  11. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770.
  12. Michael Luo, Naman Jain, Jaskirat Singh, Sijun Tan, Ameen Patel, Qingyang Wu, Alpay Ariyak, Colin Cai, Shang Zhu Tarun Venkat, Ben Athiwaratkun, and 1 others. Deepswe: Training a fully open-sourced, state-of-the-art coding agent by scaling rl. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, K… find
  13. OpenAI. 2024. Introducing swe-bench verified. https://openai.com/index/ introducing-swe-bench-verified/. OpenAI Blog. ↗
  14. Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. 2024. Training software engineering agents and verifiers with SWE-gym. arXiv preprint arXiv:2412.21139. find
  15. Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G Finlayson, David Sontag, and 1 others. 2025. Dr tulu: Reinforcement learning with evolving rubrics for deep research. arXiv preprint arXiv:2511.19399.
  16. Nishad Singhi, Hritik Bansal, Arian Hosseini, Aditya Grover, Kai-Wei Chang, Marcus Rohrbach, and Anna Rohrbach. 2025. When to solve, when to verify: Compute-optimal problem solving and generative verification for llm reasoning. arXiv preprint arXiv:2504.01005.
  17. SWE-agent Team. 2024. mini-swe-agent: A 100-line software engineering agent. https://github.com/ SWE-agent/mini-swe-agent. GitHub repository. ↗
  18. Vijay Viswanathan, Yanchao Sun, Shuang Ma, Xiang Kong, Meng Cao, Graham Neubig, and Tongshuang Wu. 2025. Checklists are better than reward models for aligning language models. arXiv preprint arXiv:2507.18624.
  19. Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, and 1 others. 2024. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741.
  20. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. Proceedings of the International Conference on Learning Representations (ICLR). ArXiv:2203.11171.
  21. Jason Wei. 2025. Asymmetry of verification and verifier’s rule. Blog post. find
  22. Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. 2025a. SWE-RL: Advancing LLM reasoning via reinforcement learning on open software evolution. arXiv preprint arXiv:2502.18449. find
  23. Yuxiang Wei, Zhiqing Sun, Emily McMilin, Jonas Gehring, David Zhang, Gabriel Synnaeve, Daniel Fried, Lingming Zhang, and Sida Wang. 2025b. Toward training superintelligent software agents through self-play swe-rl. Preprint, arXiv:2512.18552. — [arXiv:2512.18552](https://arxiv.org/abs/2512.18552) find
  24. Mian Wu, Gavin Zhang, Sewon Min, Sergey Levine, and Aviral Kumar. 2025. Rlac: Reinforcement learning with adversarial critic for free-form generation tasks. arXiv preprint arXiv:2511.01758. — [arXiv:2511.01758](https://arxiv.org/abs/2511.01758) find
  25. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying llm-based software engineering agents. Preprint, arXiv:2407.01489. find
  26. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  27. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528– 50652.
  28. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems. find
  29. Kang Zhu, Hanhao Li, Siwei Wu, Tianshun Xing, Dehua Ma, Xiangru Tang, Minghao Liu, Jian Yang, Jiaheng Liu, Yuchen Eleanor Jiang, Changwang Zhang, Chenghua Lin, Jun Wang, Ge Zhang, and Wangchunshu Zhou. 2025. Scaling test-time compute for LLM agents. arXiv preprint arXiv:2506.12928. — [arXiv:2506.12928](https://arxiv.org/abs/2506.12928) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/raghavendra-2026-agentic-rubrics-as-contextual-verifiers/