Elamur Research⌕
☰
home / papers / 2026 / Complex-IF and Beyond: Expert Rubrics for RLVR

Complex-IF and Beyond: Expert Rubrics for RLVR

Paper · 2026 · Sushant Mehta, Liudas Panavas, Eleanor Fleming et al.

Summary

The authors argue that common instruction-following benchmarks such as IFEval check only shallow, script-verifiable constraints. A model can pass them while producing poor answers, and frontier models are increasingly indistinguishable on such tests. As an alternative, they propose rubric-based evaluation authored by domain experts and lay out five guidelines for building such rubrics:

  • criteria split only down to the smallest meaningful unit,
  • criteria grounded in what the user actually wants,
  • a three-way split into standard, bonus-only and penalty-only items,
  • repeated tuning of wording against an LLM judge,
  • tasks rooted in realistic domain complexity.

They apply these ideas in COMPLEX-IF, a new public dataset of about 1,000 prompts, each carrying 10–40 atomic criteria. They then use the same rubrics as RLVR rewards. The resulting gains appear on the training distribution and also carry over to independently built benchmarks, both for instruction following and for agentic work in the Core-Craft enterprise simulation. The work suggests that the extra cost of expert rubric authoring pays off twice: once in sharper measurement and again in data-efficient training.

Key points

  • Dataset: COMPLEX-IF contains roughly 1,000 expert-written prompts, each paired with 10–40 criteria. Criteria fall into three types:
    • Primary Intent (standard),
    • Extra Credit (reward only),
    • Dodged Bullet (penalty only).
  • Instruction-following gains: RL training on COMPLEX-IF improved instruction following by +15.5 pp for a 4B model and +12.2 pp for a 235B model. It also gave +8.45% on the independently authored AdvancedIF benchmark, compared with 6.7% reported by RIFL. The authors note this comparison is not controlled.
  • Agentic transfer: One epoch of rubric-graded RL in Core-Craft transferred to unseen benchmarks: +4.5 pp BFCL, +7.4 pp τ²-Bench and +6.8 pp Toolathlon.
  • Difficulty: The tasks are deliberately hard. No model's perfect-task rate exceeds 17% on COMPLEX-IF or 43% on Core-Craft. Among frontier models, COMPLEX-IF scores range from 4.14% to 16.55%, roughly a fourfold spread.
  • Limitations:
    • Results come from single-seed runs.
    • No ablations isolate the effect of each design principle.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Sushant Mehta, Liudas Panavas, Eleanor Fleming, Paul Mains and Edwin Chen. 2026. Complex-IF and Beyond: Expert Rubrics for RLVR. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.gem-main.61

@inproceedings{mehta2026complex,
  title = {{Complex-IF and Beyond: Expert Rubrics for RLVR}},
  author = {Sushant Mehta, Liudas Panavas, Eleanor Fleming, Paul Mains and Edwin Chen},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.61},
  url = {https://doi.org/10.18653/v1/2026.gem-main.61}
}

References

31 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. HealthBench: Evaluating large language models towards improved human health. 2025. arXiv pr…
  2. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ 2 -Bench: Evaluating conversational agents in a dual-control environment. 2025. arXiv preprint arXiv:2506.07982.
  3. Khaoula Chehbouni, Mohammed Haddou, Jackie Chi Kit Cheung, and Golnoosh Farnadi. Neither valid nor reliable? investigating the use of LLMs as judges. 2025. In NeurIPS Position Paper Track. arXiv preprint arXiv:2508.18076. — [arXiv:2508.18076](https://arxiv.org/abs/2508.18076) find
  4. DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. Nature, 645:633–638, 2025.
  5. Kaustubh D. Dhole et al. RubricRAG: Towards interpretable and reliable LLM evaluation via domain knowledge retrieval for rubric generation. 2026. arXiv preprint arXiv:2603.20882. — [arXiv:2603.20882](https://arxiv.org/abs/2603.20882) find
  6. Maria Eriksson, Erasmo Purificato, Arman Noroozian, João Vinagre, Guillaume Chaslot, Emilia Gómez, and David Fernández-Llorca. Can we trust AI benchmarks? an interdisciplinary review of current issues in AI evaluation. 2025. arXiv preprint arXiv:2502.06559. — [arXiv:2502.06559](https://arxiv.org/abs/2502.06559) find
  7. Jiawei Gu, Xuhui Jiang, et al. A survey on LLM-as-ajudge. 2024. arXiv preprint arXiv:2411.15594. find
  8. Yun He, Wenzhe Li, Hejia Zhang, Songlin Li, Karishma Mandyam, Sopan Khosla, Yuanhao Xiong, Nanshu Wang, Selina Peng, Beibin Li, et al. AdvancedIF: Rubric-based benchmarking and reinforcement learning for advancing LLM instruction following. 2025. arXiv preprint arXiv:2511.10507.
  9. Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. FollowBench: A multi-level fine-grained constraints following benchmark for large language models. 2024. In Proceedings of ACL. arXiv preprint arXiv:2310.20410.
  10. Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. LLMs get lost in multi-turn conversation. 2025. arXiv preprint arXiv:2505.06120.
  11. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, et al. Tülu 3: Pushing frontiers in open language model post-training. 2024. arXiv preprint arXiv:2411.15124.
  12. Junlong Li, Wenshuo Zhao, Jian Zhao, Weihao Zeng, Haoze Wu, Xiaochen Wang, Rui Ge, et al. The tool decathlon: Benchmarking language agents for diverse, realistic, and long-horizon task execution. 2025. In ICLR. arXiv preprint arXiv:2510.25726.
  13. Sushant Mehta, Logan Ritchie, Suhaas Garre, Ian Niebres, Nick Heiner, and Edwin Chen. EnterpriseBench CoreCraft: Training generalizable agents on high-fidelity RL environments. 2026. arXiv preprint arXiv:2602.16179. — [arXiv:2602.16179](https://arxiv.org/abs/2602.16179) find
  14. Logan Ritchie, Sushant Mehta, Nick Heiner, Michelle Yu, and Edwin Chen. The hierarchy of agentic capabilities: Evaluating frontier models on realistic enterprise environments. 2026. arXiv preprint arXiv:2601.09032. — [arXiv:2601.09032](https://arxiv.org/abs/2601.09032) find
  15. Kayla Schroeder and Zach Wood-Doughty. Can you trust LLM judgments? reliability of LLM-as-a-judge. 2024. arXiv preprint arXiv:2412.12509. — [arXiv:2412.12509](https://arxiv.org/abs/2412.12509) find
  16. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. 2024. arXiv preprint arXiv:2402.03300.
  17. Vijay Viswanathan, Yanchao Sun, Shuang Ma, Xiang Kong, Meng Cao, Graham Neubig, and Tongshuang Wu. Checklists are better than reward models for aligning language models. 2025. In Advances in NeurIPS. arXiv preprint arXiv:2507.18624.
  18. Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, et al. Benchmarking complex instruction-following with multiple constraints composition. 2024. In NeurIPS Datasets and Benchmarks. arXiv preprint arXiv:2407.03978. — [arXiv:2407.03978](https://arxiv.org/abs/2407.03978) find
  19. Yusuke Yamauchi, Taro Yano, and Masafumi Oyamada. An empirical study of LLM-as-a-judge: How design choices impact evaluation reliability. 2025. arXiv preprint arXiv:2506.13639.
  20. Melissa Z. Pan, Negar Arabzadeh, Riccardo Cogo, Yuxuan Zhu, Alexander Xiong, et al. Measuring agents in production. 2025. arXiv preprint arXiv:2512.04123. — [arXiv:2512.04123](https://arxiv.org/abs/2512.04123) find
  21. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. 2024. arXiv preprint arXiv:2406.12045.
  22. Hao Peng, Yunjia Qi, Xiaozhi Wang, Bin Xu, Lei Hou, and Juanzi Li. VerIF: Verification engineering for reinforcement learning in instruction following. 2025. In Proceedings of EMNLP. arXiv preprint arXiv:2506.09942.
  23. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, et al. DAPO: An open-source LLM reinforcement learning system at scale. 2025. arXiv preprint arXiv:2503.14476.
  24. Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. Generalizing verifiable instruction following. 2025. arXiv preprint arXiv:2507.02833.
  25. Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. ToolRL: Reward is all tool learning needs. 2025. In Advances in NeurIPS. arXiv preprint arXiv:2504.13958.
  26. Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. InFoBench: Evaluating instruction following ability in large language models. 2024. In Findings of ACL. arXiv preprint arXiv:2401.03601.
  27. Delip Rao and Chris Callison-Burch. Autorubric: Unifying rubric-based LLM evaluation. 2026. arXiv preprint arXiv:2603.00077. — [arXiv:2603.00077](https://arxiv.org/abs/2603.00077) find
  28. Qiyuan Zhang, Junyi Zhou, Yufei Wang, Fuyuan Lyu, Yidong Ming, Can Xu, Qingfeng Sun, Kai Zheng, Peng Kang, Xue Liu, and Chen Ma. RubricBench: Aligning model-generated rubrics with human standards. 2026. arXiv preprint arXiv:2603.01562.
  29. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. LIMA: Less is more for alignment. In Advances in NeurIPS, 2023a. find
  30. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. 2023b. arXiv preprint arXiv:2311.07911.
  31. Shuyan Zhou, Frank F. Xu, Hao Zhu, et al. WebArena: A realistic web environment for building autonomous agents. In ICLR, 2024.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mehta-2026-complex-if-beyond-expert-rubrics/