Elamur Research⌕
☰
home / papers / 2026 / PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning

PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning

Paper · 2026 · Afra Feyza Akyürek, Advait Gosai, Chen Bo Calvin Zhang et al.

Summary

Most LLM benchmarks reward short, checkable answers in math, science and coding. They reveal little about open-ended, high-stakes professional work. Preference arenas, the usual option for open-ended evaluation, produce noisy scores without expert grounding. The Scale AI authors address this gap with PRBench, a set of 1,100 realistic Finance and Law tasks. The tasks were written by 182 vetted professionals: lawyers hold JDs, and finance contributors have a Master's, a CFA or 6+ years of practice. Each task draws on the expert's own workflow or on questions from clients. Every task has an expert-built rubric of 10–30 weighted criteria. A second expert reviews each rubric, and automated quality checks follow, so an LLM judge can grade responses interpretably. Across more than 20 proprietary and open models, even the strongest score only about 0.40 on the Hard subsets. Models handle instruction following fairly well but often make wrong judgments or reason opaquely and incompletely. The open release gives researchers a large, transparent tool for measuring professional reasoning.

Key points

  • Scale and coverage: 600 Finance and 500 Legal tasks carry 18,711 rubric criteria (9,879 Finance, 8,832 Legal). About 30% of conversations are multi-turn. The tasks span 13 Finance and 12 Legal topics, 114 countries and 47 US jurisdictions.
  • Hard subsets: 300 Finance and 250 Legal tasks were selected by low average model scores. The best results are 0.40 (Finance) and 0.41 (Legal), averaged over three runs with 95% confidence intervals.
  • Conversation construction: Experts wrote the user turns in English. Assistant turns were sampled from GPT OSS 20B, Mistral Medium or DeepSeek R1. A private held-out set is kept to detect future contamination.
  • Resources: The data is available at Hugging Face ScaleAI/PRBench, and the evaluation code at GitHub scaleapi/PRBench. Prompts are also annotated for economic impact and decision type.
  • Limitations: LLM judges can over-credit answers that only look correct. Tasks are English-only and exclude tool use (databases, spreadsheets).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksPolitical Science and International Relations · 182 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Afra Feyza Akyürek, Advait Gosai, Chen Bo Calvin Zhang, Vipul Gupta, Jaehwan Jeong, Anisha Gunjal, Tahseen Rabbani, Maria Mazzone, David Randolph IV, Mohammad Mahmoudi Meymand, Gurshaan Chattha, Paula Rodriguez, Diego A. Mares Buendia, Pavit Singh, Michael Liu, Subodh Chawla, Peter Cline, Lucy Ogaz, Ernesto Gabriel Hernández Montoya, Zihao Wang, Pavi Bhatter, Marcos Ayestaran, Bing Liu and Yunzhong He. 2026. PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1958

@inproceedings{akyurek2026prbench,
  title = {{PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning}},
  author = {Afra Feyza Akyürek, Advait Gosai, Chen Bo Calvin Zhang, Vipul Gupta, Jaehwan Jeong, Anisha Gunjal, Tahseen Rabbani, Maria Mazzone, David Randolph IV, Mohammad Mahmoudi Meymand, Gurshaan Chattha, Paula Rodriguez, Diego A. Mares Buendia, Pavit Singh, Michael Liu, Subodh Chawla, Peter Cline, Lucy Ogaz, Ernesto Gabriel Hernández Montoya, Zihao Wang, Pavi Bhatter, Marcos Ayestaran, Bing Liu and Yunzhong He},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1958},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1958}
}

References

41 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ruth Appel, Peter McCrory, Alex Tamkin, Michael Stern, Miles McCain, and Tyler Neylon. 2025. Anthropic economic index report: Uneven geographic and enterprise ai adoption. Anthropic Research. find
  2. Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. 2025. Healthbench: Evaluating large language models towards improved human health. Preprint…
  3. Aaron Chatterji, Thomas Cunningham, David J Deming, Zoe Hitzig, Christopher Ong, Carl Yan Shan, and Kevin Wadman. 2025. How people use chatgpt. Technical report, National Bureau of Economic Research. find
  4. Zhiyu Chen, SHIYANG LI, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering. In Conference on Empirical Methods in Natural Language Processing.
  5. Odysseas S. Chlapanis, Dimitrios Galanis, Nikolaos Aletras, and Ion Androutsopoulos. 2025. GreekBar-Bench: A challenging benchmark for free-text legal reasoning and citations. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 25099–25119, Suzhou, China. Association for… find
  6. Francois Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, and Henry Pinkard. 2025. Arc-agi-2: A new challenge for frontier ai reasoning systems. arXiv preprint arXiv:2505.11831. find
  7. Eva Eigner and Thorsten Händler. 2024. Determinants of llm-assisted decision-making. arXiv preprint arXiv:2402.17385. — [arXiv:2402.17385](https://arxiv.org/abs/2402.17385) find
  8. Yu Fan, Jingwei Ni, Jakob Merane, Yang Tian, Yoan Hermstruwer, Yinya Huang, Mubashara Akhtar, Etienne Salimbeni, Florian Geering, Oliver Dreyer, Daniel Brunner, Markus Leippold, Mrinmaya Sachan, Alexander Stremitzer, Christoph Engel, Elliott Ash, and Joel Niklaus. 2025. Lexam: Benchmarking legal rea… find
  9. Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Re, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, and 21 ot… — LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
  10. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. 2025. Rubrics as rewards: Reinforcement learning beyond verifiable domains. Preprint, arXiv:2507.17746. find
  11. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In Proceedings of the International Conference on Learning Representations (ICLR).
  12. Luis Hillebrand, Sebastian Raisch, and Jonathan Schad. 2025. Managing with artificial intelligence: An integrative framework. Academy of Management Annals, 19(1):343–375. find
  13. Zenan Huang, Yihong Zhuang, Guoshan Lu, Zeyu Qin, Haokai Xu, Tianyu Zhao, Ru Peng, Jiaqi Hu, Zhanming Shen, Xiaomeng Hu, Xijun Gu, Peiyi Tu, Jiaxin Liu, Wenyu Chen, Yuzhuo Fu, Zhiting Fan, Yanmei Gu, Yuanyuan Wang, Zhengkai Yang, and 2 others. 2025. Reinforcement learning with rubric anchors. Prepri… — [arXiv:2508.12790](https://arxiv.org/abs/2508.12790) find
  14. Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023. Financebench: A new benchmark for financial question answering. ArXiv, abs/2311.11944. — [arXiv:2311.11944](https://arxiv.org/abs/2311.11944) find
  15. Mohsen Khosravi, Zahra Zare, Seyyed M. Mojtabaeian, and Reyhane Izadi. Artificial intelligence and decision-making in healthcare: A thematic analysis of a systematic review of reviews. Health Services Research and Managerial Epidemiology, 11. find
  16. Sihyun Kim, Sangyoon Yi, and Sung-Pil Park. 2025. Prioritizing challenges in ai adoption for the legal domain: A systematic review and expert-driven AHP analysis. PLOS ONE, 20(6):e0326028. find
  17. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval. ↗
  18. LMArena. 2025. Arena expert and occupational categories. LMArena News. Blog post. find
  19. LMArena Team. 2025. Arena expert and occupational categories. https://news.lmarena.ai/ arena-expert/. Accessed: 2025-11-11. ↗
  20. Mantas Mazeika, Alice Gatti, Cristina Menghini, Udari Madhushani Sehwag, Shivam Singhal, Yury Orlovskiy, Steven Basart, Manasi Sharma, Denis Peskoff, Elaine Lau, Jaehyuk Lim, Lachlan Carroll, Alice Blair, Vinaya Sivakumar, Sumana Basu, Brad Kenstler, Yuntao Ma, Julian Michael, Xiaoke Li, and 28 othe… — [arXiv:2510.26787](https://arxiv.org/abs/2510.26787) find
  21. Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. 2025. Swe-lancer: Can frontier llms earn $1 million from real-world freelance software engineering? Preprint, arXiv:2502.12115. nof1.ai. 2025. Alpha arena. https://nof1.ai/. Accessed: 2025-11-11. ↗
  22. Melissa Pan, Negar Arabzadeh, Riccardo Cogo, Yuxuan Zhu, Alexander Xiong, Lakshya Agrawal, Huanzhi Mao, Emma Shen, Sid Pallerla, Liana Patel, Shu Liu, Tianneng Shi, Xiaoyuan Liu, Jared Davis, Emmanuele Lacavalla, Alessandro Basile, Shuyi Yang, Paul Castro, Daniel Kang, and 6 others. 2025. Measuring… — [arXiv:2512.04123](https://arxiv.org/abs/2512.04123) find
  23. Annalisa Szymanski, Noah Ziems, Heather A. Eicher-Miller, Toby Jia-Jun Li, Meng Jiang, and Ronald A. Metoyer. 2025. Limitations of the llm-as-a-judge approach for evaluating llm outputs in expert knowledge tasks. In Proceedings of the 30th International Conference on Intelligent User Interfaces, IUI… find
  24. Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Simón Posada Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, Natalie S. Kim, Patrick Chao, Samuel Miserendino, Gildas Chabot, David Li, Michael Sharman, Alexandra Barr, Amelia Glaese, and Jerry Two… find
  25. Zichen Tang, Haihong E, Ziyan Ma, Haoyang He, Jiacheng Liu, Zhongjun Yang, Zihua Rong, Rongjin Li, Kun Ji, Qing Huang, Xinyang Hu, Yang Liu, and Qianhe Zheng. 2025. FinanceReasoning: Benchmarking financial numerical reasoning more credible, comprehensive and challenging. In Proceedings of the 63rd A…
  26. Julio Pereyra, Elizabeth Lebens, Matthew Guillod, Laura Toulme, Cameron MacGregor, David Mur-dter, Karl de la Roche, Emilie McConnachie, Jeremy Pushkin, Rina Kim, Aaron Chan, Jenny Pan, Boling Yang, Nan Wu, Niko Grupen, Lauren Oh, Aatish Nayak, and Gabriel Pereyra. 2024. Introducing biglaw bench. ht… find
  27. Long Phan, Alice Gatti, Nathaniel Li, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Dan Hendrycks, Ziwen Han, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, and 1081 others. 2026. A benchmark of e… find
  28. Inc. Vals AI. 2025. Corpfin (v2): A private benchmark evaluating understanding of long-context credit agreements. Online benchmark. Accessed on April 20, 2026. find
  29. Bertie Vidgen, Abby Fennelly, Evan Pinnix, Chirag Mahapatra, Zach Richards, Austin Bridges, Calix Huang, Ben Hunsberger, Fez Zafar, Brendan Foody, Dominic Barton, Cass R. Sunstein, Eric Topol, and Osvald Nitski. 2025. The ai productivity index (apex). Preprint, arXiv:2509.25721. — [arXiv:2509.25721](https://arxiv.org/abs/2509.25721) find
  30. Darko B. Vuković, Senanu Dekpo-Adza, and Stefana Matović. 2025. AI integration in financial services: A systematic review of trends and regulatory challenges. Humanities and Social Sciences Communications, 12(1):562. find
  31. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  32. Zhilin Wang, Jaehun Jung, Ximing Lu, Shizhe Diao, Ellie Evans, Jiaqi Zeng, Pavlo Molchanov, Yejin Choi, Jan Kautz, and Yi Dong. 2025. Profbench: Multi-domain rubrics requiring professional knowledge to answer and judge. Preprint, arXiv:2510.18941. — [arXiv:2510.18941](https://arxiv.org/abs/2510.18941) find
  33. MohammadHossein Rezaei, Robert Vacareanu, Zihao Wang, Clinton Wang, Bing Liu, Yunzhong He, and Afra Feyza Akyürek. 2025. Online rubrics elicitation from pairwise comparisons. Preprint, arXiv:2510.07284. — [arXiv:2510.07284](https://arxiv.org/abs/2510.07284) find
  34. Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. 2025. Browsecomp: A simple yet challenging benchmark for browsing agents. Preprint, arXiv:2504.12516. find
  35. Nikki Rojas. 2024. Does ai help humans make better decisions? Harvard Gazette. find
  36. Scale AI. 2025. Seal showdown: Technical report. Technical report, Scale AI. Preliminary results and methodology for the SEAL Showdown leaderboard. find
  37. Ved Sirdeshmukh, Kaustubh Deshpande, Johannes Mols, Lifeng Jin, Ed-Yeremai Cardona, Dean Lee, Jeremy Kritz, Willow Primack, Summer Yue, and Chen Xing. 2025. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llms. Preprint, arXiv:2501.17399. find
  38. Mark Steyvers and Aakriti Kumar. 2024. Three challenges for ai-assisted decision-making. Perspectives on Psychological Science, 19(5):722–734. find
  39. Jannik Zeiser. 2024. Owning decisions: Ai decision-support and the attributability-gap. Science and Engineering Ethics, 30(4):1–19. find
  40. Junkai Zhang, Zihao Wang, Lin Gui, Swarnashree Mysore Sathyendra, Jaehwan Jeong, Victor Veitch, Wei Wang, Yunzhong He, Bing Liu, and Lifeng Jin. 2025. Chasing the tail: Effective rubric-based reward modeling for large language model post-training. Preprint, arXiv:2509.21500. — [arXiv:2509.21500](https://arxiv.org/abs/2509.21500) find
  41. Yilun Zhao, Hongjun Liu, Yitao Long, Rui Zhang, Chen Zhao, and Arman Cohan. 2024. Financemath: Knowledge-intensive math reasoning in finance domains. In Proceedings of the 62nd Annual Meeting of Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/akyurek-2026-prbench-large-scale-expert-rubrics/