Elamur Research⌕
☰
home / papers / 2026 / ChemReason-Bench: Benchmarking Large Language Models for Procedural Reasoning in Experimental Chemistry

ChemReason-Bench: Benchmarking Large Language Models for Procedural Reasoning in Experimental Chemistry

Paper · 2026 · Jinwei Zhang, Xucheng Liang, Yu Zhang et al.

Summary

Most chemistry benchmarks for large language models test factual knowledge through QA or multiple-choice questions. They say little about whether a model can make sound decisions inside a lab protocol, such as putting steps in a workable order, respecting the stated conditions, and keeping each reagent tied to its role. ChemReason-Bench targets this gap. The authors convert organic synthesis records into a canonical form of action-tagged natural-language steps, with placeholders for entities and conditions and a single shared schema. This design lets parsers and validators check many outputs automatically.

From 500 reactions they generated 7306 human-validated test items in six task formats, using human-designed templates, LLM-assisted canonicalization and human re-annotation. They also release a much larger, disjoint training resource, ChemReason-Tune. Testing open-source, proprietary and chemistry-specific models showed large differences between tasks, which a single overall score would hide. In adaptation experiments, supervised fine-tuning brought small open models close to proprietary systems.

Key points

  • Six task types: ordering, step validation, condition validation, schema-constrained step completion, contrastive choice, and evidence-grounded rationalization. Each type has roughly 1077–1483 instances in CR-Bench.
  • CR-Tune training resource: built from more than 20,000 reactions, with 123,903 instances. It includes 99,122 SFT examples (89,210 train / 9,912 val) and 24,781 GRPO examples (19,825 train / 4,956 val).
  • Evaluation protocol: each task has its own primary and auxiliary metrics. Discriminative tasks are scored in two ways, by generation and by LM likelihood (gen/lm).
  • Adaptation results: LoRA-based SFT was applied to three open-source backbones. Fine-tuned Gemma-2-9B reached the level of proprietary systems but still trailed the strongest model. Adapted models scored best on COND-VAL and STEP-COMP, while GRPO added little beyond SFT.
  • Limitations: because inputs are placeholder-normalized, the benchmark does not measure end-to-end understanding of raw lab prose. It covers only small-molecule organic synthesis, and it does not certify execution safety.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Southern CaliforniaLos Angeles · United States175 works · 37,047 citations🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Jinwei Zhang, Xucheng Liang, Yu Zhang, Ruijie Yu, Xiaokang Yang, Yaohui Jin and Yanyan Xu. 2026. ChemReason-Bench: Benchmarking Large Language Models for Procedural Reasoning in Experimental Chemistry. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1535

@inproceedings{zhang2026chemreason,
  title = {{ChemReason-Bench: Benchmarking Large Language Models for Procedural Reasoning in Experimental Chemistry}},
  author = {Jinwei Zhang, Xucheng Liang, Yu Zhang, Ruijie Yu, Xiaokang Yang, Yaohui Jin and Yanyan Xu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1535},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1535}
}

References

46 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, and 110 others. 2024. Phi-… — Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
  2. Qianxiang Ai, Fanwang Meng, Jiale Shi, Brenden Pelkie, and Connor W. Coley. 2024. Extracting structured data from organic synthesis procedures using a fine-tuned large language model. Digital Discovery, 3(9):1822–1831. find
  3. Anthropic. 2025. System card: Claude Opus 4 & Claude Sonnet 4. Accessed: 2025-12-19. find
  4. Daniil A. Boiko, Robert MacKnight, Benjamin Kline, and Gabriel Gomes. 2023. Autonomous chemical research with large language models. Nature, 624(7992):570–578.
  5. Marcus Bran, Samuel Cox, Oliver Schilter, Carlo Baldassari, Andrew D. White, and Philippe Schwaller. 2024. Augmenting large language models with chemistry tools. Nature Machine Intelligence, 6(5):525– 535. find
  6. Glenn W. Brier. 1950. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1–3. find
  7. Benjamin Burger, Phillip M. Maffettone, Vladimir V. Gusev, Catherine M. Aitchison, Yang Bai, Xiaoyan Wang, Xiaobo Li, Ben M. Alston, Buyi Li, Rob Clowes, Nicola Rankin, Brandon Harris, Reiner Sebastian Sprick, and Andrew I. Cooper. 2020. A mobile robotic chemist. Nature, 583(7815):237–241. find
  8. Xiuying Chen, Tairan Wang, Taicheng Guo, Kehan Guo, Juexiao Zhou, Haoyang Li, Zirui Song, Xin Gao, and Xiangliang Zhang. 2025. Unveiling the power of language models in chemical research question answering. Communications Chemistry, 8(1):4.
  9. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, and 3416 other…
  10. Weichen Dai, Yezeng Chen, Zijie Dai, Yubo Liu, Zhijie Huang, Yixuan Pan, Baiyang Song, Chengli Zhong, Xinhe Li, Zeyu Wang, Zhuoying Feng, and Yi Zhou. 2026. Kale-lm-chem: Vision and practice toward an ai brain for chemistry. Preprint, arXiv:2409.18695. — [arXiv:2409.18695](https://arxiv.org/abs/2409.18695) find
  11. Jesse Davis and Mark H. Goadrich. 2006. The relationship between precision-recall and ROC curves. In Proceedings of the 23rd International Conference on Machine Learning, pages 233–240. find
  12. DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, and 245 others. 2025. DeepSeek-v3.2: Pushing the frontier of open…
  13. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  14. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321– 1330. PMLR. find
  15. Taicheng Guo, Kehan Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh Chawla, Olaf Wiest, and Xiangliang Zhang. 2023. What can large language models do in chemistry? A comprehensive benchmark on eight tasks. In Advances in Neural Information Processing Systems, volume 36, pages 59662–59688. Curran… find
  16. James A. Hanley and Barbara J. McNeil. 1982. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1):29–36. find
  17. Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, and 399 others. 2024. GPT-4o…
  18. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  19. Steven M. Kearnes, Michael R. Maser, Michael Wleklinski, Adam Kast, Abigail G. Doyle, Spencer D. Dreher, John M. Hawkins, Klavs F. Jensen, and Connor W. Coley. 2021. The open reaction database. Journal of the American Chemical Society, 143(45):18820– 18826. find
  20. Hanzheng Li, Xi Fang, Yixuan Li, Chaozheng Huang, Junjie Wang, Xi Wang, Hongzhe Bai, Bojun Hao, Shenyu Lin, Huiqi Liang, Linfeng Zhang, and Guolin Ke. 2026. Rxnbench: A multimodal benchmark for evaluating large language models on chemical reaction understanding from scientific literature. Preprint,… — [arXiv:2512.23565](https://arxiv.org/abs/2512.23565) find
  21. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  22. Zhiyuan Liu, Yaorui Shi, An Zhang, Sihang Li, Enzhi Zhang, Xiang Wang, Kenji Kawaguchi, and Tat-Seng Chua. 2024. ReactXT: Understanding molecular “reaction-ship” via reaction-contextualized molecule-text pretraining. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5353–… find
  23. Daniel M. Lowe. 2017. Chemical reactions from US patents (1976–Sep 2016). Figshare dataset. Accessed: 2025-12-19. find
  24. Sam H. M. Mehr, Mark Craven, Andrei I. Leonov, Gregory Keenan, and Leroy Cronin. 2020. A universal system for digitization and automatic execution of the chemical synthesis literature. Science, 370(6512):101–108. find
  25. Anas Mirza, Nithin Alampara, Sandeep Kunchapu, Syed Moin, Jianyang Guo, Seyed Ali Chithrananda, and Connor W. Coley. 2025. A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists. Nature Chemistry, pages 1–8. find
  26. OpenAI. 2025. Update to GPT-5 system card: GPT-5.2. Accessed: 2025-12-19. find
  27. Organic Syntheses, Inc. 2025. Organic syntheses. Accessed: 2025-12-19. find
  28. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  29. Qwen Team, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2025. Qwen2.5 technical report. Preprint, arXiv:24… find
  30. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. GPQA: A graduate-level google-proof q&a benchmark. Preprint, arXiv:2311.12022.
  31. Nicholas T. Runcie, Charlotte M. Deane, and Fergus Imrie. 2025. Assessing the chemical intelligence of large language models. Preprint, arXiv:2505.07735. — [arXiv:2505.07735](https://arxiv.org/abs/2505.07735) find
  32. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, and 179… find
  33. Qwen Team, An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, and 42 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  34. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  35. Alain C. Vaucher, Philippe Schwaller, Joppe Geluykens, Vishnu H. Nair, Anna Iuliano, and Teodoro Laino. 2021. Inferring experimental procedures from text-based representations of chemical reactions. Nature Communications, 12(1):2573. find
  36. Alain C. Vaucher, Federico Zipoli, Joppe Geluykens, Vishnu H. Nair, Philippe Schwaller, and Teodoro Laino. 2020. Automated extraction of chemical synthesis actions from experimental procedures. Nature Communications, 11(1):3601. xAI. 2025. Grok 4 fast model card. Accessed: 2025-12-19. find
  37. Jiaqing Xie, Weida Wang, Ben Gao, Zhuo Yang, Haiyuan Wan, Shufei Zhang, Tianfan Fu, and Yuqiang Li. 2025. Qcbench: Evaluating large language models on domain-specific quantitative chemistry. Journal of Chemical Information and Modeling, 65(22):12268–12278. find
  38. Xianrui Zhong, Yufeng Du, Siru Ouyang, Ming Zhong, Tingfeng Luo, Qirong Ho, Hao Peng, Heng Ji, and Jiawei Han. 2024. ActionIE: Action extraction from scientific literature with programming languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1… find
  39. Botao Yu, Frazier N. Baker, Ziqi Chen, Xia Ning, and Huan Sun. 2024. LlaSMol: Advancing large language models for chemistry with a large-scale, comprehensive, high-quality instruction tuning dataset. Preprint, arXiv:2402.09391. find
  40. Zheni Zeng, Yi-Chen Nie, Ning Ding, Qian-Jun Ding, Wei-Ting Ye, Cheng Yang, Maosong Sun, Weinan E, Rong Zhu, and Zhiyuan Liu. 2023. Transcription between human-readable synthetic descriptions and machine-executable instructions: an application of the latest pre-training technology. Chemical Science,… find
  41. Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Wanli Ouyang, Dongzhan Zhou, Shufei Zhang, Mao Su, Han-Sen Zhong, and Yuqiang Li. 2024. Chem-LLM: A chemical large language model. Preprint, arXiv:2402.06852. find
  42. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with bert. Preprint, arXiv:1904.09675. find
  43. Yu Zhang, Yang Han, Shuai Chen, Ruijie Yu, Xin Zhao, Xianbin Liu, Kaipeng Zeng, Mengdi Yu, Jidong Tian, Feng Zhu, Xiaokang Yang, Yaohui Jin, and Yanyan Xu. 2025a. Large language models to accelerate organic chemistry synthesis. Nature Machine Intelligence, 7:1010–1022. find
  44. Yu Zhang, Ruijie Yu, Jidong Tian, Feng Zhu, Jiapeng Liu, Xiaokang Yang, Yaohui Jin, and Yanyan Xu. 2025b. ChemActor: Enhancing automated extraction of chemical synthesis actions with LLM-generated data. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volum… find
  45. Zehua Zhao, Zhixian Huang, Junren Li, Siyu Lin, Junting Zhou, Fengqi Cao, Kun Zhou, Rui Ge, Tingting Long, Yuexiang Zhu, Yan Liu, Jie Zheng, Junnian Wei, Rong Zhu, Peng Zou, Wenyu Li, Zekai Cheng, Tian Ding, Yaxuan Wang, and 12 others. 2025a. SU-PERChem: A multimodal reasoning benchmark in chemistry… — [arXiv:2512.01274](https://arxiv.org/abs/2512.01274) find
  46. Zihan Zhao, Da Ma, Lu Chen, Liangtai Sun, Zihao Li, Yi Xia, Bo Chen, Hongshen Xu, Zichen Zhu, Su Zhu, Shuai Fan, Guodong Shen, Kai Yu, and Xin Chen. 2025b. Developing ChemDFM as a large language foundation model for chemistry. Cell Reports Physical Science, 6(4):102523. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-chemreason-bench-benchmarking-large-language/