Elamur Research⌕
☰
home / papers / 2026 / SOP-Maze: Evaluating Large Language Models on Complicated Business Standard Operating Procedures

SOP-Maze: Evaluating Large Language Models on Complicated Business Standard Operating Procedures

Paper · 2026 · Jiaming Wang, Zhe Tang, Zehao Jin et al.

Summary

Most instruction-following benchmarks do not test whether LLMs can carry out the long, branching standard operating procedures (SOPs) used in real business workflows, where user input is often noisy. SOP-Maze addresses this gap. The authors took 300,000 logged LLM calls from Meituan's internal API router, clustered them semantically, and kept the 23 most frequent business scenarios. They then refined the data with humans in the loop. Self-contradictions were resolved, but some realistic logical inconsistencies were kept on purpose, and multiple annotators checked every task. Tasks fall into two types:

  • Lateral Root System (LRS): many options with shallow branching, requiring precise selection.
  • Heart Root System (HRS): long, deep reasoning chains.

Scoring needs no judge model. Answers must match a JSON schema, and free-form replies must cite the index of the matching reference response in the SOP, so they can be checked exactly. Nearly all 18 tested state-of-the-art LLMs struggle. Error analysis identifies three recurring weaknesses: failing to follow procedural routes, misreading nuanced dialogue, and miscalculating. This gives a realistic picture of where deployed business agents break down.

Key points

  • The benchmark has 397 instances, 3422 subtasks and 23 scenarios (10 HRS, 13 LRS), with SOP specifications averaging about 5,040 tokens. It is released at github.com/meituan-longcat/SOP-Maze.
  • Every prompt combines four components: a statement of the Objective, the SOPs themselves, the User Input, plus an Output Requirement. Evaluation uses JSON schema checks plus reference-index matching instead of LLM-as-a-judge.
  • Quality control relied on five experienced annotators, each task reviewed by at least three, with unanimous agreement required. Scoring was cross-checked on outputs from Claude-4-Sonnet, GPT-4.1 and Deepseek-R1.
  • The three error categories are route blindness, conversational fragility (e.g., sarcasm, intent reversal) and calculation errors.
  • Ablations were run by progressively simplifying the query and context:
    • On the "Customer Service Rate" task, Claude-Opus-4 scored 40% and Doubao-Seed-1.6 30% at first, but simplified versions reached 90%. This suggests the real problem is picking the right evidence, not doing the arithmetic.
    • On the dialogue-nuance task, scores levelled off near 70% even with the SOP removed, pointing to conversation understanding itself as the bottleneck.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Jiaming Wang, Zhe Tang, Zehao Jin, Hefei Chen, Yilin Jin, Peng Ding, Xiaoyu Li and Xuezhi Cao. 2026. SOP-Maze: Evaluating Large Language Models on Complicated Business Standard Operating Procedures. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.715

@inproceedings{wang2026sop,
  title = {{SOP-Maze: Evaluating Large Language Models on Complicated Business Standard Operating Procedures}},
  author = {Jiaming Wang, Zhe Tang, Zehao Jin, Hefei Chen, Yilin Jin, Peng Ding, Xiaoyu Li and Xuezhi Cao},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.715},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.715}
}

References

39 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Claude sonnet 4. Visited date: 2025-09-15. find
  2. Alessandro Berti, Hani Kourani, and Wil M. P. van der Aalst. 2024. PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining Tasks. In International Conference on Process Mining, pages 610–623, Cham. Springer Nature Switzerland. find
  3. Bytedance. 2025. Doubao-ai. Visited date: 2025-09-15. find
  4. DeepSeek-AI. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. find
  5. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. Deepseek-r1: Incentivizing reasoning capability in ll…
  6. Lingxiao Diao, Xinyue Xu, Wanxuan Sun, Cheng Yang, and Zhuosheng Zhang. 2025. Guidebench: Benchmarking domain-oriented guideline following for llm agents. arXiv preprint arXiv:2505.11368. find
  7. Dirk Fahland, Fabian Fournier, Lior Limonad, Inbal Skarbovsky, and Alexander J. Swevels. 2024. How well can large language models explain business processes? arXiv preprint. find
  8. Fabiana Fournier, Lior Limonad, and Inna Skarbovsky. 2024. Towards a benchmark for causal business process reasoning with llms. Preprint, arXiv:2406.05506. find
  9. Google DeepMind. 2025. Gemini flash. Visited date: 2025-09-15. find
  10. Michael Grohs, Luka Abb, Nourhan Elsayed, and Jana-Rebecca Rehse. 2023. Large language models can accomplish business process management tasks. Preprint, arXiv:2307.09923. — [arXiv:2307.09923](https://arxiv.org/abs/2307.09923) find
  11. Qianyu He, Jie Zeng, Wenhao Huang, Lina Chen, Jin Xiao, Qianxi He, Xunzhe Zhou, Jiaqing Liang, and Yanghua Xiao. 2024a. Can large language models understand real-world complex instructions? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18188–18196. find
  12. Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, Shruti Bhosale, Chenguang Zhu, Karthik Abinav Sankararaman, Eryk Helenowski, Melanie Kambadur, Aditya Tayade, Hao Ma, Han Fang, and Sinong Wang. 2024b. Multi-if: Benchmarking llms on mul… — [arXiv:2410.15553](https://arxiv.org/abs/2410.15553) find
  13. Humam Kourani, Alessandro Berti, Jasmin Hennrich, Wolfgang Kratsch, Robin Weidlich, Chiao-Yun Li, Ahmad Arslan, Daniel Schuster, and Wil M. P. van der Aalst. 2024a. Leveraging large language models for enhanced process model comprehension. Preprint, arXiv:2408.08892. — [arXiv:2408.08892](https://arxiv.org/abs/2408.08892) find
  14. Humam Kourani, Alessandro Berti, Daniel Schuster, and Wil M. P. van der Aalst. 2024b. Evaluating large language models on business process modeling: Framework, benchmark, and self-improvement analysis. Preprint, arXiv:2412.00023. — [arXiv:2412.00023](https://arxiv.org/abs/2412.00023) find
  15. Pranjal Kumar. 2024. Large language models (llms): survey, technical frameworks, and future challenges. Artificial Intelligence Review, 57(10):260. 14576 9 find
  16. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Joshua Adrian Cahyono, Jingkang Yang, Chunyuan Li, and Ziwei Liu. 2025a. Otter: A multi-modal model with in-context instruction tuning. IEEE Transactions on Pattern Analysis and Machine Intelligence.
  17. Jinnan Li, Jinzhe Li, Yue Wang, Yi Chang, and Yuan Wu. 2025b. Structflowbench: A structured flow benchmark for multi-turn instruction following. Preprint, arXiv:2502.14494. find
  18. Zekun Li, Shinda Huang, Jiangtian Wang, Nathan Zhang, Antonis Antoniades, Wenyue Hua, Kaijie Zhu, Sirui Zeng, Chi Wang, William Yang Wang, and Xifeng Yan. 2025c. Sopbench: Evaluating language agents at following standard operating procedures and constraints. Preprint, arXiv:2503.08669. — [arXiv:2503.08669](https://arxiv.org/abs/2503.08669) find
  19. Subhrangshu Nandi, Arghya Datta, Nikhil Vichare, Indranil Bhattacharya, Huzefa Raja, Jing Xu, Shayan Ray, Giuseppe Carenini, Abhi Srivastava, Aaron Chan, Man Ho Woo, Amar Kandola, Brandon Theresa, and Francesco Carbone. 2025. Sop-bench: Complex industrial sops for evaluating llm agents. Preprint, ar… — [arXiv:2506.08119](https://arxiv.org/abs/2506.08119) find
  20. OpenAI. 2025a. Gpt-4.1. Visited date: 2025-09-15. find
  21. OpenAI. 2025b. o3-mini. Visited date: 2025-09-15. find
  22. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730– 27744.
  23. Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip S Yu. 2024a. Large language models meet nlp: A survey. arXiv preprint arXiv:2405.12819. — [arXiv:2405.12819](https://arxiv.org/abs/2405.12819) find
  24. Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024b. Infobench: Evaluating instruction following ability in large language models. Preprint, arXiv:2401.03601.
  25. Adrian Rebmann, Fabian David Schmidt, Goran Glavaš, and Han van Der Aa. 2024. Evaluating the ability of llms to solve semantics-aware process mining tasks. In 2024 6th International Conference on Process Mining (ICPM), pages 9–16. find
  26. Yuchong Sun, Che Liu, Kun Zhou, Jinwen Huang, Ruihua Song, Wayne Xin Zhao, Fuzheng Zhang, Di Zhang, and Kun Gai. 2024. Parrot: Enhancing multi-turn instruction following for large language models. Preprint, arXiv:2310.07301. find
  27. Xiangru Tang, Yiming Zong, Jason Phang, Yilun Zhao, Wangchunshu Zhou, Arman Cohan, and Mark Gerstein. 2024. Struc-bench: Are large language models good at generating complex structured tabular data? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computatio… find
  28. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, and 150 others. 2025. Kimi k2: Open agentic intelligence. Preprint, a…
  29. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-Instruct: Aligning Language Models with Self-Generated Instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find
  30. Zichong Wang, Zhibo Chu, Thang Viet Doan, Shiwen Ni, Min Yang, and Wenbin Zhang. 2025. History, development, and principles of large language models: an introductory survey. AI and Ethics, 5(3):1955– 1971. find
  31. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint.
  32. Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxing Xu, and 1 others. 2024. Benchmarking complex instruction-following with multiple constraints composition. Advances in Neural Information Processing Systems, 37:137610–137645. find
  33. Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang, Yihao Feng, Ran Xu, Wenpeng Yin, and Caiming Xiong. 2024. Fofo: A benchmark to evaluate llms’ format-following capability. arXiv preprint arXiv:2402.18667. find
  34. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  35. Shunyu Yao, Howard Chen, Austin W. Hanjie, Runzhe Yang, and Karthik Narasimhan. 2023. Collie: Systematic construction of constrained text generation tasks. Preprint, arXiv:2307.08689.
  36. Junjie Zhang, Ruobing Xie, Yupeng Hou, Xin Zhao, Leyu Lin, and Ji-Rong Wen. 2025. Recommendation as instruction following: A large language model empowered recommendation approach. ACM Transactions on Information Systems, 43(5):1–37. find
  37. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and 14577 10 find
  38. Le Hou. 2023a. Instruction-following evaluation for large language models. Preprint, arXiv:2311.07911.
  39. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023b. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-sop-maze-evaluating-large-language/