Elamur Research⌕
☰
home / papers / 2026 / REST: Stress Testing Large Reasoning Models by Asking Multiple Problems at Once

REST: Stress Testing Large Reasoning Models by Asking Multiple Problems at Once

Paper · 2026 · Zhuoshi Pan, Qizhi Pei, Yu Li et al.

Summary

Standard reasoning benchmarks pose one question per prompt. Many, such as GSM8K and MATH500, are now close to saturated for large reasoning models (LRMs), which forces repeated, costly creation of new test sets. The single-question format also ignores real deployments where a model must handle several queries in one pass. The authors propose REST (Reasoning Evaluation through Simultaneous Testing). It turns existing benchmarks into harder versions by placing several consecutive questions into one prompt. A "stress level" parameter sets how many questions are combined, and cyclic indexing ensures each question appears equally often in every position. Applied to 9 benchmarks (including MATH500, AIME24 and GPQA) across more than 30 models, REST causes clear accuracy losses even for top systems like DeepSeek-R1. It also separates models that look nearly identical under single-question testing. Error analysis links the drop mainly to overthinking, while models trained with "Long2Short" concise-reasoning methods cope better. The approach extends the useful life of existing benchmarks without new human annotation.

Key points

  • Coverage: Over 30 LRMs ranging from 1.5B to 671B parameters were tested on 9 reasoning benchmarks (math, GPQA, code) using the OpenCompass toolkit. Output was capped at 32K tokens for reasoning models; a 128K limit made little difference.
  • Construction: Questions are combined with cyclic indexing, so each question appears once in each of the s positions. The prompt set keeps its original size N. Answers are taken from \boxed{} or Python code blocks using regex- and LLM-based extraction.
  • Main result: DeepSeek-R1 scores 97.0% on single-question MATH500 but loses 29.1% accuracy on AIME24 under REST. This contradicts the assumption that LLMs are reliable multi-problem solvers.
  • Diagnostics:
    • Accuracy falls for questions placed later in the prompt (position bias).
    • Failures include skipped questions and summary errors.
    • In the attention analysis, Qwen3-8B, which degrades less, puts only 2.1% of attention on its own earlier reasoning.
  • Training insight: Models trained with the Long2Short technique (e.g., L1-Qwen-1.5B-Max, Efficient-R1-7B) keep more of their single-question accuracy. Code is available at github.com/opendatalab/REST.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓Renmin University of ChinaBeijing · China114 works · 16,118 citations🔬Shanghai Artificial Intelligence LaboratoryShanghai · China117 works · 11,281 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Zhuoshi Pan, Qizhi Pei, Yu Li, Zinan Tang, QiYao Sun, H. Vicky Zhao, Conghui He and Lijun Wu. 2026. REST: Stress Testing Large Reasoning Models by Asking Multiple Problems at Once. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1296

@inproceedings{pan2026rest,
  title = {{REST: Stress Testing Large Reasoning Models by Asking Multiple Problems at Once}},
  author = {Zhuoshi Pan, Qizhi Pei, Yu Li, Zinan Tang, QiYao Sun, H. Vicky Zhao, Conghui He and Lijun Wu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1296},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1296}
}

References

68 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pranjal Aggarwal and Sean Welleck. 2025. L1: Controlling how long a reasoning model thinks with reinforcement learning. In Second Conference on Language Modeling.
  2. AI-MO. 2024. AIMO Validation AIME Dataset. https://huggingface.co/datasets/AI-MO/ aimo-validation-aime. Accessed: 2025-03-29. ↗
  3. Daman Arora and Andrea Zanette. 2025. Training language models to reason efficiently. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  4. Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, Ido Shahaf, Oren Tropp, Ehud Karpas, Ran Zilberstein, Jiaqi Zeng, Soumye Singhal, Alexander Bukharin, Yian Zhang, Tugrul Konuk, and 114 others. 2025. Llama-nemotron:…
  5. Federico Di Maio and Manuel Gozzi. 2025. Degradation of multi-task prompting across six nlp tasks and llm families. Electronics, 14(21):4349. find
  6. Bangde Du, Minghao Guo, Songming He, Ziyi Ye, Xi Zhu, Weihang Su, Shuqi Zhu, Yujia Zhou, Yongfeng Zhang, Qingyao Ai, and 1 others. 2025a. Twinvoice: A multi-dimensional benchmark towards digital twins via llm persona simulation. arXiv preprint arXiv:2510.25536. find
  7. Bangde Du, Ziyi Ye, Zhijing Wu, Monika A Jankowska, Shuqi Zhu, Qingyao Ai, Yujia Zhou, and Yiqun Liu. 2025b. Simvbg: Simulating individual values by backstory generation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 13104–13133. find
  8. Marc Eulerich and David A Wood. 2023. A demonstration of how chatgpt can be used in the internal auditing process. Available at SSRN 4519583. find
  9. Hugging Face. 2025. Open r1: A fully open reproduction of deepseek-r1. find
  10. Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. 2025. Do NOT think that much for 2+3=? on the overthinking of long reasoning models. In Forty-second International Conference… find
  11. Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, and 1 others. 2024. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985.
  12. Zhoujun Cheng, Jungo Kasai, and Tao Yu. 2023. Batch prompting: Efficient inference with large language model apis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 792–810. find
  13. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  14. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
  15. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021a. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  16. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021a. Measuring mathematical problem solving with the math dataset. Preprint, arXiv:2103.03874.
  17. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the math dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1.
  18. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021b. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  19. Yutao Hou, Yajing Luo, Zhiwen Ruan, Hongru Wang, Weifeng Ge, Yun Chen, and Guanhua Chen. 2024. Compound-qa: A benchmark for evaluating llms on compound questions. arXiv preprint arXiv:2411.10163. — [arXiv:2411.10163](https://arxiv.org/abs/2411.10163) find
  20. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, and 1 others. 2025. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456.
  21. Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. 2025. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290.
  22. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974.
  23. Daniel Jaroslawicz, Brendan Whiting, Parth Shah, and Karime Maamari. 2025. How many instructions can llms follow at once? In NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling. find
  24. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations.
  25. Yu Li, Qizhi Pei, Mengyuan Sun, Honglin Lin, Chenlin Ming, Xin Gao, Jiang Wu, Conghui He, and Lijun Wu. 2025. Cipherbank: Exploring the boundary of llm reasoning capabilities through cryptography challenges. arXiv preprint arXiv:2504.19093. — [arXiv:2504.19093](https://arxiv.org/abs/2504.19093) find
  26. Jianzhe Lin, Maurice Diesendruck, Liang Du, and Robin Abraham. 2024. Batchprompt: Accomplish more with less. In The Twelfth International Conference on Learning Representations. find
  27. Yen-Ting Lin. 2025. Aime 2025 dataset. Accessed: 2025-03-29. find
  28. Gili Lior, Asaf Yehudai, Ariel Gera, and Liat Ein-Dor. 2025. Wildifeval: Instruction following in the wild. arXiv preprint arXiv:2503.06573. find
  29. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12.
  30. Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. 2025. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. Notion Blog. find
  31. Zhiyu Mei, Wei Fu, Kaiwei Li, Guangju Wang, Huanchen Zhang, and Yi Wu. 2025. Real: Efficient rlhf training of large language models with parameter reallocation. In Proceedings of the Eighth Conference on Machine Learning and Systems, MLSys 2025, Santa Clara, CA, USA, May 12-15, 2025. mlsys.org. find
  32. OpenAI. 2025b. Openai o3 and o4-mini system card. find
  33. Zhuoshi Pan, Yu Li, Honglin Lin, Qizhi Pei, Zinan Tang, Wei Wu, Chenlin Ming, H Vicky Zhao, Conghui He, and Lijun Wu. 2025. Lemma: Learning from errors for mathematical advancement in llms. arXiv preprint arXiv:2503.17439. find
  34. Qizhi Pei, Lijun Wu, Zhuoshi Pan, Yu Li, Honglin Lin, Chenlin Ming, Xin Gao, Conghui He, and Rui Yan. 2025. Mathfusion: Enhancing mathematic problem-solving of llm through instruction fusion. arXiv preprint arXiv:2503.16212. — [arXiv:2503.16212](https://arxiv.org/abs/2503.16212) find
  35. Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, and 1 others. 2025. Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings. arXiv preprint arXiv:2501.01257.
  36. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  37. Tushar Semwal, Promod Yenigalla, Gaurav Mathur, and Shivashankar B Nair. 2018. A practitioners’ guide to transfer learning for text classification using convolutional neural networks. In Proceedings of the 2018 SIAM international conference on data mining, pages 513–521. SIAM. find
  38. Connor Shorten, Charles Pierse, Thomas Benjamin Smith, Erika Cardenas, Akanksha Sharma, John Trengrove, and Bob van Luijt. 2024. Structuredrag: Json response formatting with large language models. arXiv preprint arXiv:2408.11061. — [arXiv:2408.11061](https://arxiv.org/abs/2408.11061) find
  39. Guijin Son, SangWon Baek, Sangdae Nam, Ilgyun Jeong, and Seungone Kim. 2024. Multi-task inference: Can large language models follow multiple instructions at once? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5606– 5627. find
  40. Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and 1 others. 2025. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503.16419.
  41. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. Preprint, arXiv:2501.19393.
  42. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and 1 others. 2023. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: AC…
  43. OpenAI. 2025a. gpt-oss-120b & gpt-oss-20b model card. Preprint, arXiv:2508.10925.
  44. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North, page 4149. Association for Computational Linguistics.
  45. Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. 2024. Let me speak freely? a study on the impact of format restrictions on performance of large language models. arXiv preprint arXiv:2408.02442. — [arXiv:2408.02442](https://arxiv.org/abs/2408.02442) find
  46. Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, and Xiangzheng Zhang. 2025. Light-r1: Curriculum SFT, DPO and RL for long COT from scratch and beyond. In Proceedings of the 63rd Annual Meeting of… find
  47. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  48. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  49. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, and 1 others. 2025a. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. find
  50. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024a. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. find
  51. Kimi Team. 2025a. Kimi linear: An expressive, efficient attention architecture. arXiv preprint arXiv:2510.26692.
  52. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024b. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement.…
  53. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, and 1 others. 2025b. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599. — Kimi k1.5: Scaling Reinforcement Learning with LLMs
  54. Ling Team. 2025b. Every attention matters: An efficient hybrid architecture for long-context reasoning. arXiv preprint arXiv:2510.19338. — [arXiv:2510.19338](https://arxiv.org/abs/2510.19338) find
  55. Open Thoughts Team. 2025c. Open Thoughts. find
  56. Qwen Team. 2025d. Qwq-32b: Embracing the power of reinforcement learning. find
  57. Dietrich Trautmann, Alina Petrova, and Frank Schilder. 2022. Legal prompt engineering for multilingual legal judgement prediction. arXiv preprint arXiv:2212.02199. find
  58. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, and 1 others. 2024c. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122.
  59. Zhiyu Yang, Shuo Wang, Yukun Yan, and Yang Deng. 2025b. Why stop at one error? benchmarking LLMs as data science code debuggers for multi-hop and multi-bug errors. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 17377–17392, Suzhou, China. Association… find
  60. Siyuan Wang, Zhuohan Long, Zhihao Fan, Xuan-Jing Huang, and Zhongyu Wei. 2025a. Benchmark self-evolving: A multi-agent framework for dynamic llm evaluation. In Proceedings of the 31st international conference on computational linguistics, pages 3310– 3328.
  61. Yixin Ye, Yang Xiao, Tiantian Mi, and Pengfei Liu. 2025. Aime-preview: A rigorous and immediate evaluation framework for advanced mathematical reasoning. https://github.com/GAIR-NLP/ AIME-Preview. GitHub repository. ↗
  62. Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, and 1 others. 2025b. Thoughts are all over the place: On the underthinking of o1-like llms. arXiv preprint arXiv:2501.18585. find
  63. Yijiong Yu, Huiqiang Jiang, Xufang Luo, Qianhui Wu, Chin-Yew Lin, Dongsheng Li, Yuqing Yang, Yongfeng Huang, and Lili Qiu. 2024. Mitigate position bias in large language models via scaling a single dimension. In First Workshop on Long-Context Foundation Models@ ICML 2024. find
  64. Zhengxiang Wang, Jordan Kodner, and Owen Rambow. 2025c. Evaluating LLMs with multiple problems at once. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²), pages 178–199, Vienna, Austria and virtual meeting. Association for Computational Linguistics. find
  65. Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. 2025. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. Preprint, arXiv:2503.18892.
  66. Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. 2024. Marco-o1: Towards open reasoning models for open-ended solutions. Preprint, arXiv:2411.14405. find
  67. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, and 1 others. 2024. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. In The Twelfth International Conference on Learning Representations.
  68. Xiliang Zhu, Shi Zong, and David Rossouw. 2025. How accurate are llms at multi-question answering on conversational transcripts? arXiv preprint arXiv:2509.21732. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pan-2026-rest-stress-testing-large-reasoning/