Elamur Research⌕
☰
home / papers / 2024 / The Limits of Inference Scaling Through Resampling

The Limits of Inference Scaling Through Resampling

Paper · 2024 · Benedikt Stroebl, Sayash Kapoor and Arvind Narayanan

Summary

Resampling is a popular way to spend more compute at inference time: a model keeps generating candidates until one passes a verifier, such as unit tests. Many hope this lets cheap, weaker models catch up with stronger ones. Stroebl, Kapoor and Narayanan (Princeton) argue that this hope fails when the verifier is imperfect. Wrong solutions that still pass the tests (false positives) cannot be filtered out by drawing more samples, so accuracy hits a ceiling that no compute budget can raise. The authors use the original HumanEval and MBPP unit tests as verifiers. They then check the accepted solutions against the extended suites of HumanEval+ and MBPP+. Weaker models turn out to produce false positives much more often, so many of them stay below GPT-4o's Pass@1 even with unlimited samples. Once false positives are given a cost, the best number of attempts becomes small. The work matters for evaluation practice, for verifier research, and for building training data for reasoning models through rejection sampling.

Key points

  • Correlation: On HumanEval+ and MBPP+, a model's single-sample accuracy is strongly linked to its false-positive rate under the weaker original tests. Many weak models cannot reach GPT-4o's single-call accuracy through resampling, however many samples they draw.
  • Optimal attempts: When false positives carry a cost, the optimal number of samples K is usually below 10. It is often K ≤ 5, and it can drop to zero if an error costs more than a correct answer gains. A theoretical model in Appendix C supports this result.
  • Code quality: False-positive solutions score worse than true positives on every model and every readability metric tested. The metrics were snake_case/camelCase naming, line comments, line length, and function length.
  • Hidden gaps: Using HumanEval/MBPP as both verifier and benchmark would hide these gaps and could produce misleading model rankings.
  • Training implications: Imperfect verifiers can let mislabeled examples into datasets curated for reasoning models. Models trained against such verifiers may also learn to exploit the verifier's weaknesses.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Princeton UniversityPrinceton · United States137 works · 53,849 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 worksSoftware · 150 works

Cite

Cite as Benedikt Stroebl, Sayash Kapoor and Arvind Narayanan. 2024. The Limits of Inference Scaling Through Resampling. arXiv:2411.17501.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2411.17501 · arXiv 2411.17501

@misc{stroebl2024limits,
  title = {{The Limits of Inference Scaling Through Resampling}},
  author = {Benedikt Stroebl, Sayash Kapoor and Arvind Narayanan},
  year = {2024},
  doi = {10.48550/arxiv.2411.17501},
  eprint = {2411.17501},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2411.17501}
}

References

73 references, 43 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in AI Safety, July 2016. URL http://arxiv.org/abs/1606.06565. arXiv:1606.06565. ↗
  2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program Synthesis with Large Language Models, August 2021. URL http://arxiv.org/abs/2108.07732. arXiv:2108.07732 [cs]. ↗
  3. Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An Open Language Model For Mathematics, March 2024. URL http://arxiv.org/abs/2310.10631. arXiv:2310.10631. ↗
  4. Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning, June 2024. URL http://arxiv.org/abs/2406.11896. arXiv:2406.11896. — [arXiv:2406.11896](https://arxiv.org/abs/2406.11896) ↗
  5. Bespoke Labs. Bespoke-stratos: The unreasonable effectiveness of reasoning distillation. https://www.bespokelabs.ai/blog/ bespoke-stratos-the-unreasonable-effectiveness-of-reasoning-distillation, 2025. Accessed: 2025-01-22. ↗
  6. Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, URL: https://github.com/benediktstroebl/inference-scaling-limits Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Ch… ↗
  7. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling, July 2024. URL http://arxiv.org/abs/2407.21787. arXiv:2407.21787 [cs]. ↗
  8. Jürgen Börstler, Kwabena E. Bennin, Sara Hooshangi, Johan Jeuring, Hieke Keuning, Carsten Kleiner, Bonnie MacKellar, Rodrigo Duran, Harald Störrle, Daniel Toll, and Jelle van Assema. Developers talking about code quality. Empirical Software Engineering, 28(6):128, September 2023. ISSN 1573-7616. doi… — [doi:10.1007/s10664-023-10381-0](https://doi.org/10.1007/s10664-023-10381-0) find
  9. Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. CodeT: Code Generation with Generated Tests, November 2022. URL http://arxiv.org/ abs/2207.10397. arXiv:2207.10397. ↗
  10. Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou. Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems, March 2024a. URL http://arxiv.org/abs/2403.02419. arXiv:2403.02419 [cs, eess]. ↗
  11. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — Evaluating Large Language Models Trained on Code
  12. Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. ChatUniTest: A Framework for LLM-Based Test Generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, FSE 2024, pp. 572–576, New York, NY, USA, July 2024b.… — [doi:10.1145/3663529.3663801](https://doi.org/10.1145/3663529.3663801) find
  13. Ziru Chen, Michael White, Raymond Mooney, Ali Payani, Yu Su, and Huan Sun. When is Tree Search Useful for LLM Planning? It Depends on the Discriminator, June 2024c. URL http://arxiv.org/abs/2402.10890. arXiv:2402.10890. ↗
  14. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems, November 2021. URL http://arxiv.org/abs/2110.14168. arXiv:2110… ↗
  15. Jonathan Cook, Tim Rocktäschel, Jakob Foerster, Dennis Aumiller, and Alex Wang. TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation, October 2024. URL http://arxiv.org/abs/2410.03608. arXiv:2410.03608. ↗
  16. Jared Quincy Davis, Boris Hanin, Lingjiao Chen, Peter Bailis, Ion Stoica, and Matei Zaharia. Networks of Networks: Complexity Class Principles Applied to Compound AI Systems Design, July 2024. URL http://arxiv.org/abs/2407.16831. arXiv:2407.16831 [cs]. — [arXiv:2407.16831](https://arxiv.org/abs/2407.16831) ↗
  17. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chengg… — DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  18. Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of-Verification Reduces Hallucination in Large Language Models, September 2023. URL http://arxiv.org/abs/2309.11495. arXiv:2309.11495. ↗
  19. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez,… — The Llama 3 Herd of Models
  20. Emily First, Markus N. Rabe, Talia Ringer, and Yuriy Brun. Baldur: Whole-Proof Generation and Repair with Large Language Models, March 2023. URL http://arxiv.org/abs/2303. 04910. arXiv:2303.04910. ↗
  21. Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization, 2022. URL https://arxiv.org/abs/2210.10760. ↗
  22. Sumit Gulwani, Oleksandr Polozov, and Rishabh Singh. Program Synthesis. Foundations and Trends® in Programming Languages, 4(1-2):1–119, July 2017. ISSN 2325-1107, 2325-1131. doi: 10.1561/2500000010. URL https://www.nowpublishers.com/article/Details/ PGL-010. Publisher: Now Publishers, Inc. ↗
  23. Atharva Gundawar, Karthik Valmeekam, Mudit Verma, and Subbarao Kambhampati. Robust Planning with Compound LLM Architectures: An LLM-Modulo Approach, November 2024. URL http://arxiv.org/abs/2411.14484. arXiv:2411.14484. — [arXiv:2411.14484](https://arxiv.org/abs/2411.14484) ↗
  24. Michael Hassid, Tal Remez, Jonas Gehring, Roy Schwartz, and Yossi Adi. The Larger the Better? Improved LLM Code-Generation via Budget Reallocation, July 2024. URL http://arxiv.org/abs/2404.00725. arXiv:2404.00725 [cs]. — [arXiv:2404.00725](https://arxiv.org/abs/2404.00725) ↗
  25. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models, June 2024. URL http://arxiv.org/abs/2401.13919. arXiv:2401.13919. ↗
  26. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding, January 2021. URL http://arxiv.org/abs/2009.03300. arXiv:2009.03300 [cs]. ↗
  27. Arian Hosseini, Alessandro Sordoni, Daniel Toyama, Aaron Courville, and Rishabh Agarwal. Not All LLM Reasoners Are Created Equal, October 2024a. URL http://arxiv.org/abs/2410. 01748. arXiv:2410.01748 [cs]. — [arXiv:2410.01748](https://arxiv.org/abs/2410.01748) ↗
  28. Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-STaR: Training Verifiers for Self-Taught Reasoners, August 2024b. URL http://arxiv.org/abs/2402.06457. arXiv:2402.06457 [cs]. ↗
  29. Yinya Huang, Xiaohan Lin, Zhengying Liu, Qingxing Cao, Huajian Xin, Haiming Wang, Zhenguo Li, Linqi Song, and Xiaodan Liang. MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Data, May 2024. URL http://arxiv.org/abs/2402.08957. arXiv:2402.08957. ↗
  30. Naman Jain, Tianjun Zhang, Wei-Lin Chiang, Joseph E. Gonzalez, Koushik Sen, and Ion Stoica. LLM-Assisted Code Cleaning For Training Accurate Code Generators, November 2023. URL http://arxiv.org/abs/2311.14904. arXiv:2311.14904. ↗
  31. Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Kaya Stechly, Mudit Verma, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy. LLMs Can’t Plan, But Can Help Planning in LLM-Modulo Frameworks, February 2024. URL http://arxiv.org/abs/2402.01817. arXiv:2402.01817 [cs]. ↗
  32. Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI Agents That Matter, July 2024. URL http://arxiv.org/abs/2407.01502. arXiv:2407.01502 [cs]. ↗
  33. Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda. Prover-Verifier Games improve legibility of LLM outputs, August 2024. URL http://arxiv.org/abs/2407.13692. arXiv:2407.13692 [cs]. — [arXiv:2407.13692](https://arxiv.org/abs/2407.13692) ↗
  34. Victoria Krakovna, Laurent Orseau, Richard Ngo, Miljan Martic, and Shane Legg. Avoiding Side Effects By Considering Future Tasks, October 2020. URL http://arxiv.org/abs/2010. 07877. arXiv:2010.07877. — [arXiv:2010.07877](https://arxiv.org/abs/2010.07877) ↗
  35. Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7B Language Models Already Possess Strong Math Capabilities, March 2024a. URL http://arxiv.org/abs/2403.04706. arXiv:2403.04706. ↗
  36. Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. More Agents Is All You Need, February 2024b. URL http://arxiv.org/abs/2402.05120. arXiv:2402.05120 [cs]. ↗
  37. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… find
  38. Zhenwen Liang, Ye Liu, Tong Niu, Xiangliang Zhang, Yingbo Zhou, and Semih Yavuz. Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification, October 2024. URL http://arxiv.org/abs/2410.05318. arXiv:2410.05318. — [arXiv:2410.05318](https://arxiv.org/abs/2410.05318) ↗
  39. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s Verify Step by Step, May 2023. URL http://arxiv.org/abs/2305.20050. arXiv:2305.20050. ↗
  40. Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang. TinyGSM: achieving \>80% on GSM8k with small language models, December 2023a. URL http://arxiv.org/abs/2312.09241. arXiv:2312.09241. ↗
  41. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. November 2023b. URL https://openreview.net/forum?id=1qvx610Cu7. ↗
  42. Yixin Liu, Avi Singh, C. Daniel Freeman, John D. Co-Reyes, and Peter J. Liu. Improving Large Language Model Fine-tuning for Solving Math Problems, October 2023c. URL http://arxiv.org/abs/2310.10047. arXiv:2310.10047. — [arXiv:2310.10047](https://arxiv.org/abs/2310.10047) ↗
  43. Stephan Lukasczyk and Gordon Fraser. Pynguin: automated unit test generation for Python. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, ICSE ’22, pp. 168–172, New York, NY, USA, October 2022. Association for Computing Machinery. ISBN 978-… — [doi:10.1145/3510454.3516829](https://doi.org/10.1145/3510454.3516829) find
  44. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-Refine: Iterative Refinement with Self-… find
  45. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis, February 2023. URL http://arxiv.org/abs/2203.13474. arXiv:2203.13474. ↗
  46. NovaSky Team. Sky-t1: Train your own o1 preview model within $450. https://novasky-ai.github.io/posts/sky-t1, 2025. Accessed: 2025-01-09. ↗
  47. Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting, March 2024. URL http://arxiv.org/abs/2306.17563. arXiv:2306… ↗
  48. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, J… — Code Llama: Open Foundation Models for Code
  49. Jon Saad-Falcon, Adrian Gamarra Lafuente, Shlok Natarajan, Nahum Maru, Hristo Todorov, Etash Guha, E. Kelly Buchanan, Mayee Chen, Neel Guha, Christopher Ré, and Azalia Mirhoseini. Archon: An Architecture Search Framework for Inference-Time Techniques, October 2024. URL http://arxiv.org/abs/2409.1525… — [arXiv:2409.15254](https://arxiv.org/abs/2409.15254) ↗
  50. Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning, October 2024. URL http://arxiv.org/abs/2410.08146. arXiv:2410.08146. ↗
  51. Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language Agents with Verbal Reinforcement Learning, October 2023. URL http://arxiv.org/abs/2303.11366. arXiv:2303.11366 [cs]. ↗
  52. Mohammed Latif Siddiq, Joanna C. S. Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinicius Carvalho Lopes. Using Large Language Models to Generate JUnit Tests: An Empirical Study, March 2024. URL http://arxiv.org/abs/2305.00418. arXiv:2305.00418. — [arXiv:2305.00418](https://arxiv.org/abs/2305.00418) ↗
  53. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters, August 2024. URL http://arxiv.org/abs/2408.03314. arXiv:2408.03314 [cs]. ↗
  54. Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism, July 2024. URL http://arxiv.org/abs/2407.10457. arXiv:2407.10457. ↗
  55. Amitayush Thakur, George Tsoukalas, Yeming Wen, Jimmy Xin, and Swarat Chaudhuri. An In-Context Learning Agent for Formal Theorem-Proving, August 2024. URL http://arxiv.org/abs/2310.04353. arXiv:2310.04353. ↗
  56. Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. Unit Test Case Generation with Transformers and Focal Context, May 2021. URL http://arxiv.org/abs/2009.05617. arXiv:2009.05617. — [arXiv:2009.05617](https://arxiv.org/abs/2009.05617) ↗
  57. Robert Vacareanu, Anurag Pratik, Evangelia Spiliopoulou, Zheng Qi, Giovanni Paolini, Neha Anna John, Jie Ma, Yassine Benajiba, and Miguel Ballesteros. General Purpose Verification for Chain of Thought Prompting, April 2024. URL http://arxiv.org/abs/2405.00204. arXiv:2405.00204 [cs]. — [arXiv:2405.00204](https://arxiv.org/abs/2405.00204) ↗
  58. Evan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song, Vaskar Nath, Ziwen Han, Sean Hendryx, Summer Yue, and Hugh Zhang. Planning In Natural Language Improves LLM Search For Code Generation, September 2024a. URL http://arxiv.org/abs/2409.03733. arXiv:2409.03733 [cs]. ↗
  59. Haiming Wang, Huajian Xin, Chuanyang Zheng, Lin Li, Zhengying Liu, Qingxing Cao, Yinya Huang, Jing Xiong, Han Shi, Enze Xie, Jian Yin, Zhenguo Li, Heng Liao, and Xiaodan Liang. LEGO-Prover: Neural Theorem Proving with Growing Libraries, October 2023a. URL http://arxiv.org/abs/2310.00656. arXiv:2310… ↗
  60. Han Wang, Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. Soft Self-Consistency Improves Language Model Agents, June 2024b. URL http://arxiv.org/abs/2402.13212. arXiv:2402.13212. ↗
  61. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-Consistency Improves Chain of Thought Reasoning in Language Models, March 2023b. URL http://arxiv.org/abs/2203.11171. arXiv:2203.11171. ↗
  62. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, January 2023. URL http://arxiv.org/abs/2201.11903. arXiv:2201.11903. ↗
  63. Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui. From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models, June 2024. URL http://arxiv.org/abs/2406.16838. arXiv:2406.16838 [cs]. ↗
  64. Tianhao Wu, Janice Lan, Weizhe Yuan, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Thinking LLMs: General Instruction Following with Thought Generation, October 2024. URL http://arxiv.org/abs/2410.10630. arXiv:2410.10630. — [arXiv:2410.10630](https://arxiv.org/abs/2410.10630) ↗
  65. Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data, May 2024. URL http://arxiv.org/abs/2405.14333. arXiv:2405.14333. ↗
  66. Kaiyu Yang, Aidan M. Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar. LeanDojo: Theorem Proving with Retrieval-Augmented Language Models, October 2023. URL http://arxiv.org/abs/2306. 15626. arXiv:2306.15626. ↗
  67. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate Problem Solving with Large Language Models, December 2023. URL http://arxiv.org/abs/2305.10601. arXiv:2305.10601. ↗
  68. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. $\\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, June 2024. URL — $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains http://arxiv.org/ abs/2406.12045. arXiv:2406.12045 [cs]. ↗
  69. Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative Verifiers: Reward Modeling as Next-Token Prediction, August 2024a. URL http://arxiv.org/abs/2408.15240. arXiv:2408.15240 [cs]. ↗
  70. Shudan Zhang, Hanlin Zhao, Xiao Liu, Qinkai Zheng, Zehan Qi, Xiaotao Gu, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Prompts, May 2024b. URL http://arxiv.org/abs/2405.04520. arXiv:2405.04520. — [arXiv:2405.04520](https://arxiv.org/abs/2405.04520) ↗
  71. Jiasheng Zheng, Boxi Cao, Zhengzhao Ma, Ruotong Pan, Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models, October 2024. URL http://arxiv.org/abs/2407.11470. arXiv:2407.11470. — [arXiv:2407.11470](https://arxiv.org/abs/2407.11470) ↗
  72. Li Zhong, Zilong Wang, and Jingbo Shang. Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step, June 2024. URL http://arxiv.org/abs/ 2402.16906. arXiv:2402.16906. ↗
  73. Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jürgen Schmidhuber. Agent-as-a-Judge: Evaluate Agents with Agents, October 2024. URL http://arxiv.org/abs/… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/stroebl-2024-limits-inference-scaling-through-resampling/