Elamur Research⌕
☰
home / papers / 2026 / Quantifying the Accuracy and Cost Impact of Design Decisions in Budget-Constrained Agentic LLM Search

Quantifying the Accuracy and Cost Impact of Design Decisions in Budget-Constrained Agentic LLM Search

Summary

Agentic RAG systems are usually evaluated with generous tool use, but real deployments cap search calls and generated tokens. This paper measures how common design choices affect accuracy and cost once those limits are fixed. The authors use BCAS (Budget-Constrained Agentic Search), a model-agnostic harness that shows the model its remaining search and token allowance at each step, removes the search tool when the cap is reached, and logs per-question consumption. With budgets held constant, they toggle one component at a time: hybrid lexical/dense retrieval, re-ranking, pre-planning, reflection and completion limits. Experiments cover six LLMs and three multi-hop QA benchmarks, all using one shared prompt template with no per-model tuning. Accuracy rises with extra searches only up to a small cap. Hybrid retrieval with lightweight re-ranking gives the largest average gain, and bigger completion budgets help mainly on HotpotQA-style synthesis questions. The authors frame the work as controlled systems measurement rather than a new state-of-the-art agent. It offers practical configuration guidance, and the code and settings are released for reproduction.

Key points

  • BCAS has two budget levers, max_searches and max_total_tokens. Token use is updated each turn from provider-reported usage, and the loop ends when the model calls ready_to_answer or a budget runs out.
  • Three retrieval setups are compared:
    • a BM25 baseline with phrase boosting for two- and three-term phrases;
    • Hybrid Search (BM25 plus dense vectors);
    • HS+RR, which takes 100 hybrid candidates, re-scores them with a cross-encoder and keeps the top five.
  • The retrieval backend is built on ParadeDB, a PostgreSQL extension. Optional pre-planning and reflection modules follow ReAct, Self-RAG and Self-Reasoning patterns.
  • An illustrative run shows DeepSeek V3 (0324) solving a hard HotpotQA question with planning under a 4-search budget.
  • Code, prompts, dataset split hashes and evaluation outputs are available at github.com/kmccleary3301/BCAS_RAG.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3wfsiry9yjog

@inproceedings{mccleary2026quantifying,
  title = {{Quantifying the Accuracy and Cost Impact of Design Decisions in Budget-Constrained Agentic LLM Search}},
  author = {McCleary, Kyle A. and Ghawaly, James M.},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3wfsiry9yjog},
  url = {https://doi.org/10.63317/3wfsiry9yjog}
}

References

30 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2024. Introducing claude’s research mode. https://www.anthropic.com/research. Accessed: 2025-07-01. ↗
  2. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-rag: Self-reflective retrieval-augmented generation. In International Conference on Learning Representations (ICLR). find
  3. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. find
  4. Charles L.A. Clarke, Gordon V. Cormack, and Elizabeth A. Tudhope. 2000. Relevance ranking for one to three term queries. Information Processing & Management, 36(2):291–311. find
  5. Alibaba Cloud. 2024. Qwen technical report. Accessed: 2025-01-15. find
  6. DeepSeek. 2024. Deepseek-v3 technical report. Accessed: 2025-01-15. find
  7. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. find
  8. Gemini Team. 2025. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. Technical report, Google DeepMind. Accessed: 2025-07-01. find our card
  9. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, Barcelona, Spain (Online). International Comm… find our card
  10. Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874–880. find our card
  11. P. Jiang, J. Lin, L. Cao, R. Tian, S. Kang, Z. Wang, J. Sun, and J. Han. 2025. Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning. arXiv preprint arXiv:2503.00223. — [arXiv:2503.00223](https://arxiv.org/abs/2503.00223) find
  12. Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  13. B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516. find our card
  14. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada. Association for Computationa… find our card
  15. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Proc… find
  16. Chaofan Li, Zheng Liu, Shitao Xiao, and Yingxia Shao. 2023. Making large language models a better foundation for dense retrieval. find
  17. Robert G. Newcombe. 1998. Improved confidence intervals for the difference between binomial proportions based on paired data. Statistics in Medicine, 17(22):2635–2650. find
  18. Philippe Noël. 2024. Paradedb. https://github.com/paradedb/paradedb. Accessed: 2024-7-01. ↗
  19. OpenAI. 2023. Function calling and other api updates. https://platform.openai.com/docs/guides/function-calling. Accessed: 2025-07-01. ↗
  20. OpenAI. 2024. Introducing gpt-4.1 in the api. Accessed: 2025-07-04. find
  21. OpenAI. 2025. Openai o3 system card. Accessed: 2025-06-23. find
  22. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023. find
  23. A. Singh et al. 2024. Agentic retrieval-augmented generation: A survey on agentic rag. arXiv preprint arXiv:2501.09136. — [arXiv:2501.09136](https://arxiv.org/abs/2501.09136) find
  24. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. find
  25. Edwin B Wilson. 1927. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209–212. find
  26. Yuan Xia, Jingbo Zhou, Zhenhui Shi, Jun Chen, and Haifeng Huang. 2025. Improving retrieval augmented language model with self-reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, 39(24):25534–25542. find
  27. Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. Crag: Corrective retrieval augmented generation. arXiv preprint arXiv:2401.15884. find our card
  28. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  29. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR). find our card
  30. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/mccleary-2026-quantifying-accuracy-cost-impact-design/