Elamur Research⌕
☰
home / papers / 2026 / ToolDreamer: Instilling LLM Reasoning Into Tool Retrievers

ToolDreamer: Instilling LLM Reasoning Into Tool Retrievers

Paper · 2026 · Saptarshi Sengupta, Zhengyu Zhou, Jun Araki et al.

Summary

LLMs that call tools cannot fit very large tool inventories into their context window, so an external retriever must pick candidate tools. Standard retrievers match the user query against tool descriptions. This fails when a request only implies the needed tool, for example a picnic plan that really calls for a weather lookup. ToolDreamer addresses this mismatch. An LLM first imagines plausible tools for the query, producing reasoning, a name and a description for each, but no code. During training, these imagined tools are paired with the real gold tools through bipartite graph matching. The resulting data trains retrievers with an augmented InfoNCE objective that uses both the query and the hypothetical tools. At inference, results retrieved for each hypothetical tool are merged, by default with RRF. Experiments on ToolRet show gains for sparse and dense retrievers, both untrained and fine-tuned. The approach moves part of the reasoning burden into retrieval while remaining cheap. It also works with open-source generators, reducing dependence on paid APIs.

Key points

  • Hypothetical tools are generated with GPT-4.1, which is asked for as many tools as the query has gold tools. About 0.3% of questions where the count was wrong were discarded.
  • Replacing RRF with GPT-4.1 list fusion raised NDCG@10 of the trained Qwen3 retriever from 42.47 (QTND/RRF) to about 46.5. The authors still default to RRF because it is deterministic and avoids extra cost and hallucinations.
  • Using Qwen3-32B instead of GPT-4.1 as the generator costs little accuracy on the ToolRet code split:
    • BM25 NDCG@10 drops from 36.50 to 35.27.
    • NV-Embed-2 NDCG@10 drops from 47.41 to 46.11.
  • Running the full framework cost under $5 with OpenAI batch mode.
  • Latency per query rises from ~0.04 s for plain retrieval to roughly 3–8.5 s, depending on the generator.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Pennsylvania State UniversityState College · United States56 works · 5,360 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksInformation retrieval & search · 473 works

Cite

Cite as Saptarshi Sengupta, Zhengyu Zhou, Jun Araki, Xingbo Wang, Bingqing Wang, Suhang Wang and Zhe Feng. 2026. ToolDreamer: Instilling LLM Reasoning Into Tool Retrievers. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.254

@inproceedings{sengupta2026tooldreamer,
  title = {{ToolDreamer: Instilling LLM Reasoning Into Tool Retrievers}},
  author = {Saptarshi Sengupta, Zhengyu Zhou, Jun Araki, Xingbo Wang, Bingqing Wang, Suhang Wang and Zhe Feng},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.254},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.254}
}

References

64 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Norbert Braunschweiler, Rama Doddipatla, and Tudor-catalin Zorila. 2025. ToolReAGt: Tool retrieval for LLM-based complex task solution via retrieval augmented generation. In Proceedings of the 3rd Workshop on Towards Knowledgeable Foundation Models (KnowFM), pages 75–83, Vienna, Austria. Association… find
  2. Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. 2024. Large language models as tool makers. In The Twelfth International Conference on Learning Representations. find
  3. Matthew Carrigan. 2024. Tool Use, Unified — huggingface.co. https://huggingface.co/blog/ unified-tool-use. ↗
  4. QiHong Chen, Jiachen Yu, Jiawei Li, Jiecheng Deng, Justin Tian Jin Chen, and Iftekhar Ahmed. 2024a. A deep dive into large language model code generation mistakes: What and why? Computing Research Repository, arXiv:2411.01414. Version 2. — [arXiv:2411.01414](https://arxiv.org/abs/2411.01414) find
  5. Yanfei Chen, Jinsung Yoon, Devendra Singh Sachan, Qingze Wang, Vincent Cohen-Addad, Mohammadhossein Bateni, Chen-Yu Lee, and Tomas Pfister. 2024b. Re-invoke: Tool invocation rewriting for zero-shot tool retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4705–4… find
  6. Jooyoung Choi, Hyun Kim, Hansol Jang, Changwook Jun, Kyunghoon Bae, Hyewon Choi, Stanley Jungkyu Choi, Honglak Lee, and Chulmin Yun. 2025. Lgai-embedding-preview technical report. ArXiv, abs/2506.07438. — [arXiv:2506.07438](https://arxiv.org/abs/2506.07438) find
  7. Gordon V. Cormack, Charles L A Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’09, page 758–759, New York,…
  8. Debrup Das, Debopriyo Banerjee, Somak Aditya, and Ashish Kulkarni. 2024. MATHSENSEI: A tool-augmented large language model for mathematical reasoning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vo… find
  9. Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Diganta Misra, Shreeya Dhakal, Jonathan Rystrøm, Roman Solomatin, Ömer Veysel… — MMTEB: Massive Multilingual Text Embedding Benchmark
  10. Revanth Gangi Reddy, JaeHyeok Doo, Yifei Xu, Md Arafat Sultan, Deevya Swain, Avirup Sil, and Heng Ji. 2024. FIRST: Faster improved listwise reranking with single token decoding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8642–8652, Miami, Florida… find
  11. Jingtong Gao, Bo Chen, Xiangyu Zhao, Weiwen Liu, Xiangyang Li, Yichao Wang, Wanyu Wang, Huifeng Guo, and Ruiming Tang. 2025. Llm4rerank: Llm-based auto-reranking framework for recommendations. In Proceedings of the ACM on Web Conference 2025, WWW ’25, page 228–239, New York, NY, USA. Association for… find
  12. Yilin Geng, Haonan Li, Honglin Mu, Xudong Han, Timothy Baldwin, Omri Abend, Eduard H. Hovy, and Lea Frermann. 2025. Control illusion: The failure of instruction hierarchies in large language models. CoRR, abs/2502.15851. — [arXiv:2502.15851](https://arxiv.org/abs/2502.15851) find
  13. Zhichao Geng, Dongyu Ru, and Yang Yang. 2024. Towards competitive search relevance for inference-free learned sparse retrievers. CoRR, abs/2411.04403. — [arXiv:2411.04403](https://arxiv.org/abs/2411.04403) find
  14. Michael Günther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mohammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, and 1 others. 2023. Jina embeddings 2: 8192-token general-purpose text embeddings for long documents. arXiv preprint arXiv:2310.19923.
  15. Keno Harada, Yudai Yamazaki, Masachika Taniguchi, Takeshi Kojima, Yusuke Iwasawa, and Yutaka Matsuo. 2025. Curse of instructions: Large language models cannot follow multiple instructions at once. find
  16. Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. arXiv preprint arXiv:1705.00652. find
  17. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  18. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst., 43(2).
  19. Shijue Huang, Wanjun Zhong, Jianqiao Lu, Qi Zhu, Jiahui Gao, Weiwen Liu, Yutai Hou, Xingshan Zeng, Yasheng Wang, Lifeng Shang, Xin Jiang, Ruifeng Xu, and Qun Liu. 2024. Planning, creation, usage: Benchmarking LLMs for comprehensive tool utilization in real-world complex scenarios. In Findings of the… find
  20. HuggingFace. 2025. LoRA (Low-Rank Adaptation) - Hugging Face LLM Course — huggingface.co. https://huggingface.co/learn/llm-course/ en/chapter11/4\#using-trl-with-peft. [Accessed 24-09-2025]. ↗
  21. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research.
  22. Mohammad Kachuee, Sarthak Ahuja, Vaibhav Kumar, Puyang Xu, and Xiaohu Liu. 2025. Improving tool retrieval by leveraging large language models for query generation. In Proceedings of the 31st International Conference on Computational Linguistics: Industry Track, pages 29–38, Abu Dhabi, UAE. Associati… find
  23. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Onli…
  24. Besat Kassaie, Andrew Kane, and Frank Wm. Tompa. 2025. Exploiting query reformulation and reciprocal rank fusion in math-aware search engines. In Proceedings of the 2025 ACM Symposium on Document Engineering, DocEng ’25, New York, NY, USA. Association for Computing Machinery. find
  25. H. W. Kuhn. 1955. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83–97. find
  26. Yuan-Ching Kuo, Yi Yu, Chih-Ming Chen, and Chuan-Ju Wang. 2025. MMLF: Multi-query multi-passage late fusion retrieval. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 6587–6598, Albuquerque, New Mexico. Association for Computational Linguistics. find
  27. Carlos Lassance, Hervé Déjean, Thibault Formal, and Stéphane Clinchant. 2024. Splade-v3: New baselines for splade. Preprint, arXiv:2403.06789. — [arXiv:2403.06789](https://arxiv.org/abs/2403.06789) find
  28. Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2025. NV-embed: Improved techniques for training LLMs as generalist embedding models. In The Thirteenth International Conference on Learning Representations.
  29. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
  30. Do Xuan Long, Duy Dinh, Ngoc-Hai Nguyen, Kenji Kawaguchi, Nancy F. Chen, Shafiq Joty, and Min-Yen Kan. 2025. What makes a good natural language prompt? In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July… find
  31. Xing Han Lù. 2024. Bm25s: Orders of magnitude faster lexical search via eager sparse scoring. Preprint, arXiv:2407.03618. — [arXiv:2407.03618](https://arxiv.org/abs/2407.03618) find
  32. Tula Masterman, Sandi Besen, Mason Sawtell, and Alex Chao. 2024. The landscape of emerging ai agent architectures for reasoning, planning, and tool calling: A survey. ArXiv, abs/2404.11584. find
  33. Microsoft. 2025. Azure OpenAI in Azure AI Foundry Models Quotas and Limits — learn.microsoft.com. https://learn.microsoft.com/en-us/azure/ ai-foundry/openai/quotas-limits?tabs=REST. [Accessed 05-10-2025]. ↗
  34. OpenAI. 2024. text-embedding-3-small. https://platform.openai.com/docs/models/ text-embedding-3-small. [Accessed 23-09-2025]. ↗
  35. OpenAI. 2025. Introducing GPT-4.1 in the API — openai.com. https://openai.com/index/gpt-4-1/. [Accessed 12-07-2025]. ↗
  36. Avinash Patil and Aryan Jadon. 2025. Advancing reasoning in large language models: Promising methods and approaches. arXiv preprint arXiv:2502.03671. — [arXiv:2502.03671](https://arxiv.org/abs/2502.03671) find
  37. Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2024. Gorilla: Large language model connected with massive APIs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  38. Cheng Qian, Chi Han, Yi Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. 2023. CREATOR: Tool creation for disentangling abstract and concrete reasoning of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6922–6939, Singapore. Association for Computatio…
  39. Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. 2024. Towards completeness-oriented tool retrieval for large language models. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, CIKM ’24, page 1930–194… find
  40. Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. 2025. Tool learning with large language models: A survey. Frontiers of Computer Science, 19(8):198343.
  41. Saron Samuel, Dan DeGenaro, Jimena Guallar-Blasco, Kate Sanders, Seun Eisape, Arun Reddy, Alexander Martin, Andrew Yates, Eugene Yang, Cameron Carpenter, David Etter, Efsun Kayi, Matthew Wiesner, Kenton Murray, and Reno Kriz. 2025. Mmmorrf: Multimodal multilingual modularized reciprocal rank fusion.… find
  42. SentenceTransformers. 2025. Training Overview; Sentence Transformers documentation — sbert.net. https://sbert.net/docs/sentence_ transformer/training_overview.html\# training-arguments. [Accessed 24-09-2025]. ↗
  43. Tao Shen, Guodong Long, Xiubo Geng, Chongyang Tao, Yibin Lei, Tianyi Zhou, Michael Blumenstein, and Daxin Jiang. 2024. Retrieval-augmented retrieval: Large language models are strong zero-shot retriever. In Findings of the Association for Computational Linguistics: ACL 2024, pages 15933–15946, Bangk…
  44. Zhengliang Shi, Yuhan Wang, Lingyong Yan, Pengjie Ren, Shuaiqiang Wang, Dawei Yin, and Zhaochun Ren. 2025. Retrieval models aren’t tool-savvy: Benchmarking tool retrieval for large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 24497–24524, Vienna, Aus… find
  45. Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. 2025. The good, the bad, and the greedy: Evaluation of LLMs should not ignore non-determinism. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Techno…
  46. Weiwei Sun, Zhengliang Shi, Wu Jiu Long, Lingyong Yan, Xinyu Ma, Yiding Liu, Min Cao, Dawei Yin, and Zhaochun Ren. 2024. MAIR: A massive benchmark for evaluating instructed retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14044–14067, Miami,… find
  47. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). find
  48. Yongqi Tong, Dawei Li, Sizhe Wang, Yujia Wang, Fei Teng, and Jingbo Shang. 2024. Can LLMs learn from previous mistakes? investigating LLMs’ errors to boost for reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3065– 3…
  49. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2019. Representation learning with contrastive predictive coding. Preprint, arXiv:1807.03748.
  50. Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2021. MiniLMv2: Multi-head self-attention relation distillation for compressing pre-trained transformers. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2140–2151, Online. Association for Computation… find
  51. Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, and Luca Soldaini. 2025. FollowIR: Evaluating and teaching information retrieval models to follow instructions. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Ass…
  52. Georg Wölflein, Dyke Ferber, Daniel Truhn, Ognjen Arandjelovic, and Jakob Nikolas Kather. 2025. LLM agents making agent tools. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26092–26130, Vienna, Austria. Association for Compu…
  53. Mengsong Wu, Tong Zhu, Han Han, Chuanyuan Tan, Xiang Zhang, and Wenliang Chen. 2025. Seal-tools: Self-instruct tool learning dataset for agent tuning and detailed benchmark. In Natural Language Processing and Chinese Computing, pages 372–384, Singapore. Springer Nature Singapore. find
  54. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 641–649, New York…
  55. Qiancheng Xu, Yongqi Li, Heming Xia, and Wenjie Li. 2024. Enhancing tool retrieval with iterative feedback from large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9609–9619, Miami, Florida, USA. Association for Computational Linguistics. find
  56. Zhichao Xu, Fengran Mo, Zhiqi Huang, Crystina Zhang, Puxuan Yu, Bei Wang, Jimmy Lin, and Vivek Srikumar. 2025. A survey of model architectures in information retrieval. CoRR, abs/2502.14822. — [arXiv:2502.14822](https://arxiv.org/abs/2502.14822) find
  57. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  58. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. find
  59. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations.
  60. Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. 2024. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, 4(2):100211.
  61. Mert Yazan, Suzan Verberne, and Frederik Situmeang. 2024. The impact of quantization on retrieval-augmented generation: An analysis of small llms. In IR-RAG@SIGIR. find
  62. Peng Yu, En Xu, Bin Chen, Haibiao Chen, and Yinfei Xu. 2025. Qzhou-embedding technical report. Preprint, arXiv:2508.21632. — [arXiv:2508.21632](https://arxiv.org/abs/2508.21632) find
  63. Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi Fung, Hao Peng, and Heng Ji. 2024. CRAFT: Customizing LLMs by creating and retrieving from specialized toolsets. In The Twelfth International Conference on Learning Representations. find
  64. Prompting large language models to generate dense and sparse representations for zero-shot document retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4375–4391, Miami, Florida, USA. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sengupta-2026-tooldreamer-instilling-llm-reasoning-into/