Elamur Research⌕
☰
home / papers / 2026 / Tool-Aware Planning for Contact-Center Analytics: Evaluating LLMs through Lineage-Guided Query Decomposition

Tool-Aware Planning for Contact-Center Analytics: Evaluating LLMs through Lineage-Guided Query Decomposition

Paper · 2026 · Varun Nathan, Shreyas Guha and Ayush Kumar

Summary

The paper tackles the evaluation of LLM planners for business-insight questions in contact centers. Answering such questions means breaking them into executable steps across three fixed tools: T2S (Text-to-SQL over Snowflake), RAG over call transcripts, and an LLM for synthesis. Each plan is a JSON DAG whose depends_on links let independent steps run in parallel.

The authors make three contributions:

  • A dual scoring framework. One evaluator rates plans on seven metrics, aggregated to 0–100. A second, one-shot evaluator compares a plan with a reference using precision, recall and F1, then assigns a 7-point quality rating.
  • A data curation pipeline. GPT-4o generates queries that vary in subjectivity and compoundness. An initial plan is drafted and then refined by a stepwise evaluator→optimizer loop, producing a "lineage" of revisions that humans verify.
  • A model study. 14 LLMs are benchmarked, with and without lineage in the prompt.

Even the best models fall short, particularly on compound queries, plans with more than four steps, and matching tools to prompts. The benchmark gives enterprises a practical way to diagnose planning errors that would otherwise go unnoticed in agentic QA systems.

Key points

  • Top aggregate metric-wise score is 84.8 (Claude-3-7-Sonnet). Under the one-shot evaluator, o3-mini places the most plans in the two top tiers (Extremely Good and Very Good, i.e., the A+ rate), yet this share reaches only 49.75%.
  • Including the lineage in the prompt gives inconsistent overall gains, but it helps several strong models and often improves step executability.
  • The refinement loop tags each step with labels such as Incorrect Tool or Complex Prompt, repairs plans locally and globally, and stops after a pass with no changes or after at most 4 passes.
  • Results are reported on an internal 600-query benchmark that stays proprietary. A separate anonymized 200-query set, built on a synthetic test account, is released on GitHub (Observeai-Research/tool-aware-planning-dataset) with reference plans, lineages and outputs from all 14 planners.
  • Main limitations:
    • The evaluation never executes the plans.
    • The tool palette is fixed.
    • Scoring relies on LLM judges, which were validated against human annotations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Networks and Communications · 64 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Varun Nathan, Shreyas Guha and Ayush Kumar. 2026. Tool-Aware Planning for Contact-Center Analytics: Evaluating LLMs through Lineage-Guided Query Decomposition. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.gem-main.73

@inproceedings{nathan2026tool,
  title = {{Tool-Aware Planning for Contact-Center Analytics: Evaluating LLMs through Lineage-Guided Query Decomposition}},
  author = {Varun Nathan, Shreyas Guha and Ayush Kumar},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.73},
  url = {https://doi.org/10.18653/v1/2026.gem-main.73}
}

References

34 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abubakari Alidu, Michele Ciavotta, and Flavio DePaoli. 2025. Prompt2dag: A modular methodology for llm-based data enrichment pipeline generation. Preprint, arXiv:2509.13487. — [arXiv:2509.13487](https://arxiv.org/abs/2509.13487) find
  2. Anthropic. 2024. Claude 3.5 Haiku. https://www.anthropic.com/news/claude-3-5. Accessed: [Date of access]. ↗
  3. Anthropic. 2025. Claude-sonnet-4. Large language model, accessed [date]. find
  4. Pengfei Cao, Tianyi Men, Wencan Liu, Jingwen Zhang, Xuzhao Li, Xixun Lin, Dianbo Sui, Yanan Cao, Kang Liu, and Jun Zhao. 2025. Large language models for planning: A comprehensive and systematic survey. Preprint, arXiv:2505.19683. — [arXiv:2505.19683](https://arxiv.org/abs/2505.19683) find
  5. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. find
  6. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological bulletin, 70(4):213. find
  7. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. Preprint, arXiv:2306.06070.
  8. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin. find
  9. Kai Goebel and Patrik Zips. 2025. Can llm-reasoning models replace classical planning? a benchmark study. Preprint, arXiv:2507.23589. — [arXiv:2507.23589](https://arxiv.org/abs/2507.23589) find
  10. Shicheng Liu, Jialiang Xu, Wesley Tjangnaka, Sina Semnani, Chen Yu, and Monica Lam. 2024. SUQL: Conversational search over structured and unstructured data with large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4535–4555, Mexico City, Mexico. Asso… find
  11. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, and 3 others. 2023. Agentbench: Evaluating llms as agents. Preprint, arXiv:…
  12. Zixian Ma, Weikai Huang, Jieyu Zhang, Tanmay Gupta, and Ranjay Krishna. 2024. m&m’s: A benchmark to evaluate tool-use for multi-step multi-modal tasks. Preprint, arXiv:2403.11085. find
  13. Meta. 2025. Llama 4 maverick. https://ai.meta.com/blog/llama-4-multimodal-intelligence/. Model released as a part of the Llama 4 family. ↗
  14. Kazuma Obata, Tatsuya Aoki, Takato Horii, Tadahiro Taniguchi, and Takayuki Nagai. 2024. Lip-llm: Integrating linear programming and dependency graph with large language models for multi-robot task planning. Preprint, arXiv:2410.21040. — [arXiv:2410.21040](https://arxiv.org/abs/2410.21040) find
  15. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. Gpt-4o system card.…
  16. OpenAI. 2025. Gpt-4.1-mini. API Model. Available from OpenAI API https://openai.com/index/gpt-4-1/. ↗
  17. OpenAI. 2025a. GPT-5 System Card. https://openai.com/index/gpt-5-system-card/. Accessed: 2 January 2026. ↗
  18. Amazon Artificial General Intelligence. 2024. The amazon nova family of models: Technical report and model card. Amazon Technical Reports. find
  19. OpenAI. 2025b. OpenAI o3-mini. https://openai.com/index/openai-o3-mini/. Announced January 31, 2025. ↗
  20. Jungkoo Kang. 2025. Scaling llm planning: Nl2flow for parametric problem generation and rigorous evaluation. Preprint, arXiv:2507.02253. — [arXiv:2507.02253](https://arxiv.org/abs/2507.02253) find
  21. Vishal Pallagani, Bharath Chandra Muppasani, Kaushik Roy, Francesco Fabiano, Andrea Loreggia, Keerthiram Murugesan, Biplav Srivastava, Francesca Rossi, Lior Horesh, and Amit Sheth. 2024. On the prospects of incorporating large language models (llms) in automated planning and scheduling (aps). Procee… find
  22. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. Biometrics. find
  23. Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. API-bank: A comprehensive benchmark for tool-augmented LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3102–3116, Singapore. Ass…
  24. Meta AI Research. 2024. Llama 3: An open-source framework for language modeling. Published via blog posts and official releases, as no specific public paper yet. Refer to the official announcements. The Llama 3.2 models, including the 1B and 3B instruct variants, are a subsequent release built upon… find
  25. Gaurav Sahu, Abhay Puri, Juan Rodriguez, Amirhossein Abaskohi, Mohammad Chegini, Alexandre Drouin, Perouz Taslakian, Valentina Zantedeschi, Alexandre Lacoste, David Vazquez, Nicolas Chapados, Christopher Pal, Sai Rajeswar Mudumba, and Issam Hadj Laradji. 2025. Insightbench: Evaluating business analy… — [arXiv:2407.06423](https://arxiv.org/abs/2407.06423) find
  26. Common notation. Each tool call takes (i) an optional list of identifiers (e.g., call_ids or interaction_ids) passed positionally via a placeholder like “(k)” to denote step-k output, and (ii) a natural-language prompt. Placeholders MUST be used whenever a step depends on prior outputs. A.1.1 T2S: T… — [arXiv:2501.05255](https://arxiv.org/abs/2501.05255) find
  27. Marcus Tantakoun, Xiaodan Zhu, and Christian Muise. 2025. Llms as planning modelers: A survey for leveraging large language models to construct automated planning models. Preprint, arXiv:2503.18971. — [arXiv:2503.18971](https://arxiv.org/abs/2503.18971) find
  28. Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2023. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. Preprint, arXiv:2206.10498.
  29. Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. 2024. Mint: Evaluating llms in multi-turn interaction with tools and language feedback. Preprint, arXiv:2309.10691. 4. Limits. Does not track within-call temporal dynamics (e.g., sentiment shifts); those require R…
  30. A.1.2 RAG: Retrieval-Augmented Generation over Transcripts 1. Signature. RAG(call_ids: list, prompt: str) -\> text + table 2. Function. Retrieves and analyzes customer– agent transcripts, returning (i) a textual answer and (ii) a Data Insights table with columns: Ziting Wang, Shize Zhang, Haitao Yuan… — [arXiv:2509.02473](https://arxiv.org/abs/2509.02473) find
  31. Topic Name, Topic Description • Interaction Ids (list) • Interactions (%) (sample-relative) 3. Sampling. For broad thematic prompts, RAG analyzes a sample (e.g., ∼200 interactions); percentages are with respect to the sample. find
  32. Duo Wu, Jinghe Wang, Yuan Meng, Yanning Zhang, Le Sun, and Zhi Wang. 2025a. Catp-llm: Empowering large language models for cost-aware tool planning. Preprint, arXiv:2411.16313. 4. Strengths. Conversational content, phrasing, subjective judgments, within-call events (e.g., sentiment shifts). — [arXiv:2411.16313](https://arxiv.org/abs/2411.16313) find
  33. Zirui Wu, Xiao Liu, Jiayi Li, Lingpeng Kong, and Yansong Feng. 2025b. Haste makes waste: Evaluating planning abilities of llms for efficient and feasible multitasking with time constraints between actions. Preprint, arXiv:2503.02238. — [arXiv:2503.02238](https://arxiv.org/abs/2503.02238) find
  34. Jerrold H Zar. 2005. Spearman rank correlation. Encyclopedia of Biostatistics, 7. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nathan-2026-tool-aware-planning-contact-center/