Elamur Research⌕
☰
home / papers / 2026 / Transfer Learning for Generalizable Automated LLM Improvement Pipeline for IVR Navigation

Transfer Learning for Generalizable Automated LLM Improvement Pipeline for IVR Navigation

Paper · 2026 · Vishal Sankar Ram, Jason Kushner, Manas Paldhe et al.

Summary

The paper tackles a practical healthcare problem. Calls to health insurers often stall in interactive voice response (IVR) menus that are brittle, payer-specific, and sometimes deliberately obstructive, for example forcing callers through long benefit summaries before they can reach an agent. Writing an LLM prompt by hand for every navigation goal is slow. The authors treat reaching each department as a separate task and build an automated pipeline that moves an optimized prompt from a source task (claims status) to a target task (patient benefit verification) using only a small set of human-labeled examples.

The pipeline works in two stages:

  1. Transfer and optimization. Several LLM modules retrieve similar examples from both tasks, generate hints about how the tasks differ, rewrite and compress the prompt, and check it for hallucinated actions. DSPy optimizers then refine the result.
  2. Clustering and RAG. IVR utterances are grouped into clusters, each cluster gets its own prompt, and instructions and hints are injected dynamically through RAG.

Because everything happens at the prompt level, no fine-tuning is needed. This matters because fine-tuning raises PHI concerns and is costly to maintain. The resulting prompts are far shorter and slightly more accurate than the expert-written baseline.

Key points

  • Accuracy. Cluster-specific prompts reached 82.21% turn-level accuracy, compared with 79.00% for the human-designed prompt. The transfer pipeline with the SIMBA optimizer alone scored 81.39%.
  • Prompt size. The best prompts are about 78% shorter than the human baseline (405 vs. 1,819 words) and have far fewer instructions (17 vs. 47).
  • Prompt structure. Overall, character count fell by 80%, nesting depth dropped from 11 to 3 levels, and conditionals decreased by 46%.
  • Optimizer choice. In the Cluster RAG setting, MIPROv2 (light mode) worked better than SIMBA, whose stochastic exploration caused instruction drift. Cluster RAG uniquely handled timing-sensitive edge cases, such as asking for an agent after a forced benefits loop.
  • Retrieval and generalization. Retrieval uses Vertex AI text-embedding-004 with cosine similarity. On the roughly 25% of evaluation utterances shared across IVR task types, the transferred prompt scored 80.8%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Vishal Sankar Ram, Jason Kushner, Manas Paldhe and Youngseo Son. 2026. Transfer Learning for Generalizable Automated LLM Improvement Pipeline for IVR Navigation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-industry.136

@inproceedings{ram2026transfer,
  title = {{Transfer Learning for Generalizable Automated LLM Improvement Pipeline for IVR Navigation}},
  author = {Vishal Sankar Ram, Jason Kushner, Manas Paldhe and Youngseo Son},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.136},
  url = {https://doi.org/10.18653/v1/2026.acl-industry.136}
}

References

21 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901.
  2. Weize Deng, Yougang Zhao, Jingming Shi, Xiao Chen, Zhe Wang, Ying Qin, Zhiruo Liu, Yuheng Zhang, Dale Schuurmans, and Denny Zhou. 2024. Prewrite: Prompt rewriting with reinforcement learning. In Annual Meeting of the Association for Computational Linguistics (ACL). find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
  4. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024. A survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107–1128, Miami…
  5. Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. 2024. Promptbreeder: Self-referential self-improvement via prompt evolution. In International Conference on Machine Learning (ICML).
  6. Sherzod Hakimov, Yan Weiser, and David Schlangen. 2024. Evaluating modular dialogue system for form filling using large language models. In Proceedings of the 1st Workshop on Simulating Conversational Intelligence in Chat (SCI-CHAT 2024), pages 36–52, St. Julians, Malta. Association for Computationa… find
  7. Mengze Hong, Wailing Ng, Chen Jason Zhang, Yuanfeng Song, and Di Jiang. 2025. Dial-in LLM: Human-aligned LLM-in-the-loop intent clustering for customer service dialogues. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 5885–5900, Suzhou, China. Associ… find
  8. Megan Hosfield. 2024. Prior authorization and referral process in healthcare and its administration burden. The Catalyst: Propelling Scholars Forward, 2:86– 106. find
  9. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  10. Daniel Jaroslawicz, Brendan Whiting, Parth Shah, and Karime Maamari. 2025. How many instructions can llms follow at once? In NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling. find
  11. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. 2023. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714.
  12. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–9474.
  13. Sergio Rojas-Galeano. 2025. Beyond ivr touch-tones: Customer intent routing using llms. arXiv preprint arXiv:2510.21715. — [arXiv:2510.21715](https://arxiv.org/abs/2510.21715) find
  14. Jiayi Shen, Qi Wang, Zehao Xiao, Nanne Van Noord, and Marcel Worring. 2024. Go4align: Group optimization for multi-task alignment. In Advances in Neural Information Processing Systems, volume 37, pages 111382–111405. Curran Associates, Inc. find
  15. Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim. 2023. Multitask prompt tuning enables parameter-efficient transfer learning. In The Eleventh International Conference on Learning Representations. find
  16. Zifeng Wang, Hanyin Wang, Benjamin Danek, Ying Li, Christina Mack, Luk Arbuckle, Devyani Biswal, Hoifung Poon, Yajuan Wang, Pranav Rajpurkar, et al. 2025. A perspective for adapting generalist ai to specialized medical ai applications and their challenges. NPJ Digital Medicine, 8(1):429. find
  17. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837.
  18. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2024. Large language models as optimizers. In International Conference on Learning Representations (ICLR).
  19. Wenbo Zhao, Arpit Gupta, Tagyoung Chung, and Jing Huang. 2023. SPC: Soft prompt construction for cross domain generalization. In Proceedings of the 8th Workshop on Representation Learning for NLP (RepL4NLP 2023), pages 118–130. Association for Computational Linguistics. find
  20. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. In The eleventh international conference on learning representations.
  21. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large language models are human-level prompt engineers. In International Conference on Learning Representations (ICLR).

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ram-2026-transfer-learning-generalizable-automated-llm/