Elamur Research⌕
☰
home / papers / 2026 / When the Database Fails: Prompting LLM Dialogue Agents for Safe Recovery in Task-Oriented Dialogue

When the Database Fails: Prompting LLM Dialogue Agents for Safe Recovery in Task-Oriented Dialogue

Summary

LLM-based task-oriented dialogue agents often respond confidently even when the backend database fails. They may invent venues, bookings or confirmation codes that the database never returned. The authors ask whether prompting alone, with no fine-tuning and no extra model calls, can curb this behaviour.

They built fault-injected test sets from the first user turns of MultiWOZ 2.2 (5 domains) and SGD (20 domains). Each dataset covers four database conditions: clean, empty result, wrong-domain retrieval and API error. They then compared three system prompts across six open-weight model families at roughly 7–9B scale:

  • Naive: no special instruction.
  • Inform: a general instruction to be honest.
  • Guided-Retry: a step-by-step procedure keyed to the database status field.

Evaluation combined regex- and keyword-based metrics (HR, AAR, CSR, UFS) with a human check of commitment safety.

Guided-Retry roughly halved hallucination on MultiWOZ and cut it by about 40% on SGD, with the same pattern on both benchmarks. Hallucination was still far from eliminated, which suggests prompt-level fixes are a cheap first line of defence rather than a complete solution.

Key points

  • Average hallucination rate (HR): fell from 30.5% to 15.3% on MultiWOZ 2.2 and from 20.9% to 12.2% on SGD. Appropriate Action Rate rose from 66.5% to 83.0% on MultiWOZ and from 58.7% to 83.9% on SGD.
  • Residual hallucination by model: ranged from 5.8% for DeepSeek-R1 to 35.2% for Phi-3 on MultiWOZ. Phi-3 was the only model where Guided-Retry did worse than Inform.
  • Wrong-domain retrieval was the hardest case: on MultiWOZ, HR dropped from 47.8% (Naive) to 20.8% (Guided-Retry); on SGD, from 31.0% to 17.0%.
  • Human validation: nine annotators rated 60 responses, with Fleiss' κ = 0.7672. Human-judged CSR was 100% for Guided-Retry, 89.47% for Inform and 80.95% for Naive.
  • Setup and code: the evaluation used 400 dialogues per dataset (100 per condition) at temperature 0.2. Models were DeepSeek-R1, Gemma-2, Llama-3, Mistral, Phi-3 and Qwen-2.5. Code and prompts are available at github.com/mohammad-AJP/llm-db-failure-recovery.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.sigdial-1.57

@inproceedings{shalmani2026when,
  title = {{When the Database Fails: Prompting LLM Dialogue Agents for Safe Recovery in Task-Oriented Dialogue}},
  author = {Shalmani, Mohammad Alijanpour and Boroujeni, Alale Rezvani and Yuan, Jiann S.},
  booktitle = {Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2026},
  doi = {10.18653/v1/2026.sigdial-1.57},
  url = {https://doi.org/10.18653/v1/2026.sigdial-1.57}
}

References

15 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, and 110 others. 2024. Phi-… find our card
  2. Avinash Baidya, Kamalika Das, and Xiang Gao. 2025. The behavior gap: Evaluating zero-shot LLM agents in complex task-oriented dialogs. In Findings of ACL 2025. find
  3. Mihail Eric, Rahul Goel, Shachi Paul, Adarsh Kumar, Abhishek Sethi, Peter Ku, Anuj Kumar Goyal, Sanchit Agarwal, Shuyang Gao, and Dilek Hakkani-Tur. 2020. MultiWOZ 2.1: A consolidated multi-domain dialogue dataset. In Proceedings of LREC 2020, pages 422–428. find
  4. Gemma Team, Google DeepMind. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find
  5. Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. MultiWOZ 2.2: A dialogue dataset with additional annotation corrections. In Proceedings of the 2nd Workshop on NLP for Conversational AI, pages 109–117. find
  6. Daya Guo, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, and 3 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via r… find our card
  7. Aayush Gupta. 2026. ReliabilityBench: Evaluating LLM agent reliability under production-like stress conditions. arXiv preprint arXiv:2601.06112. — [arXiv:2601.06112](https://arxiv.org/abs/2601.06112) find
  8. Zhaoyi Joey Hou, Tanya Shourya, Yingfan Wang, Shamik Roy, Vinayshekhar Bannihatti Kumar, and Rashmi Gangadharaiah. 2025. Multi-faceted evaluation of tool-augmented dialogue systems. arXiv preprint arXiv:2510.19186. — [arXiv:2510.19186](https://arxiv.org/abs/2510.19186) find
  9. Vojtěch Hudeček and Ondřej Dušek. 2023. Are large language models all you need for task-oriented dialogue? In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 216–228. find
  10. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  11. Meta Llama Team. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. find
  12. Qwen Team. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. find
  13. Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, pages 8689–8696. find
  14. Jeonghoon Shim, Woojung Song, Cheyon Jin, Seung-won Kook, and Yohan Jo. 2025. Non-collaborative user simulators for tool agents. arXiv preprint arXiv:2509.23124. — [arXiv:2509.23124](https://arxiv.org/abs/2509.23124) find
  15. Sri Vatsa Vuddanti, Aarav Shah, Satwik Kumar Chit-tiprolu, Tony Song, Sunishchal Dev, Kevin Zhu, and Maheep Chaudhary. 2025. PALADIN: Self-correcting language model agents to cure tool-failure cases. arXiv preprint arXiv:2509.25238. — [arXiv:2509.25238](https://arxiv.org/abs/2509.25238) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shalmani-2026-when-database-fails-prompting-llm/