Elamur Research⌕
☰
home / papers / 2026 / Structured Dialogue Refinement: Building Retrieval-Augmented Question Answering on Goal-Oriented Dialogues

Structured Dialogue Refinement: Building Retrieval-Augmented Question Answering on Goal-Oriented Dialogues

Paper · 2026 · Bin Wu, Sawan Kumar, Prasetya Ajie Utama et al.

Summary

Standard RAG assumes that evidence lives in coherent, static documents. Much practical troubleshooting knowledge, however, sits in goal-oriented conversations such as help-forum threads and support chats. Users and helpers begin with mismatched common ground, so the actual problem and its fix are scattered across turns and keep shifting as the issue is clarified and the answer is extended. The authors propose Structured Dialogue Refinement (SDR), which reshapes a dialogue corpus while leaving the RAG pipeline itself unchanged. At retrieval time, Dual Dialogue Querying derives latent queries describing each dialogue's issue and its solution. These are stored as two pseudo-document indices and combined through hybrid retrieval. At generation time, each dialogue becomes a directed graph of issue–solution units linked by refinement or elaboration edges, and a relevance-driven roaming procedure selects compact subgraphs as context. Evaluation decomposes reference answers into atomic facts (nuggets) and uses the Ubuntu Dialogue Corpus aligned with AskUbuntu questions. SDR improves both retrieval coverage and answer accuracy, which suggests that conversational logs can become a usable knowledge source for QA.

Key points

  • Retrieval bottleneck: In a pilot analysis with question-based retrieval, BM25 falls 45.4% short of gold-dialogue nugget coverage, and BGE falls 20.7% short.

  • Generation bottleneck: Qwen2.5-72b-instruct reaches these nugget-support levels:

    • closed-book: 19.8%
    • top-20 RAG: 22.9%
    • oracle gold dialogues: 29.2%

    Gold dialogues themselves cover 65.5% of nuggets, so even perfect retrieval leaves much evidence unused.

  • Whole-pipeline ablation (benchmark subset, bge): Nugget accuracy rises from 21.30 to 22.13 with Qwen2.5-72b and from 24.67 to 27.43 with gpt-4o-mini. For Qwen, removing either the retrieval or the generation refinement drops results below naive RAG.

  • Subgraph selection: Roaming-based selection (27.43) outperforms picking top-k nodes, whole graphs, or average-score subgraphs.

  • Limitations: Evaluation covers only one domain (Ubuntu/AskUbuntu), and the method assumes a static corpus rather than one that grows over time.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University College LondonLondon · United Kingdom107 works · 22,924 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 3Research subjects and languages the work is about.Dialogue systems · 654 worksQuestion answering · 487 worksInformation retrieval & search · 552 works

Cite

Cite as Bin Wu, Sawan Kumar, Prasetya Ajie Utama and Mohamed Yahya. 2026. Structured Dialogue Refinement: Building Retrieval-Augmented Question Answering on Goal-Oriented Dialogues. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1571

@inproceedings{wu2026structured,
  title = {{Structured Dialogue Refinement: Building Retrieval-Augmented Question Answering on Goal-Oriented Dialogues}},
  author = {Bin Wu, Sawan Kumar, Prasetya Ajie Utama and Mohamed Yahya},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1571},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1571}
}

References

40 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations.
  2. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, and 1 others. 2022. Improving language models by retrieving from trillions of tokens. In International conference on machin… find
  3. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language P…
  4. Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The power of noise: Redefining retrieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Developm…
  5. Zhuyun Dai, Arun Tejasvi Chaganty, Vincent Y Zhao, Aida Amini, Qazi Mamunur Rashid, Mike Green, and Kelvin Guu. 2022. Dialog inpainting: Turning documents into dialogs. In International conference on machine learning, pages 4558–4586. PMLR. find
  6. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powered conversational agents. In International Conference on Learning Representations.
  7. Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130. find
  8. Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D Manning. 2017. Key-value retrieval networks for task-oriented dialogue. In Proceedings of the 18th annual SIGdial meeting on discourse and dialogue, pages 37–49.
  9. Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pages 6491–6501.
  10. Song Feng, Siva Sankalp Patel, Hui Wan, and Sachindra Joshi. 2021. Multidoc2dial: Modeling dialogues grounded in multiple documents. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6162–6176. find
  11. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2(1).
  12. Mitko Gospodinov, Sean MacAvaney, and Craig Macdonald. 2023. Doc2query–: when less is more. In European Conference on Information Retrieval, pages 414–422. Springer. find
  13. Zirui Guo, Lianghao Xia, Yanhua Yu, Tu Ao, and Chao Huang. 2025. LightRAG: Simple and fast retrieval-augmented generation. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 10746–10761, Suzhou, China. Association for Computational Linguistics.
  14. Bernal Jiménez Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su. 2025. From rag to memory: Non-parametric continual learning for large language models. In Forty-second International Conference on Machine Learning.
  15. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929–3938. PMLR.
  16. Bernal Jimenez Gutierrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. 2024. Hipporag: Neurobiologically inspired long-term memory for large language models. Advances in Neural Information Processing Systems, 37:59532–59569.
  17. Jiajie Jin, Xiaoxi Li, Guanting Dong, Yuyao Zhang, Yutao Zhu, Yongkang Wu, Zhonghua Li, Ye Qi, and Zhicheng Dou. 2025a. Hierarchical document refinement for long-context retrieval-augmented generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume…
  18. Jiajie Jin, Yutao Zhu, Zhicheng Dou, Guanting Dong, Xinyu Yang, Chenghao Zhang, Tong Zhao, Zhao Yang, and Ji-Rong Wen. 2025b. Flashrag: A modular toolkit for efficient retrieval-augmented generation research. In Companion Proceedings of the ACM on Web Conference 2025, pages 737–740.
  19. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781.
  20. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–…
  21. Haitao Li, Yifan Chen, Hu YiRan, Qingyao Ai, Junjie Chen, Xiaoyu Yang, Jianhui Yang, Yueyue Wu, Zeyang Liu, and Yiqun Liu. 2025a. Lexrag: Benchmarking retrieval-augmented generation in multi-turn legal consultation conversation. In Proceedings of the 48th International ACM SIGIR Conference on Resear… find
  22. Qiwei Li, Teng Xiao, Zuchao Li, Ping Wang, Mengjia Shen, and Hai Zhao. 2025b. Dialogue-rag: Enhancing retrieval for llms via node-linking utterance rewriting. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 24423–24438. find
  23. Jimmy Lin and Dina Demner-Fushman. 2005. Automatically evaluating answers to definition questions. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 931–938. find
  24. Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, and 1 others. 2024. Ra-dit: Retrieval-augmented dual instruction tuning. In The Twelfth International Conference on Learning Representations.
  25. Ryan Lowe, Nissan Pow, Iulian Vlad Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In Proceedings of the 16th annual meeting of the special interest group on discourse and dialogue, pages 285–294.
  26. Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document expansion by query prediction. arXiv preprint arXiv:1904.08375.
  27. Virgil Pavlu, Shahzad Rajput, Peter B Golbus, and Javed A Aslam. 2012. Ir system evaluation using nugget-based test collections. In Proceedings of the fifth ACM international conference on Web search and data mining, pages 393–402. find
  28. Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, Ian Soboroff, Hoa Trang Dang, and Jimmy Lin. 2025. The great nugget recall: Automating fact extraction and rag evaluation with large language models. In Proceedings of the 48th International ACM SIGIR Conference on Researc… find
  29. Qitao Qin, Yucong Luo, Yihang Lu, Zhibo Chu, Xiaoman Liu, and Xianwei Meng. 2025. Towards adaptive memory-based optimization for enhanced retrieval-augmented generation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 7991–8004. find
  30. Rupak Sarkar, Neha Srikanth, Taylor Pellegrin, Rachel Rudinger, Claire Bonial, and Philip Resnik. 2025. Understanding common ground misalignment in goal-oriented dialog: A case-study with ubuntu chat logs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vo… find
  31. Xiao Yu, Maximillian Chen, and Zhou Yu. 2023. Prompt-based monte-carlo tree search for goal-oriented dialogue policy planning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7101–7125. find
  32. Qiang Zhang, Jason Naradowsky, and Yusuke Miyao. 2023. Ask an expert: Leveraging language models to improve strategic reasoning in goal-oriented dialogue models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6665–6694. find
  33. Xuanwang Zhang, Yun-Ze Song, Yidong Wang, Shuyun Tang, Xinfeng Li, Zhengran Zeng, Zhen Wu, Wei Ye, Wenyuan Xu, Yue Zhang, and 1 others. 2024. Raglab: A modular and research-oriented unified framework for retrieval-augmented generation. In Proceedings of the 2024 Conference on Empirical Methods in Na… find
  34. Nandan Thakur, Jimmy Lin, Sam Havens, Michael Carbin, Omar Khattab, and Andrew Drozdov. 2025. Freshstack: Building realistic benchmarks for evaluating retrieval on technical documents. arXiv preprint arXiv:2504.13128. — [arXiv:2504.13128](https://arxiv.org/abs/2504.13128) find
  35. SMTI Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. 2024. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXiv:2401.01313, 6.
  36. Ruobing Wang, Qingfei Zhao, Yukun Yan, Daren Zha, Yuxuan Chen, Shi Yu, Zhenghao Liu, Yixuan Wang, Shuo Wang, Xu Han, and 1 others. 2024. Deepnote: Note-centric deep retrieval-augmented generation. arXiv preprint arXiv:2410.08821. find
  37. Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2024a. Recomp: Improving retrieval-augmented lms with context compression and selective augmentation. In The Twelfth International Conference on Learning Representations. find
  38. Zhentao Xu, Mark Jerome Cruz, Matthew Guevara, Tie Wang, Manasi Deshpande, Xiaofeng Wang, and Zheng Li. 2024b. Retrieval-augmented generation with knowledge graphs for customer service question answering. In Proceedings of the 47th international ACM SIGIR conference on research and development in in… find
  39. Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2024. Making retrieval-augmented language models robust to irrelevant context. In The Twelfth International Conference on Learning Representations.
  40. Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5):1160–1179. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2026-structured-dialogue-refinement-building-retrieval/