Elamur Research⌕
☰
home / papers / 2026 / Retrieving Responses Useful as References in Counseling via LLM-Generated Structurally Analogous Dialogues

Retrieving Responses Useful as References in Counseling via LLM-Generated Structurally Analogous Dialogues

Summary

The paper addresses a retrieval problem in counseling support. Given an ongoing conversation that ends with a client message, the goal is to find a past dialogue history whose following counselor reply could be adapted for the current case. The authors call this property referenceability. It depends on the interaction's trajectory (problem structure, emotional shifts, support flow) more than on topic overlap.

Labeling referenceability by hand at scale is impractical. Instead, the authors have LLMs rewrite counseling sessions into new sessions with a different surface situation. Each generated session keeps a strict one-to-one message alignment with its original and preserves the original's counseling flow. The aligned histories then act as weak positive pairs for supervised contrastive learning (SCL) of a Japanese embedding model, Ruri. They also serve as approximate evaluation targets.

On KokoroChat, the fine-tuned retriever roughly tripled Hit@1. It also improved on a proprietary dataset it was not trained on. A human rater judged its top-1 results more referenceable, even when generated dialogues were excluded from the candidate pool. The approach could help trainee and volunteer counselors draft replies and could feed RAG pipelines.

Key points

  • Model and training: The base encoder is cl-nagoya/ruri-v3-310m, chosen as the strongest of several Japanese embedding models in a preliminary comparison. It was fine-tuned with SCL on LLM-generated pseudo-similar dialogues built from KokoroChat, using a dialogue history length of n = 5.
  • KokoroChat results:
    • Hit@1 rose from 0.167 to 0.500.
    • Hit@3 rose from 0.255 to 0.599.
    • Hit@5 rose from 0.295 to 0.638.
    • MRR rose from 0.232 to 0.576.
  • Proprietary dataset (same KokoroChat-trained model):
    • Hit@1 rose from 0.253 to 0.444.
    • MRR rose from 0.321 to 0.507.
  • Human evaluation:
    • Setup: one annotator with training in volunteer text support rated top-1 results for 960 randomly sampled KokoroChat queries.
    • Scores: the mean Referenceability score improved from 2.22 to 2.66.
    • Significance: McNemar's test on scores binarized at ≥4 gave p < 0.000001.
  • Limitations and context: About half of the retrieved replies still received the lowest score (not referenceable). Some failures were topically similar but did not serve the conversational function the query required. The authors stress that professional supervision is needed for any real deployment. The work was supported by the Japanese nonprofit IbashoChat.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.sigdial-1.20

@inproceedings{nakagawa2026retrieving,
  title = {{Retrieving Responses Useful as References in Counseling via LLM-Generated Structurally Analogous Dialogues}},
  author = {Nakagawa, Yu and Ikeda, Nozomu and Funakoshi, Kotaro and Okumura, Manabu},
  booktitle = {Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2026},
  doi = {10.18653/v1/2026.sigdial-1.20},
  url = {https://doi.org/10.18653/v1/2026.sigdial-1.20}
}

References

30 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jie Cao, Michael Tanana, Zac Imel, Eric Poitras, David Atkins, and Vivek Srikumar. 2019. Observing dialogue in therapy: Categorizing and forecasting behavioral codes. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5599–5611. Association for Computat… find
  2. Mingyu Chen, Jingkai Lin, Zhaojie Chu, Xiaofen Xing, Yirong Chen, and Xiangmin Xu. 2025. CATCH: A novel data synthesis framework for high therapy fidelity and memory-driven planning chain of thought in AI counseling. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 102… find
  3. Chao-Wei Huang, Chen-Yu Hsu, Tsu-Yuan Hsu, Chen-An Li, and Yun-Nung Chen. 2023. CON-VERSER: Few-shot conversational dense retrieval with synthetic data generation. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 381–387, Prague, Czechia. Assoc… find
  4. Anqi Li, Lizhi Ma, Yaling Mei, Hongliang He, Shuai Zhang, Huachuan Qiu, and Zhenzhong Lan. 2023. Understanding client reactions in online mental health counseling. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10358–10376, T… find
  5. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Towards unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118. find our card
  6. Baihan Lin, Guillermo Cecchi, and Djallel Bouneffouf. 2024. Working alliance transformer for psychotherapy dialogue classification. In Proceedings of the 6th Clinical Natural Language Processing Workshop, pages 64–69. Association for Computational Linguistics. find
  7. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781. Asso… find our card
  8. Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. 2021. Towards emotional support dialog systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan… find our card
  9. Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, and Marina Danilevsky. 2025. mt RAG: A multi-turn conversational benchmark for evaluating retrieval-augmented generation systems. Transactions of the Association f… find
  10. Kelong Mao, Zhicheng Dou, Hongjin Qian, Fengran Mo, Xiaohua Cheng, and Zhao Cao. 2022. ConvTrans: Transforming web search sessions for conversational dense retrieval. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2935–2946, Abu Dhabi, United Arab Em… find
  11. Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. In Advances in Neural Information Processing Systems, volume 33, pages 18661–18673. Curran Associates, Inc. find
  12. Fengran Mo, Chen Qu, Kelong Mao, Tianyu Zhu, Zhan Su, Kaiyu Huang, and Jian-Yun Nie. 2024. History-aware conversational dense retrieval. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13366–13378. Association for Computational Linguistics. find
  13. Sungdong Kim and Gangwoo Kim. 2022. Saving dense retriever from shortcut dependency in conversational search. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10278–10287, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  14. Verónica Pérez-Rosas, Xinyi Wu, Kenneth Resnicow, and Rada Mihalcea. 2019. What makes a good counselor? learning to distinguish between high-quality and low-quality counseling conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 926–935, Fl… find
  15. Fei-Tzin Lee, Derrick Hull, Jacob Levine, Bonnie Ray, and Kathy McKeown. 2019. Identifying therapist conversational actions across diverse psychotherapeutic approaches. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology, pages 12–23. Association for Computation… find
  16. Zhiyang Qi, Takumasa Kaneko, Keiko Takamizo, Mariko Ukiyo, and Michimasa Inaba. 2025. KokoroChat: A Japanese psychological counseling dialogue dataset collected via role-playing by trained counselors. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume… find
  17. Yohan Lee, Yongwoo Song, and Sangyeop Kim. 2025. Finding diamonds in conversation haystacks: A benchmark for conversational data retrieval. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 2343–2366, Suzhou (China). Association for Comp… find
  18. Siqi Shen, Veronica Perez-Rosas, Charles Welch, Soujanya Poria, and Rada Mihalcea. 2022. Knowledge enhanced reflection generation for counseling dialogues. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3096–3107, Dublin, Ire… find
  19. Hayeon Yang, Jiheun Hong, Seongjin Jo, Jooyoung Lim, and Hayoung Oh. 2025. GREEN: Generative retrieval-enhanced emotional support conversations. SAGE Open. find
  20. Hayato Tsukagoshi and Ryohei Sasano. 2024. Ruri: Japanese general text embeddings. Preprint, arXiv:2409.07737. — [arXiv:2409.07737](https://arxiv.org/abs/2409.07737) find
  21. Hayeon Yang, Jiheun Hong, Seongjin Jo, and Hayoung Oh. 2026. Sage: Self-retrieval-augmented generative llm for emotional support conversation. Expert Systems with Applications, 313:131524. find
  22. Quan Tu, Yanran Li, Jianwei Cui, Bin Wang, Ji-Rong Wen, and Rui Yan. 2022. MISC: A mixed strategy-aware model integrating COMET for emotional support conversation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 308–319. Assoc… find
  23. Weixiang Zhao, Yanyan Zhao, Shilong Wang, and Bing Qin. 2023. TransESC: Smoothing emotional support conversation via turn-level state transition. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6725–6739. Association for Computational Linguistics. find
  24. Jinfeng Zhou, Zhuang Chen, Bo Wang, and Minlie Huang. 2023. Facilitating multi-turn emotional support conversation with positive emotion elicitation: A reinforcement learning approach. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… find
  25. Quan Tu, Chongyang Tao, and Rui Yan. 2024. Multi-grained conversational graph network for retrieval-based dialogue systems. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 11756–11765. ELRA and ICCL. find
  26. Longxiang Wang, Pukun Zhao, Chen Chen, Jinhe Bi, Huacan Wang, Tong Zhang, and Ronghao Chen. 2026. PsyPARSE: Retrieval-augmented slow thinking for personalized empathetic counseling. In Proceedings of the Fortieth AAAI Conference on Artificial Intelligence (AAAI-26). find
  27. WHO. 2022. World mental health report: Transforming mental health for all. find
  28. Haojie Xie, Yirong Chen, Xiaofen Xing, Jingkai Lin, and Xiangmin Xu. 2025. PsyDT: Using LLMs to construct the digital twin of psychological counselor with personalized counseling style for psychological counseling. In Proceedings of the 63rd Annual Meeting of the Association for Computational Lingui… find
  29. Yilong Xu, Jinhua Gao, Xiaoming Yu, Yuanhai Xue, Baolong Bi, Huawei Shen, and Xueqi Cheng. 2025. Training a utility-based retriever through shared context attribution for retrieval-augmented language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p… find
  30. Zhe Xu, Daoyuan Chen, Jiayi Kuang, Zihao Yi, Yaliang Li, and Ying Shen. 2024. Dynamic demonstration retrieval and cognitive understanding for emotional support conversation. In Proceedings of the 47th International ACM SI-GIR Conference on Research and Development in Information Retrieval (SIGIR ’24… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/nakagawa-2026-retrieving-responses-useful-as-references/