Elamur Research⌕
☰
home / papers / 2026 / Disagreement-Driven Joint Refinement of Retrieval and Decision Rules for Imbalanced Counseling Risk Classification

Disagreement-Driven Joint Refinement of Retrieval and Decision Rules for Imbalanced Counseling Risk Classification

Paper · 2026 · Zhihao Shao, Ryo Sekizaki, Shengzhou Yi et al.

Summary

The paper addresses automatic triage of online counseling records by suicide-risk acuity. This binary task is hard because high-risk cases are rare (about 7% versus 93% low-risk), and LLM predictions on such data tend to be unstable and systematically biased. As a solution, the authors introduce Disagreement-Driven Joint Refinement (DDJR), a framework that refines its behavior over repeated rounds at inference time while leaving model weights untouched. Each labeled training dialogue is classified twice, once zero-shot and once with retrieval-augmented in-context learning (RA-ICL). The two correctness outcomes sort instances into three types:

  • Consistently-Correct (CC): both settings are right.
  • Contrastive Rationalization (CR): exactly one setting is right, so the correct and incorrect rationales are kept.
  • Failure-Diagnostic (FD): both settings fail, so an error diagnosis is generated.

These typed signals drive two updates in each round. The exemplar pool is revised, and an executable rule set is refined with curriculum-based rule construction. At test time, type-weighted retrieval favors CR and FD exemplars. On 6,481 real counseling records, DDJR beat existing methods, reaching 0.915 accuracy and 0.583 MCC. This points to steadier, more interpretable support for prioritizing urgent cases for human review.

Key points

  • The dataset contains 6,481 real-world counseling records with a 93% low-risk / 7% high-risk split. The rules were originally produced in Japanese, which suggests Japanese-language data.
  • DDJR achieves accuracy 0.915 and Matthews Correlation Coefficient (MCC) 0.583, outperforming existing methods.
  • The exemplar pool keeps a fixed size. Each new CR or FD sample displaces one CC sample, and retrieval uses top-k with k = 64.
  • The learned rules separate concrete risk signals (methods, attempts, overdose, planning, suicide notes) from ambivalent or passing death wishes. They also include explicit rules for preventing false positives and false negatives.
  • Cost:
    • Every method makes one LLM call per test case.
    • Prompt sizes are about 1,000 characters for zero-shot, about 25,000 for RA-ICL, and just over 50,000 for DDJR.
    • Estimated input cost per case is $0.06–$0.09 for DDJR.
    • DDJR also requires a one-time offline refinement cost of about $318.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of TokyoTokyo · Japan90 works · 9,156 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksSocial Psychology · 306 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 473 works

Cite

Cite as Zhihao Shao, Ryo Sekizaki, Shengzhou Yi and Toshihiko Yamasaki. 2026. Disagreement-Driven Joint Refinement of Retrieval and Decision Rules for Imbalanced Counseling Risk Classification. Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2f8zttankw7p

@inproceedings{shao2026disagreement,
  title = {{Disagreement-Driven Joint Refinement of Retrieval and Decision Rules for Imbalanced Counseling Risk Classification}},
  author = {Zhihao Shao, Ryo Sekizaki, Shengzhou Yi and Toshihiko Yamasaki},
  booktitle = {Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026},
  year = {2026},
  doi = {10.63317/2f8zttankw7p},
  url = {https://doi.org/10.63317/2f8zttankw7p}
}

References

23 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alaa A. Abd-Alrazaq, Asma Rababeh, Mohannad Alajlani, Bridget M. Bewick, and Mowafa Househ. 2020. Effectiveness and safety of using chatbots to improve mental health: Systematic review and meta-analysis. Journal of Medical Internet Research, 22(7):e16021. find
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774.
  3. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, and J. Kaplan. 2022. Constitutional ai: Kathleen Kara Fitzpatrick, Alison Darcy, and Molly Vierhile. 2017. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversatio… find
  4. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In ICML, pages 3929–3938.
  5. Y. Jiang, Y. Xiong, Y. Yuan, C. Xin, W. Xu, Y. Yue, and L. Yan. 2025. Pag: Multi-turn reinforced llm self-correction with policy as generative verifier. arXiv preprint arXiv:2506.10406.
  6. Arshia Kermani, Veronica Perez-Rosas, and Vangelis Metsis. 2025. A systematic evaluation of LLM strategies for mental health text analysis: Fine-tuning vs. prompt engineering vs. RAG. pages 172–180. find
  7. Liliana Laranjo, Adam G Dunn, Huong Ly Tong, Ahmet Baki Kocaballi, Jessica Chen, Rabia Bashir, Didi Surian, Blanca Gallego, Farah Magrabi, Annie Y S Lau, and Enrico Coiera. 2018. Conversational agents in healthcare: a systematic review. Journal of the American Medical Informatics Association, 25(9):… find
  8. Yoon Kyung Lee, Inju Lee, Minjung Shin, Seoyeon Bae, and Sowon Hahn. 2023. Chain of empathy: enhancing empathetic response of large language models based on psychotherapy models. arXiv preprint arXiv:2311.04915. — [arXiv:2311.04915](https://arxiv.org/abs/2311.04915) find
  9. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. NeurIPS, pages 9459–9474.
  10. Wenjie Li, Tianyu Sun, Kun Qian, and Wenhong Wang. 2024. Optimizing psychological counseling with instruction-tuned large language models. arXiv preprint arXiv:2406.13617. — [arXiv:2406.13617](https://arxiv.org/abs/2406.13617) find
  11. Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Mehran Kazemi, Chitta Baral, Vaiva Imbrasaite, and Vincent Y. Zhao. 2024. Dr.icl: Demonstration-retrieved in-context learning. Data Intelligence, pages 909–922. find
  12. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. NeurIPS, pages 46534–46594. find
  13. Matthew Matero, Akash Idnani, Youngseo Son, Salvatore Giorgi, Huy Vu, Mohammad Zamani, Parth Limbachiya, Sharath Chandra Guntuku, and H Andrew Schwartz. 2019. Suicide risk assessment with multi-level dual-context language and BERT. In CLPsych, pages 39–44. find
  14. Calvin Mercer. 2022. Mental and spiritual health needs of cognitively enhanced people: a therapeutic and spiritual care model for responding. Religions, page 701. find
  15. Sungjoon Park, Donghyun Kim, and Alice Oh. 2019. Conversation model fine-tuning for classifying client utterances in counseling dialogues. In NAACL-HLT, pages 1448–1459. find
  16. Michiel Rauws, John Quick, and Nancy Spangler. 2019. X2 AI Tess: working with AI technology partners. The Journal of Employee Assistance, 49(1):22–25. find
  17. Hao Shen, Zihan Li, Minqiang Yang, Minghui Ni, Yongfeng Tao, Zhengyang Yu, Weihao Zheng, Chen Xu, and Bin Hu. 2024. Are large language models possible to conduct cognitive behavioral therapy? In BIBM, pages 3695–3700. find
  18. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. NeurIPS, pages 8634–8652.
  19. Anand N. Vaidyam, Hannah Wisniewski, John D. Halamka, Matcheri S. Kashavan, and John B. Torous. 2019. Chatbots and conversational agents in mental health: A review of the psychiatric landscape. The Canadian Journal of Psychiatry, 64(7):456–464. find
  20. World Health Organization. 2023. Depressive disorder (depression). find
  21. Z. Wu, Q. Zeng, Z. Zhang, Z. Tan, C. Shen, and M. Jiang. 2024. Large language models can self-correct with key condition verification. In EMNLP, pages 12846–12867. find
  22. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022. STaR: Bootstrapping reasoning with reasoning. In NeurIPS, pages 15476– 15488.
  23. Tianlin Zhang, Annika M Schoene, Shaoxiong Ji, and Sophia Ananiadou. 2022. Natural language find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shao-2026-disagreement-driven-joint-refinement-retrieval/