Elamur Research⌕
☰
home / papers / 2026 / Rethinking Evaluation in Retrieval-Augmented Personalized Dialogue: A Cognitive and Linguistic Perspective

Rethinking Evaluation in Retrieval-Augmented Personalized Dialogue: A Cognitive and Linguistic Perspective

Summary

Personalized dialogue systems, including retrieval-augmented ones, are usually scored by how many words their replies share with a reference reply (BLEU, ROUGE, F1). Such scores reveal little about whether a conversation stays coherent, keeps the persona consistent and preserves common ground. Zhang and Traum use LAPDOG as a case study and re-examine it from a cognitive and linguistic angle. LAPDOG enriches sparse personas with ROCStory narratives, which a dual-encoder retriever selects and which is trained jointly with the generator. The authors identify four weaknesses: reliance on similarity metrics for training and evaluation, incoherent outputs, retrieved stories that contradict the persona, and corrupted dialogue histories. Two human annotators and two LLM judges assessed candidate responses in ConvAI2 (Persona-Chat) dialogues, giving each a 1–5 quality score and ranking three candidates against one another. Human and LLM judgments agree with each other far better than either agrees with the overlap metrics. This means gains reported by those metrics do not reliably reflect better communication. The study argues for discourse-aware evaluation and supports LLM judges for persona-grounded dialogue, provided they are validated against humans.

Key points

  • LAPDOG has four problems: similarity-driven optimization and evaluation, incoherent generation, persona–story contradictions, and corrupted dialogue histories.
  • The annotation protocol rates each of three candidate responses on a 1–5 scale (coherent, engaging, persona-consistent at the top) and assigns relative ranks 1–3, with ties allowed.
  • Correlations with the first human annotator were 0.555 for the second human, 0.571 for the DS LLM judge and 0.487 for the o1 LLM judge. Correlations with BLEU, ROUGE and F1 were only 0.038–0.081.
  • Williams tests on 40 paired dialogues show that human–LLM and human–human correlations are significantly higher than correlations with the lexical metrics (α = 0.05). Differences among the human and LLM judges themselves were mostly non-significant.
  • Resources involved are ROCStory, ConvAI2/Persona-Chat and ParlAI. Proposed future work includes taxonomies of persona–story relations and multi-objective training targeting coherence, engagement and common ground.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/29xyfmxaqr72

@inproceedings{zhang2026rethinking,
  title = {{Rethinking Evaluation in Retrieval-Augmented Personalized Dialogue: A Cognitive and Linguistic Perspective}},
  author = {Zhang, Tianyi and Traum, David},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/29xyfmxaqr72},
  url = {https://doi.org/10.63317/29xyfmxaqr72}
}

References

41 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Sur… find our card
  2. Susan E. Brennan and Herbert H. Clark. 1996. Conceptual pacts and lexical choice in conversation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22(6):1482–1493. find
  3. Xueliang Cheng, Ziwei Ji, Jiashuo Wang, Zheng Gong, Yuxuan Lai, and Baobao Chang. 2024. In-dialogue learning for personalized dialogue generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10448–10465, Miami, Florida, USA. Association for Comput… find
  4. Cheng-Han Chiang and Hung yi Lee. 2023. Can large language models be an alternative to human evaluations? find
  5. Herbert H. Clark. 1996. Using Language. Cambridge University Press, Cambridge, UK. find
  6. Herbert H. Clark and Susan E. Brennan. 1991. Grounding in communication. In Lauren B. Resnick, John M. Levine, and Stephanie D. Teasley, editors, Perspectives on Socially Shared Cognition, pages 127–149. American Psychological Association, Washington, DC. find
  7. Simon Garrod and Martin J. Pickering. 2009. Joint action, interactive alignment, and dialogue. Topics in Cognitive Science, 1(2):292–304. find
  8. Barbara J. Grosz, Aravind K. Joshi, and Scott Weinstein. 1995. Centering: A framework for modeling the local coherence of discourse. Computational Linguistics, 21(2):203–225. find our card
  9. Barbara J. Grosz and Candace L. Sidner. 1986. Attention, intention, and the structure of discourse. Computational Linguistics, 12(3):175–204. find
  10. Jerry R. Hobbs. 1979. Coherence and coreference. Cognitive Science, 3(1):67–90. find
  11. Qiushi Huang, Shuai Fu, Xubo Liu, Wenwu Wang, Tom Ko, Yu Zhang, and Lilian Tang. 2023. Learning retrieval augmentation for personalized dialogue generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2523–2540, Singapore. Association for Computat… find
  12. Zhihao Li, Yixin Zhang, Qian Chen, and Wenqiang Lei. 2024. Unims-rag: Unifying multi-source retrieval-augmented generation for personalized dialogue. arXiv preprint arXiv:2401.13256. — [arXiv:2401.13256](https://arxiv.org/abs/2401.13256) find
  13. Chia-Wei Liu, Ryan Lowe, Iulian V. Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016a. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Na… find
  14. Chia-Wei Liu, Ryan Lowe, Iulian V. Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016b. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Na… find
  15. Ryan Lowe, Michael Noseworthy, Iulian V. Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic Turing test: Learning to evaluate dialogue responses. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1116–1… find our card
  16. Fanghua Lu, Yizhe Zhang, Xiang Li, and Minlie Huang. 2023. Miracle: Towards personalized dialogue generation with latent-space multiple personal attribute control. arXiv preprint arXiv:2310.18342. find
  17. Bhaskar Komaragiri Majumder, Y-Lan Boureau, Jason Weston, and Antoine Bordes. 2021. Unsupervised enrichment of persona-grounded dialog with background stories. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL): Short Papers, pages 604–611. find
  18. Rui Mao, Guanyi Chen, Xiao Li, Mengshi Ge, and Erik Cambria. 2025. A comparative analysis of metaphorical cognition in ChatGPT and human minds. Cognitive Computation, 17(35):1–12. find
  19. Shikib Mehri and Maxine Eskenazi. 2020. Unsupervised evaluation of interactive dialog with DialoGPT. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 225–235, 1st virtual meeting. Association for Computational Linguistics. find our card
  20. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Associati… find
  21. Jihyun Oh, Chan-Hyeong Park, Sang-Hyun Park, and Dongchan Kim. 2023. Pk-icr: Persona-knowledge interactive context retrieval for personalized dialogue generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  22. Martin J. Pickering and Simon Garrod. 2004. Toward a mechanistic psychology of dialogue. Behavioral and Brain Sciences, 27(2):169–190. find
  23. Martin J. Pickering and Simon Garrod. 2013. An integrated theory of language production and comprehension. Behavioral and Brain Sciences, 36(4):329–347. find
  24. Massimo Poesio and Hannes Rieser. 2001. Completions, coordination, and alignment in dialogue. In Proceedings of the 2nd SIGdial Workshop on Discourse and Dialogue, pages 1–10, Aalborg, Denmark. Association for Computational Linguistics. find
  25. Matthias Reiss. 2023. Testing the reliability of chatgpt for text annotation and classification: A cautionary remark. arXiv preprint arXiv:2305.15780. — [arXiv:2305.15780](https://arxiv.org/abs/2305.15780) find
  26. Shikhar Sharma, Hannes Schulz, Maxine Eskenazi, and Yoshua Bengio. 2017. Natural language understanding for task-oriented dialogues in the presence of out-of-domain utterances. In Proceedings of the 8th International Workshop on Spoken Dialogue Systems (IWSDS). find
  27. Chrysanthi Siro, Mohammad Aliannejadi, and Maarten de Rijke. 2024. Rethinking the evaluation of dialogue systems: Effects of user feedback on crowdworkers and llms. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). find
  28. Dan Sperber and Deirdre Wilson. Relevance: Communication and cognition, volume 142. find
  29. Chen Tang, Chengguang Tang, Yating Zhang, Jinchao Zhang, and Jie Zhou. 2023. Enhancing personalized dialogue generation with contrastive latent variables: Combining sparse and dense persona. arXiv preprint arXiv:2305.11482. find
  30. Yi Wang, Qian Liu, Yunchang Cui, Yining Wang, and Tatsunori Hashimoto. 2023. Safari: Large language models as source planner for personalized knowledge-grounded dialogues. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4129–4143. find
  31. Yicheng Wang, Jiayi Yuan, Yu-Neng Chuang, Zhuoer Wang, Yingchi Liu, Mark Cusick, Param Kulkarni, Zhengping Ji, Yasser Ibrahim, and Xia Hu. 2025. Dhp benchmark: Are llms good nlg evaluators? find
  32. Frank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80– 83. find
  33. E. J. Williams. 1959. The comparison of regression variables. Journal of the Royal Statistical Society: Series B (Methodological), 21(2):396–399. find
  34. Yunhe Xie and Rui Mao. 2025. PGIF: A personality-guided iterative feedback graph network for multimodal conversational emotion recognition. IEEE Transactions on Computational Social Systems, pages 1–13. find
  35. Yuxin Ye, Zhaoqing Li, and Guodong Zhou. 2023. Retrieval-enhanced generation for personalized dialogue: Balancing consistency and diversity. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023), pages 9675–9690, Toronto, Canada. Association for Comput… find
  36. Xiaoyu Zhan, Wenhao Chen, and Qiang Liu. 2024. Contextual rag: Enhancing coherence in retrieval-augmented dialogue generation. IEEE Transactions on Affective Computing. find
  37. Luyao Zhu, Wei Li, Rui Mao, and Erik Cambria. 2024a. HIPPL: Hierarchical intent-inferring pointer network with pseudo labeling for consistent persona-driven dialogue generation. IEEE Computational Intelligence Magazine, 19(4):63– 78. find
  38. Luyao Zhu, Wei Li, Rui Mao, Vlad Pandelea, and Erik Cambria. 2023. PAED: Zero-shot persona attribute extraction in dialogues. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), volume 1, page 9771–9787, Toronto, Canada. Association for Computational Lin… find
  39. Luyao Zhu, Rui Mao, Erik Cambria, and Bernard J. Jansen. 2024b. Neurosymbolic ai for personalized sentiment analysis. In HCI International 2024 – Late Breaking Papers, volume 15119 of Lecture Notes in Computer Science, pages 269– 290, Washington, DC, USA. Springer. find
  40. Language Resource References Dinan, Emily and Logacheva, Varvara and Malykh, Valentin and Miller, Alexander and Shuster, Kurt and Urbanek, Jack and Kiela, Douwe and Szlam, Arthur and Serban, Iulian and Lowe, Ryan and Prabhumoye, Shrimai and Black, Alan W. and Rudnicky, Alexander and Williams, Jason… find
  41. Miller, Alexander H. and Feng, Will and Fisch, Adam and Lu, Jiasen and Batra, Dhruv and Bordes, Antoine and Parikh, Devi and Weston, Jason. 2017. ParlAI: A Dialog Research Software Platform. Association for Computational Linguistics. Available at https://parl.ai. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-rethinking-evaluation-retrieval-augmented-personalized/