📄 EmoHarbor: Evaluating Personalized Emotional Support by Simulating the User's Internal World

Summary

Most automatic evaluations of emotional support conversations (ESC) take an outside "expert" view. They tend to give high scores to generically empathetic replies without checking whether the help suits a particular person. EmoHarbor changes this with a User-as-a-Judge approach: an agent simulates the seeker's internal world and rates the supporter from that user's perspective. A Chain-of-Agent design divides the simulated user into three agents:

• a User Thinker that infers hidden thoughts and emotional reactions,
• a User Talker that produces replies consistent with the user's personality,
• a User Evaluator that scores the supporter as the conversation unfolds.

The benchmark uses 100 profiles. Seeds came from real questionnaire responses and were then expanded with LLMs. Responses are judged on 10 dimensions of personalized support quality. The authors tested 20 advanced LLMs. These models handle broad empathy well but repeatedly fail to adapt support to the individual user. The authors also report strong agreement with human judgments. This matters because it suggests ESC research should aim for user-aware support rather than simply more empathetic wording.

Key points

• Each user profile has four parts: demographic attributes, preference-related attributes (Big-Five, MBTI, habits, speech style), counseling-related attributes (problem, emotion, goals, relations) and a scenario script.
• The 100 profiles span ages from adolescence to senior adulthood, 28 occupations, all 16 MBTI types and 16 counseling scenario types, such as workplace stress and life transitions.
• During simulation, the supporter sees only the visible dialogue turns. The simulated user keeps a separate memory that also stores its latent internal states.
• In a turn-by-turn analysis of Doubao-Pro, every metric drifts slightly downward over longer dialogues. Redundancy degrades most sharply, indicating increasingly repetitive replies.
• The authors list three limitations: simulated users follow predefined structures, human-agreement studies may carry biases, and multi-turn simulation is computationally costly. Code is released at github.com/kaedeharakazuha678/EmoHarbor-25.