📄 Goal Alignment in LLM-Based User Simulators for Conversational AI

Summary

LLM-based user simulators are used to build and evaluate conversational agents, but they often lose track of their assigned goals over long dialogues. They may drop persona details, ignore behavioral constraints, or accept outcomes the goal forbids. The authors call this goal misalignment and propose User Goal State Tracking (UGST), a framework inspired by dialog state tracking. UGST splits a user goal into sub-components covering profile, policy, task objectives, requirements and preferences, and updates each one's status after every turn.

They use UGST in three stages:

• Inference-time steering: the current goal state is fed to the simulator before each reply.
• Cold-start SFT: models are trained on reasoning-annotated dialogues produced with that steering.
• GRPO: reinforcement learning with a composite reward derived from UGST.

On MultiWOZ 2.4 and τ-Bench (Airline, Retail), each stage raises goal-alignment success rates. The resulting 7–8B simulators match or beat 70B+ models. This matters because misaligned simulators distort agent evaluation and RL reward signals.

Key points

• Strong open models, including Llama-3.3-70B-Instruct, Qwen-2.5-72B-Instruct and Gemma-3-27B-Instruct, fail to stay goal-aligned in roughly 10–40% of cases.
• The three stages improve average success rate by up to 5.4% (steering), 11.0% (cold-start SFT) and 14.1% (GRPO with UGST rewards), all absolute.
• The SFT data has 1,000 conversations generated by Llama-3.3-70B-Instruct: 500 τ-Bench Retail goals plus 500 synthetic MultiWOZ goals. GRPO then trains on about 5,000 samples for 350 steps with 8 rollouts.
• Diversity (MTLD, HDD) and BERTScore goal similarity increase across the stages. LLM-rated naturalness and coherence stay roughly unchanged, and human annotators agree closely with the automated UGST judgments.
• Limitations: UGST depends on the costly Qwen-2.5-72B-Instruct and on LLM judgments, which may hallucinate. The GRPO reward weights all conditions equally and ignores naturalness. Code and data are on GitHub.