📄 Automatic Evaluation: Using a DATE Dialogue Act Tagger for User Satisfaction and Task Completion Prediction

Summary

The paper tackles the high cost of evaluating spoken dialogue systems. Normally this requires paid subjects, surveys, transcription and hand labelling. The authors ask how much of that evaluation can be replaced by signals drawn automatically from system logfiles. They work with the 2001 DARPA Communicator corpus: 1242 travel-planning dialogues collected from eight systems. Building on the PARADISE framework, they train a CART regression tree to estimate UserSatisfaction, which is the sum of five Likert-scale survey answers. The tree uses three groups of automatic features:

• an automatically predicted Task Success value;
• efficiency measures such as turn counts and duration;
• dialogue-act metrics produced by their DATE tagger, which labels system utterances.

On a held-out 20% test split, the tree that includes DATE features correlates at 0.614 with real satisfaction scores. Without those features the correlation is 0.595. Task completion itself is predicted with 85% accuracy. These results give a baseline for fully automatic evaluation. Such a predictor could help decide which dialogues deserve transcription, or feed into online system adaptation.

Key points

• Corpus: 1242 Communicator dialogues from 2001, in three task types:

  • 350 complex trips;
  • 694 real trips;
  • 198 round trips.

• Data split: 994 dialogues for training and 248 for testing.

• The DATE tagger covers 98.4% of system utterances with 96% accuracy. It labels each utterance on three independent axes:

  • SPEECH ACT;
  • CONVERSATIONAL-DOMAIN (ABOUT-TASK, ABOUT-COMMUNICATION, ABOUT-SITUATION-FRAME);
  • TASK-SUBTASK, which has 28 subtasks.

• TaskCompletion has two versions:

  • ternary: 0 = failure, 1 = flight booked, 2 = flight plus car/hotel booked;
  • binary.

  An automatic predictor of it reaches 85% accuracy.

• UserSatisfaction prediction:

  • with DATE features: r = 0.614 (R² 0.376);
  • without DATE features: r = 0.595 (R² 0.35).

• Feature usage in the tree:

  • efficiency measures: 47% of queries;
  • dialogue-act metrics: 32%;
  • Task Success: 21%, and it sits at the root of the tree.

Interesting ideas

• I1. Each utterance is labelled along several separate axes (what act it performs, what it is about, which subtask it serves), so general labels and domain-specific labels stay apart and can be combined freely.
• I2. The output of one automatic predictor (task completion) becomes an input feature for a second predictor (satisfaction), stacking cheap automatic signals instead of relying on human labels.
• I3. Feature importance is measured by how often each feature is consulted while the tree predicts each item, normalised per tree, which shows which kinds of evidence actually drive decisions.
• I4. An automatic quality predictor works as a triage tool, picking which items are worth expensive human transcription or annotation.
• I5. The value of a feature group is shown by retraining with and without it and comparing held-out correlation, a simple ablation that guards against overclaiming.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): Speech-act annotations in prag could likewise keep act type, discourse domain (task vs. managing communication) and goal slot as separate fields, so each can be queried and explained on its own.
• I2 (learn): A confidence or success estimate from an earlier pipeline stage could serve as an explicit feature for judging downstream answer quality, keeping the reasoning chain inspectable.
• I3 (learn): Self-induced repair rules or decision trees could report usage shares per feature family, showing whether syntax, frames or world knowledge carries a given decision.
• I4 (learn): A predicted-difficulty or predicted-failure score could rank sentences so that scarce LLM-teacher or human checking goes to the cases most likely to be wrong.
• I5 (answer): Each knowledge source used by the readers (e.g., VerbNet roles, world state) could be evaluated by toggling it off on fable-QA or word-problem tests to measure its real contribution.