📄 In-Context Evaluation of Unsupervised Dialogue Act Models for Tutorial Dialogue

Summary

Unsupervised dialogue act models avoid costly labelling. However, they are usually scored by how well they agree with hand-made tags. That brings the annotation cost back and forces data-driven clusters to imitate a human tagset. Ezen-Can and Boyer propose judging such a model by what it enables: driving a simple dialogue policy.

The data is a corpus of typed human-human tutoring sessions about programming. The authors cluster student turns with a query-likelihood method: bigram features plus top-level Stanford parser constituents, fed to k-means, with 21 clusters chosen at a knee of the SSE curve. On the 75% training portion, they write one tutor reply per cluster. They then apply this policy to held-out student turns, and seven untrained judges rate how sensible each reply is.

The unsupervised policy scored slightly higher than a comparable policy built on manual tags. This held even though its accuracy at recovering those tags was only modestly above the majority baseline. The study suggests that task-based evaluation can reveal usefulness that label agreement hides, and at a small fraction of the annotation effort.

Key points

• Corpus and tags: the corpus (Boyer et al., 2009) has 1,525 student and 3,332 tutor utterances. Earlier work labelled it with nine tutoring-specific acts (Kappa 0.76), such as Evaluation Question, Lukewarm Feedback and Extra-domain. These tags were never used for training or tuning.
• Model and protocol: the model uses query-likelihood clustering (Ezen-Can & Boyer, 2013), with k tested from 2 to 24 and k=21 selected. Of 373 held-out test pairs, 29 were rated by seven judges on a 1–4 scale, in randomized order, interleaved with 29 manual-tag pairs.
• Ratings: the unsupervised policy averaged 2.35, with 44.8% rated positive and 65.5% judged at least somewhat sensible. The manual-tag policy averaged 2.22, with 31% positive.
• Label accuracy: mapping each cluster to its majority manual tag gave 30.4% accuracy, against a 25.87% majority baseline (Evaluation Question). Judge agreement was only fair: weighted Kappa 0.30, and binary Kappa 0.36.
• Cost: judging took about 15 s per item, roughly 1 hour in total across judges, and writing the policy took about 1 hour. Full act annotation had cost about 35 s per utterance, roughly 50 hours per annotator.

Interesting ideas

• I1. Test an induced category set extrinsically: attach one canned action to each category and have people judge the resulting behaviour on unseen data, with no full end-to-end system needed.
• I2. Represent each utterance by its retrieval-likelihood scores against every other training utterance, then cluster those similarity vectors.
• I3. Show the experimental and comparison conditions to the same judges, interleaved, in a fresh random order per judge, and measure ordinal agreement with weighted Kappa.
• I4. Report evaluation cost (judge-minutes versus annotation hours) next to quality numbers, treating cheap judgement as a goal in its own right.
• I5. When comparing clusters with gold labels, give each cluster its majority tag and always report the majority-class baseline beside the accuracy.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): speech-act labels from prag could be scored by whether a fixed act-to-response mapping produces sensible replies, rather than only by agreement with a gold tagset.
• I2 (learn): candidate speech-act or construction classes could be induced by ranking sentences by similarity over UD subtree features, keeping the grouping explainable.
• I3 (learn): human checks of answers or generated text could mix Mova outputs with a baseline's in a blind shuffle, which acts as a built-in negative control.
• I4 (learn): logging the human time spent per evaluation round would help decide where LLM-teacher or silver checks can replace manual gold labelling.
• I5 (tokens): any unsupervised tag or lemma-class induction in crate en could use this mapping plus baseline to avoid overstating weak gains.