📄 Detecting Inappropriate Clarification Requests in Spoken Dialogue Systems

Summary

Spoken dialogue systems that suspect a misrecognition ask the user for clarification. When they misjudge where the error lies or what kind it is, the question confuses the user. Examples include asking for a synonym of a proper name or demanding that a long phrase be spelled in the NATO alphabet. The authors frame spotting such flawed questions as a new classification task, decided from the user's reply rather than from the system's internal state. They built a laboratory corpus with a simulated system. Participants read sentences aloud, then answered scripted clarification requests of six types modelled on SRI's Thunder-BOLT translator. Each type had appropriate variants and deliberately flawed ones, all derived from TRANSTAC material. A Weka J48 decision tree combines lexical, POS-bigram, LIWC, numeric and request-type features. It reaches 88.5% accuracy and an F-measure of .885, well above a 63.2% majority baseline. A system that notices it has asked a bad question can switch to another strategy or a generic fallback instead of letting the dialogue go off track.

Key points

• Corpus design:
  • 228 scripted requests (84 appropriate, 144 inappropriate), 12 for each of 19 condition categories, built from TRANSTAC data.
  • 17 native American English speakers each answered 114 requests, giving 714 responses to appropriate and 1224 to inappropriate requests (63% inappropriate).
• Request types: confirmation; rephrase-part; spell; disambiguate; targeted reprise; and targeted-reprise-rephrase. Confirmation was always treated as appropriate.
• Features: word unigrams and bigrams with frequency cutoffs, Stanford-tagger POS bigrams, LIWC 2007 categories, function-word ratio, total spelled-letter count, and the type of the preceding request.
• Results (J48, 10-fold cross-validation; J48 beat JRip and LibSVM):
  • full feature set: 88.5% accuracy, F .885;
  • request-type-only baseline: 73.7%, F .686;
  • majority baseline: 63.2%, F .489.
• Leave-one-out ablation: removing request type costs 7.6 points, LIWC 2.3, and POS bigrams 2.0. The top tree splits use "name", the collapsed SPELL token, the bigram of start marker plus "neither", a LIWC category, and request type.

Interesting ideas

• I1. The system's own previous move is judged by the reaction it provokes, so the user's reply becomes a diagnostic signal for an earlier mistake.
• I2. Before tagging, spelled-out letter runs and disfluencies are collapsed into single placeholder tokens, and a handful of highly diagnostic words (negations, "word", "name") get their own POS classes; the resulting POS bigrams then carry start and end markers.
• I3. Errors are injected by construction in a simulated system, so every item's gold label is known, and each error condition gets the same number of items.
• I4. The paper reports a context-only baseline (request type alone) next to the full model, then runs leave-one-feature-out ablation and inspects the top tree splits.
• I5. Error analysis shows users often silently ignore a bad question and just give the right information, so the lack of a complaint does not prove the question was fine.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): prag could recognise a correction speech act that rejects a presupposition of the prior turn (e.g. "not a name, a phrase") and flag which earlier frame or label was wrong.
• I2 (tokens): the en tokenizer could merge letter-by-letter spellings into one token with a "spelled" feature, so the tagger and parser are not derailed by streams of single letters.
• I3 (learn): negative-control sets could be built the same way, by deliberately corrupting a referent, role or event order, to check that evaluators and repair rules actually catch known faults.
• I4 (learn): every evaluation could include a "prior-only" baseline plus per-component ablations, to show how much each explainable stage really contributes.
• I5 (answer): in dialogue-style QA, a reader should not treat an uncontested earlier statement as confirmed, only as not yet challenged.