📄 Contextual Dialogue Act Classification for Open-Domain Conversational Agents

Summary

The paper addresses dialogue act (DA) recognition, which means labelling each user turn with its communicative function, such as an opinion, a yes/no question or a closing. The setting is open-domain human-machine chat, where labelled data is scarce, user turns are very short and ASR output is noisy. The authors present CDAC, a compact neural classifier. For each utterance it combines CNN encodings of Word2Vec embeddings and POS tags with simple lexical counts and with features describing the bot's own state. It then adds the predicted DAs of a few earlier turns as context. Only past turns are used, so the model can run in real time. CDAC is first trained on the human-human Switchboard corpus and then fine-tuned on a small hand-labelled sample of Alexa Prize 2018 conversations. On Switchboard it comes close to the strongest, more complex contextual model. On Alexa data it beats tf-idf SVM and Multinomial Bayes baselines, and transfer learning adds a further gain. The work shows that cheap transfer from human-human data, combined with explicit system state, makes DA tagging feasible for live chatbots.

Key points

• On Switchboard, CDAC with a 3-turn context reaches 76.70 accuracy. This is a relative gain of about 8.0% over the Stolcke et al. baseline (71.00) and close to Bothe et al. (77.34). Annotator agreement is 84.00.
• On Alexa Prize data with a 3-turn context:
  • The context-augmented baselines score 68.70 (SVM) and 59.26 (Multinomial Bayes).
  • CDAC alone scores 73.25.
  • CDAC pre-trained on Switchboard and then fine-tuned scores 75.34.
• The human-machine label set covers only the 13 most frequent DAs, against 42 in Switchboard, yet accuracy is still lower. User utterances average just 2.8 words.
• Fine-tuning updates all weights at a learning rate of 1e-4, ten times smaller than the 1e-3 used in pre-training.
• In the ablation with a 1-turn window, removing both lexical and syntactic features costs about 1.6%. The most frequent confusion is between statement-opinion and open-question.

Interesting ideas

• I1. The context fed to the model is the earlier turns' predicted DA labels, not their raw text, which keeps the context signal compact and easy to interpret.
• I2. The system's own state (current topic, the topic or item it just suggested) is treated as an explicit feature, because the agent knows it exactly, unlike a human partner's mind.
• I3. Only earlier turns are used, so classification works incrementally, whereas prior systems peeked at future utterances.
• I4. The model is trained on a large related-domain corpus and then fine-tuned on a small target sample at a much lower learning rate, which cuts annotation needs.
• I5. The error analysis singles out one ambiguity, opinion versus request, as the dominant failure that only context can resolve.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): The speech-act layer in prag could condition each sentence's act on the acts already assigned to preceding sentences, so the reason for a decision stays visible as a short label sequence.
• I2 (answer): When a reader answers a follow-up question, its own previous answer or offered entity could be recorded as structured state that helps interpret the short next user turn.
• I3 (world): Building the world of a text strictly left-to-right, without lookahead, would make each state update explainable at the moment it happens and suit streaming input.
• I4 (learn): The gparse model could be pretrained on large UD treebanks and then gently tuned on a small set of verified dialogue or tale sentences, without overwriting general syntax.
• I5 (frames): Evaluation of speech-act tagging could report a confusion matrix and flag such pairs, and those cases could trigger an explicit rule that consults the previous turn.