📄 A Generative Attentional Neural Network Model for Dialogue Act Classification

Summary

The paper addresses dialogue act (DA) classification, which assigns a speech-act label to each utterance in a conversation. The authors build a generative neural model of the whole dialogue. At each turn, the DA label depends on the previous label and the previous utterance, and the current utterance's tokens are then produced by an RNN conditioned on that label and on the prior turn. Although the model is generative, it is trained discriminatively by maximising the conditional likelihood of the label sequence. The model has two novelties. The first is an HMM-like link from one label to the next, implemented by switching softmax parameters (or only the bias) according to the previous DA. The second is a "gated attention" that rescales each input embedding before it enters the RNN, which avoids the usual tendency of attention to favour the end of a sequence. Because the utterance encodings do not depend on the labels, exact Viterbi decoding is possible. On MapTask and Switchboard, the model beats strong baselines. Ablations show that the label-transition link contributes more than attention.

Key points

• Data: the MapTask corpus has 128 dialogues, over 27k utterances and 13 DA types, split 103/13/12 into train/test/validation. Switchboard has 1155 calls, about 205k utterances and 42 DAs, collapsed from 226.
• On MapTask, the best variants reach 65.94% accuracy, compared with 60.97% for Ji et al. (2016) rerun under the same settings and 55.72% for an n-gram random forest.
• On Switchboard, the variant with gated attention and gated bias reaches 74.2%, compared with 73.9% for Kalchbrenner and Blunsom (2013) and 71.0% for an HMM (Stolcke et al., 2000).
• In a 3×3 ablation over label connection and attention type, gated attention beats standard attention by 0.49–1.21 points, while the HMM link adds 2.58–3.73 points. McNemar's test shows the full model beats the plain variant at α < 0.01.
• The implementation uses the CNN toolkit with Adagrad and dropout, 128-dimensional embeddings and 64-dimensional hidden states. Switchboard training on a CPU took up to four weeks.

Interesting ideas

• I1. Applying attention as a multiplicative gate on each token's embedding before the recurrent layer, rather than as weights over hidden states, removes the bias toward late positions, since late hidden states summarise everything that came before.
• I2. Conditioning the classifier for the current label on the previous label by swapping in label-specific output parameters, with a cheaper option that swaps only the bias, adds an explicit transition model to a neural classifier.
• I3. The model is deliberately designed so that utterance encodings do not depend on the label sequence, which keeps exact dynamic-programming decoding available even inside a neural model.
• I4. The method is evaluated as a full factorial ablation of two design choices, with McNemar's test on paired predictions, and competing systems' code is rerun under the same split instead of trusting reported numbers.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): A speech-act cue can sit anywhere in a sentence, so scoring each token's relevance on its own lets prag explain which specific word triggered a question or request reading.
• I2 (frames): A learned table of transitions between successive speech acts (e.g. question followed by answer) could serve as an inspectable prior when prag labels utterances in a dialogue or a fable's exchanges.
• I3 (tokens): A tagger could keep its neural features label-independent and put all label interaction into transition scores, so Viterbi gives exact, explainable best sequences.
• I4 (learn): Mova's evaluation could adopt paired McNemar tests and factorial ablations of repair rules or witness parsers, so that each component's contribution is shown to be significant rather than noise.