📄 Detecting Interrogative Utterances with Recurrent Neural Networks

Summary

The paper asks whether a spoken utterance is a question or a statement. This is a useful signal for speech translation, where speech recognition output often loses it. Earlier systems relied on hand-crafted features and shallow classifiers such as SVMs or decision trees. The authors instead train recurrent networks that learn their own features from text transcripts, audio (MFCC frames), or both. They test 7 architectures:

• text-only, audio-only and two ways of fusing the modalities;
• two pooling choices, either the last hidden state or attention-weighted averaging.

Each architecture is trained with GRU or LSTM cells and with no regularization, dropout or batch normalization, giving 42 configurations. These are evaluated on MSR-Skype and a CALLHOME subset. Combining text and audio gives the top F1 scores: 91.0 on MSR-Skype and 84.6 on CALLHOME. Batch normalization mainly helps models that read text. Attention pooling helps the audio-only models, which see long sequences. Audio proves important for declarative questions, which have no lexical question markers. The study is an early systematic comparison of neural design choices for multimodal question detection.

Key points

• MSR-Skype has 18,006 manually segmented, roughly balanced text–audio pairs, restricted to 3–25 words. CALLHOME (human transcripts) has 2,528 pairs. Both use an 80/20 split.
• The models are bidirectional RNNs with 200 hidden units per layer, ReLU feedforward layers and a logistic output. Word embeddings are learned from scratch over a joint vocabulary of 13,911 words.
• Audio features are MFCC with 40 ms frames and 15 ms overlap. Every four frames are merged into one step to shorten the sequences.
• Best scores:
  • MSR-Skype: combinational fusion with attention, BN and LSTM reaches F1 91.0.
  • CALLHOME: combinational fusion with last-state pooling, BN and GRU reaches F1 84.6.
  • Audio-only models stay far lower, around 66–81.
• All models do best on utterances of 5–20 words. Fused models lose the least accuracy on very short or very long inputs; the conditional model scores 79.3 / 91.2 / 84.0 on short / intermediate / long.

Interesting ideas

• I1. The authors deleted question marks and other punctuation from transcripts so the classifier could not cheat on a trivial surface cue and had to learn real evidence.
• I2. Questions are split into yes-no, wh-word and declarative types, and declarative ones are shown to be undetectable from words alone.
• I3. Scores are broken down by utterance length buckets, which shows where each model degrades instead of hiding it in one average.
• I4. When text and audio disagree, the fused models produce compromise scores rather than trusting either source, which blurs the decision.
• I5. Attention-style pooling over all time steps pays off mainly for long inputs, while taking just the final state suffices for short ones.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Build a negative-control test set where '?' is stripped and check that the speech-act component still identifies questions from syntax (inversion, wh-words), proving it is not just reading punctuation.
• I2 (frames): Tag speech acts with the type of evidence that triggered them (auxiliary inversion, wh-fronting, or only a final '?'), so declarative questions are explicitly marked as resting on punctuation alone.
• I3 (learn): Report parser and frame accuracy per sentence-length bucket, so regressions on very short fragments or very long sentences become visible rather than being diluted.
• I4 (frames): When syntactic form and punctuation conflict (a declarative ending in '?'), keep both signals and record the conflict in the explanation, instead of averaging them into an opaque confidence.
• I5 (tree): In the gparse transformer parser, consider choosing the sentence-level pooling by input length, and confirm the choice with length-bucketed evaluation.