📄 Hierarchical Fusion for Online Multimodal Dialog Act Classification

Summary

This paper tackles online dialog act (DA) classification. Here no human (oracle) transcripts are available, and a system must work from raw speech plus whatever an ASR system produces. Earlier multimodal DA approaches used audio poorly and merged modalities only at a late stage. The authors transcribe speech with Whisper and take audio frame features from Whisper's last encoder layer. Text token features come from a frozen RoBERTa. Their Hierarchical Fusion (HF) model encodes each modality with a Bi-LSTM and self-attention pooling. It joins text and audio at the utterance level before attention-based dialog-context modeling. Separate unimodal context encoders add further information, and a late-fusion classifier merges all branches. On MRDA and EMOTyDA, the model beats prior state-of-the-art approaches by about 3 F1 points. It also beats a text-only baseline by roughly 4.5–5.5 F1 points. The work shows that fusing modalities early, with strong pretrained audio features, makes DA recognition more robust to noisy ASR. This matters for real-time conversational agents.

Key points

• Inputs and features: Whisper serves as both the ASR system and the audio feature extractor, and RoBERTa supplies text embeddings. Whisper features outperformed MFCC, eGeMAPS and Wav2Vec2 embeddings.
• Ablation results:
  • Early fusion improved F1 by about 2 points over late fusion on both datasets.
  • Adding the unimodal encoders raised accuracy by about 2 points.
  • Cross-attention in utterance encoding gave no notable benefit.
• Class-wise gains: HF improved 32 of 52 MRDA classes, including +24 F1 on Or Question (qr), and 10 of 12 EMOTyDA classes.
• MRDA with oracle transcripts: HF reaches 91.8 accuracy, which is behind only Li et al. (2019) at 92.2 and Chapuis et al. (2020) at 92.4. The online HF accuracy is 82.6.
• EMOTyDA:
  • With oracle input, HF (63.42 accuracy / 50.3 F1) exceeds the best Saha et al. (2020) model (51.00 / 44.52).
  • In the online setting, HF reaches 55.58 accuracy and 41.95 F1.
  • The code is publicly released.