Elamur Research⌕
☰
home / papers / 2018 / Lexico-acoustic Neural-based Models for Dialog Act Classification

Lexico-acoustic Neural-based Models for Dialog Act Classification

Paper · 2018 · Daniel Ortega and Ngoc Thang Vu

Summary

Ortega and Vu ask whether the sound of speech, and not only its transcript, helps assign dialog acts (DAs) such as Statement, Question or Backchannel to utterances in spoken conversation. Earlier neural DA taggers worked on words alone. The authors pair two networks. The first is a lexical CNN that encodes the current utterance and a few preceding ones, then combines them with an LSTM and an attention layer (ROA). The second is an acoustic CNN that reads frame-level MFCC features. The two utterance vectors are joined before a final softmax. On MRDA (5 tags) and the NXT-format Switchboard (SwDA, 42 tags), the combined model beats each part alone. Gains are small with clean transcripts. They grow when the question mark is stripped from the text, which mimics ASR output, and they show up on one-word replies. Acoustic cues therefore seem most useful where lexical evidence is weak. They also need enough training examples per class to pay off.

Key points

  • Accuracy:
    • MRDA: lexical 84.1, acoustic 67.8, lexico-acoustic (LAM) 84.7.
    • SwDA: 73.6, 50.9 and 75.1.
    • Majority-class baselines are 59.1 and 34.7.
  • Acoustic input:
    • 13 MFCCs per 25 ms frame with a 10 ms shift, extracted with openSMILE.
    • One CNN layer with filters covering all 13 coefficients × 5 frames, followed by max pooling.
  • SwDA data: The NXT-format version was chosen for its word-level time stamps. Only about half of it carries DA labels, so the splits are roughly halves of those in earlier work. As a result, SwDA comparisons with other papers are not strictly like-for-like; CNN+DAP reports 79.9 on different data.
  • Question-mark ablation:
    • Overall accuracy falls to 80.8 (LM) and 81.9 (LAM), so the LAM margin grows from 0.6 to 1.1.
    • On Question utterances that originally had "?", the LM collapses from 97.7% to 46.6%, while the LAM reaches 50.2%.
  • Where gains land: Improvements cluster in the most frequent DAs (e.g., Statement, Backchannel, Opinion). A few classes lose slightly, and most rare SwDA classes are unchanged.

Interesting ideas

  • I1. Removing the single strongest surface cue (the "?") reveals whether a second signal carries real information or was merely masked by an easy shortcut.
  • I2. Each channel is encoded separately and the two are fused only at the utterance level, so each source's contribution can be measured by comparing the parts with the whole.
  • I3. A per-class breakdown shows that the added signal mainly helps well-populated classes, which is hidden by a single overall accuracy number.
  • I4. The DA of an utterance is predicted using the previous one to three turns, weighted by attention, rather than in isolation.
  • I5. One-word replies like "yeah" or "right" are ambiguous between Statement and Backchannel, and extra non-lexical evidence helps separate them.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Use cue-ablation as a negative control in evaluation, e.g. strip punctuation or key function words and check whether the frame and speech-act rules still fire for the right reasons.
  • I2 (frames): Keep separate, inspectable evidence sources for speech-act decisions (punctuation, word order, preceding turn) and log which source decided, so every act label stays explainable.
  • I3 (learn): Report silver-data and repair-rule gains per speech-act or frame type alongside class frequency, to spot rules that only help frequent cases or quietly hurt rare ones.
  • I4 (frames): Speech-act classification in prag could explicitly consult the prior turn (e.g., a question before "right"), with the chosen context turn recorded as a visible justification.
  • I5 (frames): Treat bare response tokens as a dedicated case whose act is resolved from dialog context rather than from the word itself, so they do not get a fixed lexical label.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of StuttgartStuttgart · Germany75 works · 46,113 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSignal Processing · 213 works

🧭 Topics 3Research subjects and languages the work is about.Dialogue systems · 401 worksSpeech & spoken language · 233 worksSpeech act theory · 160 works

Cite

Cite as Daniel Ortega and Ngoc Thang Vu. 2018. Lexico-acoustic Neural-based Models for Dialog Act Classification. arXiv:1803.00831.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1803.00831 · arXiv 1803.00831

@misc{ortega2018lexico,
  title = {{Lexico-acoustic Neural-based Models for Dialog Act Classification}},
  author = {Daniel Ortega and Ngoc Thang Vu},
  year = {2018},
  doi = {10.48550/arxiv.1803.00831},
  eprint = {1803.00831},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1803.00831}
}

References

24 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J.L. Austin, How to Do Things with Words, 1962. find
  2. Kent Bach, Concise Routledge Encyclopedia of Philosophy, chapter Speech Acts, 2000. find
  3. Andreas Stolcke, Noah Coccaro, Rebecca Bates, Paul Taylor, Carol Van Ess-Dykema, Klaus Ries, Elizabeth Shriberg, Daniel Jurafsky, Rachel Martin, and Marie Meteer, “Dialogue act modeling for automatic tagging and recognition of conversational speech,” Comput. Linguist., 2000. ↗
  4. Matthias Zimmermann, “Joint segmentation and classification of dialog acts using conditional random fields.,” in INTERSPEECH, 2009. find
  5. M. Henderson, M. Gai, B. Thomson, P. Tsiakoulis, K. Yu, and S. Young, “Discriminative spoken language understanding using word confusion networks,” in IEEE SLT, 2012. find
  6. Nal Kalchbrenner and Phil Blunsom, “Recurrent convolutional neural networks for discourse compositionality,” CoRR, 2013.
  7. Ji Young Lee and Franck Dernoncourt, “Sequential short-text classification with recurrent and convolutional neural networks,” CoRR, 2016.
  8. Yangfeng Ji, Gholamreza Haffari, and Jacob Eisenstein, “A latent variable recurrent neural network for discourse relation language models,” CoRR, 2016. ↗
  9. Sheng-syun Shen and Hung-yi Lee, “Neural attention models for sequence classification: Analysis and application to key term extraction and dialogue act detection,” CoRR, 2016. find
  10. Elizabeth Shriberg, Rebecca A. Bates, Andreas Stolcke, Paul Taylor, Daniel Jurafsky, Klaus Ries, Noah Coccaro, Rachel Martin, Marie Meteer, and Carol Van Ess-Dykema, “Can prosody aid the automatic classification of dialog acts in conversational speech?,” CoRR, 2000. find
  11. H. Arsikere, A. Sen, A. P. Prathosh, and V. Tyagi, “Novel acoustic features for automatic dialog-act tagging,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016. find
  12. Stanislav Ond and Jozef Juhr, “Distance-based dialog acts labeling,” in Cognitive Infocommunications (CogInfoCom), 2015 6th IEEE International Conference on, 2015. find
  13. Daniel Ortega and Ngoc Thang Vu, “Neural-based context representation learning for dialog act classification,” in SIGdial, 2017.
  14. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio, “Neural machine translation by jointly learning to align and translate,” CoRR, 2014. ↗
  15. Florian Eyben, Felix Weninger, Florian Gross, and Björn Schuller, “Recent developments in opensmile, the munich open-source multimedia feature extractor,” in ACM International Conference on Multimedia, 2013. find
  16. Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey, “The icsi meeting recorder dialog act (mrda) corpus,” in SIGdial, 2004. ↗
  17. Jeremy Ang, Yang Liu, and Elizabeth Shriberg, “Automatic dialog act segmentation and classification in multi-party meetings,” in ICASSP, 2005. find
  18. Sasha Calhoun, Jean Carletta, Jason M. Brenier, Neil Mayo, Dan Jurafsky, Mark Steedman, and David Beaver, “The nxt-format switchboard corpus: A rich resource for investigating the syntax, semantics, pragmatics and prosody of dialogue,” Lang. Resour. Eval., 2010. ↗
  19. John J. Godfrey, Edward C. Holliman, and Jane Mc-Daniel, “Switchboard: Telephone speech corpus for research and development,” in ICASSP, 1992. find
  20. D. Jurafsky, E. Shriberg, and D. Biasca, “Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual,” Tech. Rep., 1997. find
  21. B. T. Polyak and A. B. Juditsky, “Acceleration of stochastic approximation by averaging,” SIAM J. Control Optim., 1992. ↗
  22. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean, “Distributed representations of words and phrases and their compositionality,” CoRR, 2013. ↗
  23. Yang Liu, Kun Han, Zhao Tan, and Yun Lei, “Using context information for dialog act classification in dnn framework,” in EMNLP, 2017.
  24. Gang Ji and Jeff Bilmes, “Backoff model training using partially observed data: Application to dialog act tagging,” in HLT-NAACL, 2006. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ortega-2018-lexico-acoustic-neural-based-models/