Elamur Research⌕
☰
home / papers / 2019 / Context-aware Neural-based Dialog Act Classification on Automatically Generated Transcriptions

Context-aware Neural-based Dialog Act Classification on Automatically Generated Transcriptions

Paper · 2019 · Daniel Ortega, Chia-Yu Li, Gisela Vallejo et al.

Summary

The paper tackles dialog act (DA) classification, which labels each utterance with the speaker's intention, such as a question, statement or backchannel. Earlier work evaluated on clean human transcripts, so the authors test what happens with speech recognizer output. Their model encodes the current utterance and up to five previous ones with a shared CNN, using word2vec embeddings and utterance-wise max pooling. A linear-chain CRF on top then labels the whole utterance sequence jointly. They evaluate on two benchmark corpora, MRDA (multiparty meetings, 5 tags) and SwDA (two-speaker phone calls, 42 tags). They compare manual transcripts with automatic ones from two ASR systems: a Kaldi hybrid TDNN/HMM and an ESPnet End-to-End model. On manual transcripts the CNN+CRF beats a context-free CNN and reaches state-of-the-art accuracy. On ASR output, E2E transcripts give better DA accuracy than TDNN/HMM ones, even though their word error rates are similar. The paper also shows that punctuation, which ASR output lacks, is a major source of train/test mismatch. This matters for any real voice-driven dialog system.

Key points

  • The context-free CNN baseline scores 80.2% on MRDA and 72.0% on SwDA. CNN+CRF with two previous utterances reaches 84.7% and 74.6%, averaged over 5 runs. Longer contexts (3–5) bring no further gain.
  • Test WERs are close between the two recognizers:
    • SwDA: 18.02 for TDNN/HMM and 18.80 for E2E.
    • MRDA: 21.48 for TDNN/HMM and 18.80 for E2E.
  • Training and testing on E2E transcripts gave the best results on automatic speech. The text reports 76.6% and 68.7%, though the extracted tables are ambiguous about which corpus each figure belongs to.
  • A model trained on MRDA manual transcripts with punctuation falls from 84.7% to 59.2% on TDNN/HMM output and 66.1% on E2E output. Removing punctuation from the training data raises these to 69.3% and 76.2%, about 10 points better.
  • Splits follow Lee & Dernoncourt (2016). Tuning was done only on validation data, changing one hyperparameter at a time.

Interesting ideas

  • I1. Treat a dialog as a sequence and let a CRF learn which DA label tends to follow which, instead of classifying each utterance alone. Neighbouring acts constrain each other, as when a question is followed by an answer.
  • I2. Encode each context utterance separately with the same encoder and then combine the vectors. This keeps context distinguishable from the current utterance, which the cited analysis says is required for context to help.
  • I3. Word error rate alone does not predict how well a downstream task will do: two recognizers with similar WER produced different DA accuracy.
  • I4. Train on input that looks like test input, for example by stripping punctuation that the real source will never provide. A strong cue that disappears at test time hurts more than it helps.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): The speech-act layer in prag could score act transitions across turns, keeping each transition weight inspectable, so an utterance's act is explained partly by the previous one.
  • I2 (frames): When prag decides a speech act or implicature, previous turns should be kept as separate, labelled evidence rather than merged into one span, so explanations can say which turn contributed.
  • I3 (learn): When comparing witness parsers or teachers, Mova should measure downstream frame and answer quality, not only token-level or LAS-style scores.
  • I4 (tokens): Add an evaluation variant of the tokenizer and tagger on de-punctuated, lowercased text, and train robustness on it, so Mova does not silently depend on punctuation it may not get from transcripts or chat.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of StuttgartStuttgart · Germany75 works · 46,113 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 401 worksSpeech act theory · 160 works

Cite

Cite as Daniel Ortega, Chia-Yu Li, Gisela Vallejo, Pavel Denisov and Ngoc Thang Vu. 2019. Context-aware Neural-based Dialog Act Classification on Automatically Generated Transcriptions. arXiv:1902.11060.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1902.11060 · arXiv 1902.11060

@misc{ortega2019context,
  title = {{Context-aware Neural-based Dialog Act Classification on Automatically Generated Transcriptions}},
  author = {Daniel Ortega, Chia-Yu Li, Gisela Vallejo, Pavel Denisov and Ngoc Thang Vu},
  year = {2019},
  doi = {10.48550/arxiv.1902.11060},
  eprint = {1902.11060},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1902.11060}
}

References

29 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. L. Austin, How to Do Things with Words, Oxford university press, 1976. find
  2. K. Bach and R. M. Harnish, Linguistic Communication and Speech Acts, MIT Press, 1979. find
  3. A. Stolcke et al., “Dialogue act modeling for automatic tagging and recognition of conversational speech,” Computational Linguistics, vol. 26, pp. 339–373, 2000. ↗
  4. M. Zimmermann, “Joint segmentation and classification of dialog acts using conditional random fields,” in Proc. of ISCA, 2009. find
  5. J. Y. Lee and F. Dernoncourt, “Sequential short-text classification with recurrent and convolutional neural networks,” in Proc. of NAACL, 2016.
  6. N. Kalchbrenner and P. Blunsom, “Recurrent convolutional neural networks for discourse compositionality,” in Proc. of CVSC, 2013.
  7. D. Ortega and N. T. Vu, “Lexico-acoustic neural-based models for dialog act classification,” in Proc. of ICASSP, 2018.
  8. D. Ortega and N. T. Vu, “Neural-based context representation learning for dialog act classification,” in Proc. of SIGDIAL, 2017.
  9. Eugénio Ribeiro, Ricardo Ribeiro, and David Martins de Mato, “The influence of context on dialogue act recognition,” CoRR, 2015. ↗
  10. G. Lample et al., “Neural architectures for named entity recognition,” in Proc. of NAACL, 2016. ↗
  11. D. Andor et al., “Globally normalized transition-based neural networks,” in Proc. of ACL, 2016.
  12. H. Adel et al., “Global normalization of convolutional neural networks for joint entity and relation classification,” in Proc. of EMNLP, 2017. find
  13. W. Xiong et al., “The microsoft 2017 conversational speech recognition system,” in Proc. of ICASSP, 2018. find
  14. A. Graves et al., “Speech recognition with deep recurrent neural networks,” in Proc. of ICASSP, 2013. ↗
  15. D. Povey et al., “Purely sequence-trained neural networks for ASR based on lattice-free MMI,” in Proc. of INTERSPEECH, 2016. find
  16. S. Watanabe et al., “ESPnet: end-to-end speech processing toolkit,” in Proc. of INTERSPEECH, 2018. find
  17. Y. Miao et al., “EESEN: end-to-end speech recognition using deep RNN models and wfst-based decoding,” in Proc. of IEEE ASRU, 2015. find
  18. D. Bahdanau et al., “End-to-end attention-based large vocabulary speech recognition,” in Proc. of ICASSP, 2016. find
  19. A. Janin et al., “The ICSI meeting corpus,” in Proc. of ICASSP, 2003. find
  20. E. Shriberg et al., “The ICSI meeting recorder dialog act (MRDA) corpus,” in Proc. of SIGdial Workshop on Discourse and Dialogue at HLT-NAACL, 2004. ↗
  21. R. Dhillon et al., “Meeting recorder project: Dialog act labeling guide,” Tech. Rep., ICSI Berkeley CA, 2004. ↗
  22. J. Ang et al., “Automatic dialog act segmentation and classification in multiparty meetings,” in Proc. of ICASSP, 2005. find
  23. S. Calhoun et al, “The NXT-format switchboard corpus: A rich resource for investigating the syntax, semantics, pragmatics and prosody of dialogue,” Language Resources and Evaluation, vol. 44, pp. 387–419, 2010. ↗
  24. B. T. Polyak et al., “Acceleration of stochastic approximation by averaging,” SICON, vol. 30, pp. 838–855, 1992. ↗
  25. J. Duchi et al., “Adaptive subgradient methods for online learning and stochastic optimization,” Journal of Machine Learning Research, vol. 12, pp. 2121–2159, 2011. find
  26. T. Mikolov et al., “Efficient estimation of word representations in vector space,” in Proc. of ICLR, 2013. ↗
  27. D. Povey et al., “The Kaldi speech recognition toolkit,” in Proc. of IEEE ASRU, 2011. find
  28. C. Cieri et al., “The Fisher corpus: a resource for the next generations of speech-to-text,” in Proc. of LREC, 2004.
  29. Z. Xiao et al., “Hybrid ctc-attention based end-to-end speech recognition using subword units,” in Proc. of ISCSLP, 2018. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ortega-2019-context-aware-neural-based-dialog/