Elamur Research⌕
☰
home / papers / 2018 / Exploring Textual and Speech information in Dialogue Act Classification with Speaker Domain Adaptation

Exploring Textual and Speech information in Dialogue Act Classification with Speaker Domain Adaptation

Paper · 2018 · Xuanli He, Quan Tran, William Havard et al.

Summary

Dialogue Act (DA) classification usually assumes human (oracle) transcripts. A real spoken dialogue system only has noisy ASR output, and it also meets speakers it never heard in training. The authors build an online multimodal DA tagger. A CNN over MFCC frames feeds a speech LSTM, a word-embedding LSTM encodes the text, and the two final states are concatenated. Labels come from an uncertainty-propagation layer, in which the softmax parameters are a mixture of DA-specific weights weighted by the predicted distribution for the previous utterance. For unseen speakers they adapt adversarially without labels, following Tzeng et al. (2017). The source model is frozen, and a copy of the speech encoder is retrained until a small domain classifier cannot separate target representations from source ones. On Switchboard (speech synthesised by TTS) and MapTask, adding audio helps clearly when the text comes from CMUSphinx ASR, but adds nothing when oracle text is available. The unsupervised adaptation recovers most of the gap to a model trained with labelled data from the new speakers.

Key points

  • Data:
    • Switchboard has 1155 conversations and 42 DAs, split 1115/21/19 following Stolcke et al. (2000). Its audio was generated by TTS because the original recordings were not available.
    • MapTask has 128 conversations and 13 DAs, split randomly 103/12/13.
  • Switchboard accuracy:
    • With oracle text, text only reaches 74.97% and text+speech 74.98%.
    • With ASR, speech only reaches 59.71%, ASR text 66.39% and text+speech 68.25%.
  • MapTask accuracy:
    • With oracle text, text only reaches 61.73% and text+speech 61.67%.
    • With ASR, the scores collapse to about 38–39%, which the authors attribute to poor audio quality.
  • Speaker adaptation on Switchboard: the source domain is 5 known speakers and the target is 3 new speakers.
    • Speech only goes from 48.73% unadapted to 54.37% adapted, against a supervised upper bound of 56.19%.
    • Text+speech goes from 63.57% to 67.21%, against an upper bound of 68.04%.
  • What gets adapted: only the target speech RNN is updated. Also tuning the CNN, the embeddings or the text RNN gave about the same result.

Interesting ideas

  • I1. Feeding the previous step's full predicted label distribution forward, rather than its single best label, softens the mismatch between training on gold history and running on predicted history.
  • I2. A second, noisier channel is worth most exactly when the main input is degraded. With clean text the audio added nothing, but with ASR text it added about 2 points.
  • I3. Adapt to a new domain without labels by freezing the source model and retraining only the sub-encoder where the shift actually lives, until a domain discriminator is fooled.
  • I4. Report the unsupervised result between two anchors: the unadapted model below and a model trained on a small labelled target sample above.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): A speech-act tagger over a dialogue or story could condition each utterance's act on the soft distribution of the previous act, so that one early mistake does not cascade.
  • I2 (learn): Evaluation should report gains separately on clean and on degraded input (typos, OCR noise, ASR text), since helpers that look useless on clean data may matter on noisy data.
  • I3 (tree): The GPU parser gparse could adapt only its lower encoder layers to an unlabelled new genre (fables, school texts) while keeping the trained arc and label heads fixed.
  • I4 (learn): Silver-data or repair-rule experiments could always include a "small gold target set" ceiling, so we can say how much of the reachable gain was actually recovered.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Monash UniversityMelbourne · Australia31 works · 54,201 citations🔬Laboratoire d'Informatique de GrenobleGrenoble · France8 works · 522 citations🎓Institut Universitaire de FranceParis · France4 works · 69 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 3 Dialogue systems · 401 works Speech & spoken language · 233 works Speech act theory · 160 works

Cite

Cite as Xuanli He, Quan Tran, William Havard, Laurent Besacier, Ingrid Zukerman and Gholamreza Haffari. 2018. Exploring Textual and Speech information in Dialogue Act Classification with Speaker Domain Adaptation. Proceedings of the Australasian Language Technology Association Workshop 2018.

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1810.07455

@inproceedings{he2018exploring,
  title = {{Exploring Textual and Speech information in Dialogue Act Classification with Speaker Domain Adaptation}},
  author = {Xuanli He, Quan Tran, William Havard, Laurent Besacier, Ingrid Zukerman and Gholamreza Haffari},
  booktitle = {Proceedings of the Australasian Language Technology Association Workshop 2018},
  year = {2018},
  eprint = {1810.07455},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.07455}
}

References

16 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. CoRR cs.CL/0006023. our card
  2. Anne H Anderson, Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jacqueline Kowtko, Jan McAllister, Jim Miller, et al. 1991. The hcrc map task corpus. Language and speech 34(4):351–366. ↗
  3. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. CoRR abs/1409.3215. ↗
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR abs/1409.0473. ↗ our card
  5. Quan Hung Tran, Ingrid Zukerman, and Gholamreza Haffari. 2017a. A hierarchical neural model for learning sequences of dialogue acts. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. volume 1, pages 428–437. our card
  6. John J Godfrey, Edward C Holliman, and Jane Mc-Daniel. 1992. Switchboard: Telephone speech corpus for research and development. In Acoustics, Speech, and Signal Processing, 1992. ICASSP-92., 1992 IEEE International Conference on. IEEE, volume 1, pages 517–520. find
  7. Quan Hung Tran, Ingrid Zukerman, and Gholamreza Haffari. 2017b. Preserving distributional information in dialogue act classification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pages 2141–2146. our card
  8. Yangfeng Ji, Gholamreza Haffari, and Jacob Eisenstein. 2016. A latent variable recurrent neural network for discourse relation language models. CoRR abs/1603.01913. ↗
  9. Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. 2017. Adversarial discriminative domain adaptation. CoRR abs/1702.05464. ↗
  10. Fatema N. Julia, Khan M. Iftekharuddin, and Atiq U. Islam. 2010. Dialog act classification using acoustic and discourse information of maptask data. International Journal of Computational Intelligence and Applications 09(04):289–311. ↗
  11. D. Jurafsky, E. Shriberg, and D. Biasca. 1997. Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual. Technical Report Draft 13, University of Colorado, Institute of Cognitive Science. find
  12. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR abs/1412.6980. http://arxiv.org/abs/1412.6980. ↗
  13. Abdel-rahman Mohamed, Geoffrey Hinton, and Gerald Penn. 2012. Understanding how deep belief networks perform acoustic modelling. In Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on. IEEE, pages 4273– 4276. find
  14. Ramesh Nallapati, Bing Xiang, and Bowen Zhou. 2016. Sequence-to-sequence rnns for text summarization. CoRR abs/1602.06023. ↗
  15. Alexander Popov. 2016. Deep Learning Architecture for Part-of-Speech Tagging with Word and Suffix Embeddings, Springer International Publishing, Cham, pages 68–77. find
  16. Sheng-syun Shen and Hung-yi Lee. 2016. Neural attention models for sequence classification: Analysis and application to key term extraction and dialogue act detection. CoRR abs/1604.00077. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca A. Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/he-2018-exploring-textual-speech-information-dialogue/