Elamur Research⌕
☰
home / papers / 2020 / Guiding attention in Sequence-to-sequence models for Dialogue Act prediction

Guiding attention in Sequence-to-sequence models for Dialogue Act prediction

Paper · 2020 · Pierre Colombo, Emile Chapuis, Matteo Manica et al.

Summary

Colombo and colleagues address dialogue act (DA) classification, which assigns each utterance a label for the speaker's intention, such as question, backchannel or opinion. Earlier top systems pair a recurrent encoder with a linear CRF. A CRF links only neighbouring labels, so it misses cases where an answer refers back to a question several turns earlier. The authors instead treat the task like neural machine translation. A window of utterances is the source sequence, and the matching tag sequence is the target. Their seq2seq model has three parts:

  • a hierarchical GRU encoder, with one level over words and one over utterances;
  • a new "guided attention" in the decoder, which uses the fact that each utterance has exactly one tag;
  • beam search, used in inference and also in fine-tuning with a sequence-level loss.

The model is trained end-to-end without hand-engineered features. It reaches 85.0% accuracy on SwDA, about two points above the earlier best of 82.9%. On MRDA it scores 91.6%, which is close to the best reported systems. The work shows that NMT decoding tools carry over to structured labelling tasks with a small tag set.

Key points

  • Benchmarks: SwDA (42 tags) and MRDA (5 tags). Test accuracy was 85.0% on SwDA and 91.6% on MRDA. On MRDA, Li et al. (2018a) report 92.2% and Chen et al. 91.7%.
  • Three encoders were compared: a vanilla averaged-embedding BiGRU, the hierarchical HGRU and a speaker-aware PersoHGRU. HGRU won in most settings. The persona layer brought no reliable gain and lowered accuracy on MRDA.
  • Hard guided attention gave the best overall results on both corpora and adds no parameters. On SwDA, Wilcoxon tests over 10 seeds gave p < 0.01 against the other decoders.
  • Beam search used only at inference added at most 0.2%. Fine-tuning with a beam-based sequence-level loss added about 1 point on SwDA (84→85) and 1.2 points on MRDA (90.4→91.6).
  • Larger beams made the decoder conservative. It drifted towards frequent labels such as sd, or towards runs of one tag like "s s s s s".

Interesting ideas

  • I1. Model the label sequence with its own autoregressive decoder rather than a first-order CRF, so that a tag can depend on tags several steps back (for example, an answer that follows a backchannel and points back to an earlier question).
  • I2. When input and output units are aligned one-to-one, attention can be fixed to the aligned position (the hard version) instead of being learned freely, which costs no parameters and keeps attention interpretable.
  • I3. With a tiny output vocabulary, plain beam search at inference barely helps, but using the beam inside a sequence-level training loss gives a real gain.
  • I4. In imbalanced, small label spaces, wider beams collapse onto majority-class sequences, which is a failure mode worth testing for explicitly.
  • I5. Report weak or harmful additions (here the speaker-persona layer) and back decoder choices with Wilcoxon tests over many seeds.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): the speech-act layer in prag could score acts over a whole dialogue window, so that answer/question pairing survives intervening backchannels; the reason for each act would be shown as an attended earlier turn.
  • I2 (tokens): per-token taggers in crate en that use contextual decoders could fix attention to the aligned token, making each tag decision traceable to one input position.
  • I3 (learn): when training small-label-set components (speech acts, roles), a sequence-level fine-tuning stage may pay off more than larger inference beams.
  • I4 (learn): add a negative control that measures how much a decoder over-predicts the dominant tag as beam width grows, and flag models that drift towards it.
  • I5 (learn): adopt multi-seed significance testing in evaluation, so that a new feature such as speaker turns is kept only if it beats the baseline across seeds.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IBMArmonk · United States85 works · 66,118 citations🎓Institut Polytechnique de ParisPalaiseau · France10 works · 1,555 citations🔬Laboratoire Traitement et Communication de l’InformationPalaiseau · France5 works · 318 citations🎓Télécom ParisPalaiseau · France6 works · 759 citations🔬IBM Research - ZurichRüschlikon · Switzerland2 works · 1,615 citations

Also on the byline: IBM (France).

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 Dialogue systems · 401 works Speech act theory · 160 works

Cite

Cite as Pierre Colombo, Emile Chapuis, Matteo Manica, Emmanuel Vignon, Giovanna Varni and Chloe Clavel. 2020. Guiding attention in Sequence-to-sequence models for Dialogue Act prediction. AAAI 2020.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2002.08801 · arXiv 2002.08801

@inproceedings{colombo2020guiding,
  title = {{Guiding attention in Sequence-to-sequence models for Dialogue Act prediction}},
  author = {Pierre Colombo, Emile Chapuis, Matteo Manica, Emmanuel Vignon, Giovanna Varni and Chloe Clavel},
  booktitle = {AAAI 2020},
  year = {2020},
  doi = {10.48550/arxiv.2002.08801},
  eprint = {2002.08801},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2002.08801}
}

References

30 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [2014] Bahdanau, D.; Cho, K.; and Bengio, Y. 2014. Neural machine translation by jointly learning to align and translate. CoRR abs/1409.0473. ↗ our card
  2. [2017] Bojanowski, P.; Grave, E.; Joulin, A.; and Mikolov, T. 2017. Enriching word vectors with subword information. Transactions of ACL 5:135–146. ↗ our card
  3. [2018] Bothe, C.; Weber, C.; Magg, S.; and Wermter, S. 2018. A context-based approach for dialogue act recognition using simple recurrent neural networks. CoRR abs/1805.06280. ↗ our card
  4. [2018] Chen, Z.; Yang, R.; Zhao, Z.; Cai, D.; and He, X. 2018. Dialogue act recognition via crf-attentive structured network. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, 225– 234. ACM. our card
  5. [2014] Cho, K.; van Merrienboer, B.; Gülçehre, Ç.; Bougares, F.; Schwenk, H.; and Bengio, Y. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR abs/1406.1078. ↗
  6. [2013] Gimpel, K.; Batra, D.; Dyer, C.; and Shakhnarovich, G. 2013. A systematic exploration of diversity in machine translation. In Proceedings of EMNLP 2013, 1100–1111. find
  7. [2003] Janin, A.; Baron, D.; Edwards, J.; Ellis, D.; Gelbart, D.; Morgan, N.; Peskin, B.; Pfau, T.; Shriberg, E.; Stolcke, A.; et al. 2003. The icsi meeting corpus. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03)., volume 1, I–I. IEEE. find
  8. [2015] Jozefowicz, R.; Zaremba, W.; and Sutskever, I. 2015. An empirical exploration of recurrent network architectures. In International Conference on Machine Learning, 2342– 2350. find
  9. [2002] Keizer, S.; op den Akker, R.; and Nijholt, A. 2002. Dialogue act recognition with bayesian networks for dutch dialogues. In Proceedings of the Third SIGdial Workshop on Discourse and Dialogue. our card
  10. [2016] Khanpour, H.; Guntakandla, N.; and Nielsen, R. 2016. Dialogue act classification in domain-independent conversations using a deep recurrent neural network. In COLING. our card
  11. [2014] Kingma, D. P., and Ba, J. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  12. [2018] Kumar, H.; Agarwal, A.; Dasgupta, R.; and Joshi, S. 2018. Dialogue act sequence labeling using hierarchical encoder with crf. In Thirty-Second AAAI Conference on Artificial Intelligence. ↗ our card
  13. [2015] LeCun, Y.; Bengio, Y.; and Hinton, G. 2015. Deep learning. nature 521(7553):436. find
  14. [2016a] Li, J.; Galley, M.; Brockett, C.; Gao, J.; and Dolan, W. B. 2016a. A diversity-promoting objective function for neural conversation models. In HLT-NAACL. ↗ our card
  15. [2016b] Li, J.; Galley, M.; Brockett, C.; Spithourakis, G.; Gao, J.; and Dolan, B. 2016b. A persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of ACL (Volume 1: Long Papers), 994–1003. Association for Computational Linguistics. ↗ our card
  16. [2018a] Li, R.; Lin, C.; Collinson, M.; Li, X.; and Chen, G. 2018a. A dual-attention hierarchical recurrent neural network for dialogue act classification. CoRR. ↗
  17. [2018b] Li, Z.; Cai, J.; He, S.; and Zhao, H. 2018b. Seq2seq dependency parsing. In Proceedings of the 27th International Conference on Computational Linguistics, 3203– 3214. our card
  18. [2017] Loshchilov, I., and Hutter, F. 2017. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101. ↗
  19. [2015] Luong, T.; Pham, H.; and Manning, C. D. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of EMNLP 2015, 1412–1421. Lisbon, Portugal: Association for Computational Linguistics. ↗ our card
  20. [2013] Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; and Dean, J. 2013. Distributed representations of words and phrases and their compositionality. In NeurIPS, 3111–3119. ↗ our card
  21. [2014] Pennington, J.; Socher, R.; and Manning, C. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP 2014, 1532–1543. our card
  22. [2019] Raheja, and Tetreault. 2019. Dialogue act classification with context-aware self-attention. CoRR abs/1904.02594. ↗ our card
  23. [2004] Shriberg, E.; Dhillon, R.; Bhagat, S.; Ang, J.; and Carvey, H. 2004. The icsi meeting recorder dialog act (mrda) corpus. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004. ↗ our card
  24. [2015] Sordoni, A.; Bengio, Y.; Vahabi, H.; Lioma, C.; Grue Simonsen, J.; and Nie, J.-Y. 2015. A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, 553–562. ACM. find
  25. [1998] Stolcke, A.; Shriberg, E.; Bates, R.; Coccaro, N.; Jurafsky, D.; Martin, R.; Meteer, M.; Ries, K.; Taylor, P.; Van Ess-Dykema, C.; et al. 1998. Dialog act modeling for conversational speech. In AAAI Spring Symposium on Applying Machine Learning to Discourse Processing, 98–105. find
  26. [2000] Stolcke, A.; Ries, K.; Coccaro, N.; Shriberg, E.; Bates, R.; Jurafsky, D.; Taylor, P.; Martin, R.; Ess-Dykema, C. V.; and Meteer, M. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics 26(3):339–373. ↗ our card
  27. [2006] Surendran, D., and Levow, G.-A. 2006. Dialog act tagging with support vector machines and hidden markov models. In Ninth International Conference on Spoken Language Processing. find
  28. [2014] Sutskever, I.; Vinyals, O.; and Le, Q. V. 2014. Sequence to sequence learning with neural networks. In NeurIPS, 3104–3112. ↗
  29. [2016] Wiseman, S., and Rush, A. M. 2016. Sequence-to-sequence learning as beam-search optimization. In EMNLP. find
  30. [2016] Wu, Y.; Schuster, M.; Chen, Z.; Le, Q. V.; Norouzi, M.; Macherey, W.; Krikun, M.; Cao, Y.; Gao, Q.; Macherey, K.; et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/colombo-2020-guiding-attention-sequence-sequence-models/