Elamur Research⌕
☰
home / papers / 2020 / Local Contextual Attention with Hierarchical Structure for Dialogue Act Recognition

Local Contextual Attention with Hierarchical Structure for Dialogue Act Recognition

Paper · 2020 · Zhigang Dai, Jinhua Fu, Qile Zhu et al.

Summary

Dialogue act (DA) recognition assigns each utterance a label for the speaker's intention, such as question, acknowledgement or greeting. The same short reply ("Okay.") can carry different acts depending on its surroundings. Earlier hierarchical models attended over the whole conversation, which lets distant, unrelated turns add noise.

The authors build a two-level model:

  • An LSTM with max-pooling turns each utterance into a vector.
  • A self-attention layer with a residual connection models how utterances relate to each other.
  • The attention scores receive a learnable Gaussian bias based on relative position. Each utterance predicts its own focus center, kept within a fixed distance of itself, and its own spread, so attention favours nearby turns.

They also split dialogues into sliding windows with extra context padding. This lets them measure how dialogue length and amount of context affect accuracy, both offline (future turns visible) and online (only past turns visible).

On Switchboard (SwDA) and DailyDialog the model reaches 80.34% and 85.81% accuracy. Attention maps show the bias suppresses spurious long-range links.

Key points

  • Datasets are Switchboard Dialogue Act (SwDA) and DailyDialog. The best reported accuracies are 80.34% and 85.81%.
  • The architecture is an LSTM utterance encoder with max-pooling, then utterance-level self-attention with a residual connection, then a two-layer ReLU classifier. Max-pooling did slightly better than taking the last LSTM state.
  • The Gaussian bias added before the softmax works as follows:
    • Center = i + C·tanh(·), which keeps it within C positions of the utterance.
    • Width = D·sigmoid(·).
    • Both are learned for each utterance.
  • Segmentation has two settings: a window size W and a context padding length P.
    • The baselines (LSTM+BLSTM, LSTM+Attention) peaked at W=1 and degraded as W grew.
    • LSTM+LC Attention stayed robust and peaked at larger windows: W=10 on SwDA and W=5 with P=2 on DailyDialog.
  • Results on DailyDialog:
    • Sentence-only baselines scored 82.22 (CNN), 82.58 (LSTM) and 83.22 (fine-tuned BERT).
    • In the online setting, LC Attention scored 84.83 versus 84.68 for LSTM+Attention and 84.55 for LSTM+LSTM.
    • SwDA online prediction used 5 preceding turns; DailyDialog used 2.

Interesting ideas

  • I1. Adding a learnable distance penalty to attention logits, with a center allowed to shift but only within a bounded range, injects a locality prior while still letting the model pick which neighbour matters most.
  • I2. Evaluating with a sliding window plus separate context padding cleanly separates "how much text is labelled" from "how much context is visible".
  • I3. Reporting both offline accuracy (future turns visible) and online accuracy (past only) shows how much a system relies on look-ahead.
  • I4. Visualizing attention matrices to flag implausible long-distance links offers a cheap sanity check on what context a model uses.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): In speech-act and coreference resolution within crate prag, candidate antecedent or triggering utterances could be scored with an explicit, inspectable distance prior, so the choice of context stays explainable.
  • I2 (learn): Evaluation of discourse-level components could sweep window and padding sizes to show how much context each decision truly needs, with zero-context runs serving as a negative control.
  • I3 (answer): Readers answering questions on fables could be tested with incremental, prefix-only input to check that answers do not silently depend on later sentences.
  • I4 (frames): Linking decisions between utterances could be dumped as a matrix and checked for long-range links that lack justification, which would feed self-induced repair rules.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Ant GroupHangzhou · China5 works · 633 citations🎓South China University of TechnologyGuangzhou · China8 works · 1,433 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 Dialogue systems · 401 works Speech act theory · 160 works

Cite

Cite as Zhigang Dai, Jinhua Fu, Qile Zhu, Hengbin Cui, Xiaolong li and Yuan Qi. 2020. Local Contextual Attention with Hierarchical Structure for Dialogue Act Recognition. arXiv:2003.06044.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2003.06044 · arXiv 2003.06044

@misc{dai2020local,
  title = {{Local Contextual Attention with Hierarchical Structure for Dialogue Act Recognition}},
  author = {Zhigang Dai, Jinhua Fu, Qile Zhu, Hengbin Cui, Xiaolong li and Yuan Qi},
  year = {2020},
  doi = {10.48550/arxiv.2003.06044},
  eprint = {2003.06044},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2003.06044}
}

References

35 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Conneau Alexis, Kiela Douwe, Schwenk Holger, Barraul Lo c, and Bordes Antoine. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, page 670680. our card
  2. Nicholas Asher, Julie Hunter, Mathieu Morey, Farah Benamara, and Stergos D Afantenos. 2016. Discourse structure and dialogue acts in multiparty dialogue: the stac corpus. In LREC. our card
  3. J. L. Austin. 1962. How to Do Things with Words. Clarendon Press. find
  4. Chandrakant Bothe, Cornelius Weber, Sven Magg, and Stefan Wermter. 2018. A context-based approach for dialogue act recognition using simple recurrent neural networks. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018). our card
  5. Lin Chen and Barbara Di Eugenio. 2013. Multimodality and dialogue act classification in the robohelper project. In Proceedings of the SIGDIAL 2013 Conference, pages 183–192. our card
  6. Zheqian Chen, Rongqin Yang, Zhou Zhao, Deng Cai, and Xiaofei He. 2018. Dialogue act recognition via crf-attentive structured network. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 225–234. ACM. our card
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗ our card
  8. Alfred Dielmann and Steve Renals. 2008. Recognition of dialogue acts in multiparty meetings using a switching dbn. IEEE transactions on audio, speech, and language processing, 16(7):1303–1314. ↗
  9. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778. ↗
  10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. ↗
  11. Yangfeng Ji, Gholamreza Haffari, and Jacob Eisenstein. 2016. A latent variable recurrent neural network for discourse relation language models. In Proceedings of NAACL-HLT, pages 332–342. ↗
  12. Dan Jurafsky, Elizabeth Shriberg, and Debra Biasca. 1997. Switchboard swbd-damsl labeling project coders manual. Draft 13. Technical Report 97-02. find
  13. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent convolutional neural networks for discourse compositionality. In Proceedings of the Workshop on Continuous Vector Space Models and their Compositionality, pages 119–126. our card
  14. Hamed Khanpour, Nishitha Guntakandla, and Rodney Nielsen. 2016. Dialogue act classification in domain-independent conversations using a deep recurrent neural network. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2012– 2021. our card
  15. Su Nam Kim, Lawrence Cavedon, and Timothy Baldwin. 2010. Classifying dialogue acts in one-on-one live chats. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 862–871. Association for Computational Linguistics. our card
  16. Harshit Kumar, Arvind Agarwal, Riddhiman Dasgupta, and Sachindra Joshi. 2018. Dialogue act sequence labeling using hierarchical encoder with crf. In Thirty-Second AAAI Conference on Artificial Intelligence. ↗ our card
  17. Ji Young Lee and Franck Dernoncourt. 2016. Sequential short-text classification with recurrent and convolutional neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 515–520. our card
  18. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986–995. Asian Federation of Natural Language Processing. ↗ our card
  19. Yang Liu, Kun Han, Zhao Tan, and Yun Lei. 2017. Using context information for dialog act classification in dnn framework. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2170–2178. our card
  20. Dmitrijs Milajevs and Matthew Purver. 2014. Investigating the contribution of distributional semantic information for dialogue act classification. In Proceedings of the 2nd Workshop on Continuous Vector Space Models and their Compositionality (CVSC), pages 40–47. our card
  21. Ryosuke Nakanishi, Koji Inoue, Shizuka Nakamura, Katsuya Takanashi, and Tatsuya Kawahara. 2018. Generating fillers based on dialog act pairs for smooth turn-taking by humanoid robot. In Proc. Intl Workshop Spoken Dialogue Systems (IWSDS). ↗
  22. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. ↗ our card
  23. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237. our card
  24. Vipul Raheja and Joel Tetreault. 2019. Dialogue act classification with context-aware self-attention. arXiv preprint arXiv:1904.02594. ↗ our card
  25. John Rogers Searle. 1969. Speech acts: An essay in the philosophy of language, volume 626. Cambridge university press. find
  26. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), volume 2, pages 464–468. our card
  27. Zhouxing Shi and Minlie Huang. 2018. A deep sequential model for discourse parsing on multi-party dialogues. arXiv preprint arXiv:1812.00176. ↗
  28. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339–373. ↗ our card
  29. Maryam Tavafi, Yashar Mehdad, Shafiq Joty, Giuseppe Carenini, and Raymond Ng. 2013. Dialogue act recognition in synchronous and asynchronous conversations. In Proceedings of the SIGDIAL 2013 Conference, pages 117–121. ↗ our card
  30. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. our card
  31. Yizhong Wang, Sujian Li, and Houfeng Wang. 2017. A two-stage parsing method for text-level discourse analysis. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 184–188. ↗ our card
  32. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. ↗
  33. Baosong Yang, Zhaopeng Tu, Derek F Wong, Fandong Meng, Lidia S Chao, and Tong Zhang. 2018. Modeling localness for self-attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4449– 4458. our card
  34. Tiancheng Zhao, Kyusong Lee, and Maxine Eskenazi. 2018. Unsupervised discrete sentence representation learning for interpretable neural dialog generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1098–1107. Association for Computational Linguistics. find
  35. Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. arXiv preprint arXiv:1703.10960. ↗ our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dai-2020-local-contextual-attention-hierarchical-structure/