Elamur Research⌕
☰
home / papers / 2021 / A Context-Aware Hierarchical BERT Fusion Network for Multi-turn Dialog Act Detection

A Context-Aware Hierarchical BERT Fusion Network for Multi-turn Dialog Act Detection

Paper · 2021 · Ting-Wei Wu, Ruolin Su and Biing-Hwang Juang

Summary

Most spoken language understanding (SLU) systems look at each utterance alone. They therefore miss how dialog acts in one turn depend on earlier turns, and they usually assume each utterance carries only one intent. This paper presents CaBERT-SLU, a model that reads a whole multi-turn dialog. For every turn it predicts a set of dialog acts and tags slots.

Each utterance is encoded with BERT, and multi-head self-attentive pooling turns the tokens into a sentence vector. A masked, unidirectional transformer then mixes these sentence vectors across turns, and an LSTM on top adds dialog-level order. Act detection is treated as multi-label classification with sigmoid outputs, while a BiLSTM tagger over BERT token states predicts slots. Both losses are trained jointly.

On the Microsoft Dialogue Challenge (MDC) and Schema-Guided Dialogue (SGD) datasets, the model beats single-utterance multi-intent baselines and the contextual baseline BERT+CASA-NLU. The margin is largest on SGD. The work shows that dialog history matters for recognizing several co-occurring acts.

Key points

  • Architecture: BERT utterance encoder → self-attentive pooling (K heads, following Lin et al.) → masked unidirectional transformer "context fusion encoder" with positional encoding → unidirectional LSTM "global recurrent unit" → sigmoid act logits.
    • Slots come from a BiLSTM with softmax.
    • The joint loss is binary cross entropy for acts plus cross entropy for slots.
  • MDC results (CaBERT-SLU): ID F1 91.26, ID exact-match accuracy 83.05, slot F1 79.64. The strongest baseline, BERT+CASA-NLU, scored 90.98 / 82.17 / 78.16.
  • SGD results (CaBERT-SLU): ID F1 99.14, ID accuracy 98.59, slot F1 95.71. BERT+CASA-NLU scored 97.07 / 95.24 / 90.46. The improvements are reported as significant at p < 0.05 against CaBERT-SLU's closest contextual competitor.
  • Baselines: Stack-Prop (single intent only), Joint MID-SF, AGIF, ECA, plain BERT and BERT+SA. The authors also report intent-only training ("IO"), where CaBERT-SLU still leads (e.g., SGD IO accuracy 98.24).
  • Availability: Code is released at github.com/waynewu6250/CaBERT-SLU.

Interesting ideas

  • I1. Each utterance gets a set of acts via independent sigmoid outputs instead of one forced intent label, so a turn can confirm and ask a new question at the same time.
  • I2. Context is causal: a masked encoder lets each turn see only the turns before it, which matches how a listener actually receives a conversation.
  • I3. The authors observe regular act transitions, such as Select tending to follow Offer and reusing one of Offer's slot values. This suggests acts form a sequence grammar with argument sharing.
  • I4. Slot values recur across turns in shortened forms, for example "2g Japanese" for "2g Japanese Brasserie", so filling them depends on linking mentions across the dialog.
  • I5. Evaluation reports both exact-match accuracy over the whole act set and per-label F1, and compares joint training with act-only training.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): The speech-act layer in prag could allow several acts per sentence (e.g., affirm plus request), each with its own evidence, instead of picking one winner.
  • I2 (frames): Speech-act and implicature decisions could be conditioned only on earlier sentences, which keeps the reasoning honest and explainable as "this act follows that prior act".
  • I3 (frames): An explicit, inspectable table of act-to-act transitions could check or rerank prag's act guesses, and shared arguments could be linked to the earlier frame.
  • I4 (world): Entity merging in the text world could treat a truncated name in a later turn as the same entity introduced earlier, with the alias match recorded as an explanation.
  • I5 (learn): Multi-label outputs could be scored with both strict set match and per-label F1, with single-task runs as an ablation control, to see whether joint learning really helps.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgia Institute of TechnologyAtlanta · United States69 works · 22,788 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 401 worksSpeech act theory · 160 works

Cite

Cite as Ting-Wei Wu, Ruolin Su and Biing-Hwang Juang. 2021. A Context-Aware Hierarchical BERT Fusion Network for Multi-turn Dialog Act Detection. arXiv:2109.01267.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2109.01267 · arXiv 2109.01267

@misc{wu2021context,
  title = {{A Context-Aware Hierarchical BERT Fusion Network for Multi-turn Dialog Act Detection}},
  author = {Ting-Wei Wu, Ruolin Su and Biing-Hwang Juang},
  year = {2021},
  doi = {10.48550/arxiv.2109.01267},
  eprint = {2109.01267},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2109.01267}
}

References

25 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Z. Zhang, R. Takanobu, Q. Zhu, M. Huang, and X. Zhu, “Recent advances and challenges in task-oriented dialog system,” 2020. find
  2. H. Weld, X. Huang, S. Long, J. Poon, and S. C. Han, “A survey of joint intent detection and slot-filling models in natural language understanding,” 2021. ↗
  3. L. Abbeduto, “Linguistic communication and speech acts. kent bach robert m. harnish. cambridge: M.i.t. press, 1979, pp. xvii 327.” Applied Psycholinguistics, vol. 4, no. 4, p. 397–407, 1983. find
  4. X. Li, S. Panda, J. Liu, and J. Gao, “Microsoft dialogue challenge: Building end-to-end task-completion dialogue systems,” arXiv preprint arXiv:1807.11125, 2018. ↗
  5. T. Liu, X. DING, Y. QIAN, and Y. CHEN, “Identification method of user’s travel consumption intention in chatting robot,” SCIEN-TIA SINICA Informationis, vol. 47, p. 997, 08 2017. find
  6. C. Xia, C. Zhang, X. Yan, Y. Chang, and P. Yu, “Zero-shot user intent detection via capsule neural networks,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 3090–3099. [Online]. Available: https://www.aclweb.org/anthology/D18-1348 ↗
  7. C.-W. Goo, G. Gao, Y.-K. Hsu, C.-L. Huo, T.-C. Chen, K.-W. Hsu, and Y.-N. Chen, “Slot-gated modeling for joint slot filling and intent prediction,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). New Orleans, Louisiana: Association for Computational Linguistics, Jun. 2018, pp. 753–757. [Online]. Available: https://www.aclweb.org/anthology/N18-2118 ↗
  8. C. Li, L. Li, and J. Qi, “A self-attentive model with gate mechanism for spoken language understanding,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 3824–3833. [Online]. Available: https://www.aclweb.org/anthology/D18-1417 ↗
  9. H. E, P. Niu, Z. Chen, and M. Song, “A novel bi-directional interrelated model for joint intent detection and slot filling,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 5467–5471. [Online]. Available: https://www.aclweb.org/anthology/P19-1544 ↗
  10. Y. Liu, F. Meng, J. Zhang, J. Zhou, Y. Chen, and J. Xu, “Cm-net: A novel collaborative memory network for spoken language understanding,” 2019. find
  11. H. Purohit, G. Dong, V. Shalin, K. Thirunarayan, and A. Sheth, “Intent classification of short-text on social media,” in 2015 IEEE International Conference on Smart City/SocialCom/SustainCom (SmartCity), 2015, pp. 222–228. find
  12. L. Qin, W. Che, Y. Li, H. Wen, and T. Liu, “A stack-propagation framework with token-level intent detection for spoken language understanding,” 2019. ↗
  13. L. Qin, X. Xu, W. Che, and T. Liu, “Agif: An adaptive graph-interactive framework for joint multiple intent detection and slot filling,” 2020. find
  14. R. Gangadharaiah and Balakrishnan, Joint multiple intent detection and slot labeling for goal-oriented dialog. Proc. of NAACL, 2019. find
  15. A. Rastogi, X. Zang, S. Sunkara, R. Gupta, and P. Khaitan, “Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset,” arXiv preprint arXiv:1909.05855, 2019. ↗
  16. C. Xia, C. Xiong, P. Yu, and R. Socher, “Composed variational natural language generation for few-shot intents,” 2020. find
  17. Y. Shan, Z. Li, J. Zhang, F. Meng, Y. Feng, C. Niu, and J. Zhou, “A contextual hierarchical attention network with adaptive objective for dialogue state tracking,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, Jul. 2020, pp. 6322–6333. [Online]. Available: https://www.aclweb.org/anthology/2020.acl-main.563 ↗
  18. T.-H. Wen, D. Vandyke, N. Mrkšić, M. Gašić, L. M. Rojas-Barahona, P.-H. Su, S. Ultes, and S. Young, “A network-based end-to-end trainable task-oriented dialogue system,” in Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. Valencia, Spain: Association for Computational Linguistics, Apr. 2017, pp. 438–449. [Online]. Available: https://www.aclweb.org/anthology/E17-1042 ↗
  19. A. Bordes, Y.-L. Boureau, and J. Weston, “Learning end-to-end goal-oriented dialog,” 2017. ↗
  20. N. Bertomeu, H. Uszkoreit, A. Frank, H.-U. Krieger, and B. Jörg, “Contextual phenomena and thematic relations in database QA dialogues: results from a Wizard-of-Oz experiment,” in Proceedings of the Interactive Question Answering Workshop at HLT-NAACL 2006. New York, NY, USA: Association for Computational Linguistics, Jun. 2006, pp. 1–8. [Online]. Available: https://www.aclweb.org/anthology/W06-3001 ↗
  21. Y. Shi, K. Yao, H. Chen, Y.-C. Pan, M.-Y. Hwang, and B. Peng, “Contextual spoken language understanding using recurrent neural networks,” April 2015. find
  22. A. Gupta, P. Zhang, G. Lalwani, and M. Diab, “Casa-nlu: Context-aware self-attentive natural language understanding for task-oriented chatbots,” 2019. ↗
  23. V. Zhong, C. Xiong, and R. Socher, “Global-locally self-attentive dialogue state tracker,” 2018. find
  24. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2019.
  25. Z. Lin, M. Feng, C. N. dos Santos, M. Yu, B. Xiang, B. Zhou, and Y. Bengio, “A structured self-attentive sentence embedding,” 2017. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2021-context-aware-hierarchical-bert-fusion/