Elamur Research⌕
☰
home / papers / 2022 / A Multi-Dimensional, Cross-Domain and Hierarchy-Aware Neural Architecture for ISO-Standard Dialogue Act Tagging

A Multi-Dimensional, Cross-Domain and Hierarchy-Aware Neural Architecture for ISO-Standard Dialogue Act Tagging

Paper · 2022 · Stefano Mezza, Wayne Wobcke and Alan Blair

Summary

ISO 24617-2 is a domain-independent, hierarchical and multi-dimensional standard for Dialogue Act tagging. Practitioners rarely adopt it, mainly because no training corpus of adequate size is annotated with it. The authors address this from two directions.

  • Data: they build a General Dialogue Corpus (GDC). It maps several existing corpora (SWDA, MRDA, DailyDialog, Mastodon, and Internet Argument Corpus v2.0 debates) onto a subset of the ISO taxonomy.
  • Model: they design DASHNet, a multi-label neural tagger. It combines four signals:
    • the utterance itself,
    • POS and dependency tags,
    • context from earlier turns,
    • a Tree-LSTM prior that encodes the label hierarchy.

The model is tested out-of-domain on the whole DialogBank, which is never used for training. There it reaches state-of-the-art results (83.7 Micro-F1, 57.1 Macro-F1). Ablations show three things:

  • Context helps most overall.
  • Syntax mainly lifts rare tags.
  • The hierarchy prior matters chiefly for cross-domain generalisation.

The work matters because it shows that heterogeneous legacy annotations, once mapped onto one standard, can train a tagger that transfers to unseen dialogue. The code and the mapped GDC are released.

Key points

  • The taxonomy subset spans the Task, Social and Feedback dimensions. One utterance can carry several tags, even within a single dimension. Tags such as Task:Answer or Task:Agreement depend on the preceding turns.

  • On DialogBank, the full model scores 83.7 Micro-F1 and 57.1 Macro-F1. On the GDC test set it scores 90.6 Micro-F1 and 76.9 Macro-F1.

  • The ablation results on DialogBank (Micro / Macro-F1) are:

    Removed componentMicro-F1Macro-F1
    Contextual features80.050.4
    Tree-LSTM prior81.851.2
    POS tags83.554.3
    Dependency tags83.652.1

    Removing POS or dependency tags barely changes Micro-F1 but clearly lowers Macro-F1.

  • Training-set ablations show that every corpus contributes:

    • Dropping SWDA lowers DialogBank Micro-F1 to 77.3.
    • Dropping the small Mastodon corpus lowers it to 79.6. The likely reason is that Mastodon's scheme was designed with ISO in mind, so its mapping is less noisy.

  • Corpora that barely move Micro-F1 still help specific labels:

    • With DailyDialog in training, Directive accuracy rises from 8% to 42% and Commissive from 10% to 25%.
    • With IAC, Agree accuracy rises from 25% to 35%.

Interesting ideas

  • I1. Feeding the classifier a structural prior over the label tree makes rare, fine-grained labels benefit from their coarser ancestors, and this gain appears mostly on out-of-domain data.
  • I2. Explicit POS and dependency tags add little to aggregate accuracy but noticeably help low-frequency classes, so Macro-F1 is the metric that reveals their value.
  • I3. Several heterogeneous corpora are pooled into one training set by mapping their tag schemes onto a shared standard. Corpora whose schemes already resemble the target help more per example, because their mapping adds less noise.
  • I4. Ablating each training corpus and then inspecting per-label accuracy shows which source teaches which phenomenon, for example DailyDialog for directives and IAC for agreement.
  • I5. Holding out an entire benchmark corpus as an out-of-domain test, instead of using a split of the training data, gives a stricter measure of generalisation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Speech-act and frame labels in prag could be scored with an explicit hierarchy prior, so that a rare act falls back to its parent instead of being confused with an unrelated label.
  • I2 (learn): Evaluations should report per-label and Macro-F1 next to Micro-F1, so that gains from tree features on rare phenomena are not hidden by frequent classes.
  • I3 (tree): When converting treebanks between UD dialects, conversion noise could be estimated per source, and closer-matching sources could be given more weight.
  • I4 (learn): Silver-data sources could be audited per label in the same way, to decide which witness corpora to keep or expand for specific constructions.
  • I5 (answer): Readers could be tested on a whole unseen collection of fables or tests rather than on random splits, to check that answers do not rely on domain quirks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 Dialogue systems · 401 works Speech act theory · 160 works

Cite

Cite as Stefano Mezza, Wayne Wobcke and Alan Blair. 2022. A Multi-Dimensional, Cross-Domain and Hierarchy-Aware Neural Architecture for ISO-Standard Dialogue Act Tagging. Proceedings of the 29th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mezza2022multi,
  title = {{A Multi-Dimensional, Cross-Domain and Hierarchy-Aware Neural Architecture for ISO-Standard Dialogue Act Tagging}},
  author = {Stefano Mezza, Wayne Wobcke and Alan Blair},
  booktitle = {Proceedings of the 29th International Conference on Computational Linguistics},
  year = {2022},
  doi = {10.48448/73ph-x489},
  url = {https://doi.org/10.48448/73ph-x489}
}

References

44 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Philippe Blache, Massina Abderrahmane, Stéphane Rauzy, Magalie Ochs, and Houda Oufaida. 2020. Two-level Classification for Dialogue Act Recognition in Task-Oriented Dialogues. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4915–4925. find
  2. Harry Bunt. 2005. A Framework for Dialogue Act Specification. WG: Proceedings of the ACL SIGSEM Working Group on Representation of Multimodal Semantic Information. find
  3. Harry Bunt. 2006. Dimensions in Dialogue Act Annotation. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), pages 919–924. find
  4. Harry Bunt. 2009. The DIT++ Taxonomy for Functional Dialogue Markup. In AAMAS 2009 Workshop, Towards a Standard Markup Language for Embodied Dialogue Acts, pages 13–24. find
  5. Harry Bunt, Jan Alexandersson, Jean Carletta, Jae-Woong Choe, Alex Chengyu Fang, Koiti Hasida, Kiyong Lee, Volha Petukhova, Andrei Popescu-Belis, Laurent Romary, et al. 2010. Towards an ISO Standard for Dialogue Act Annotation. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), pages 2548– 2555. our card
  6. Rob Abbott, Brian Ecker, Pranav Anand, and Marilyn Walker. 2016. Internet Argument Corpus 2.0: An SQL schema for Dialogic Social Media and the Corpora to go with it. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4445–4452. find
  7. Harry Bunt, Jan Alexandersson, Jae-Woong Choe, Alex Chengyu Fang, Koiti Hasida, Volha Petukhova, Andrei Popescu-Belis, and David Traum. 2012. ISO 24617-2: A Semantically-Based Standard for Dialogue Annotation. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 430–437. our card
  8. James Allen and Mark Core. 1997. Draft of DAMSL: Dialog Act Markup in Several Layers, https://www.cs.rochester.edu/research/cisd/ resources/damsl/RevisedManual/, accessed on May 15, 2022. ↗
  9. Harry Bunt, Volha Petukhova, Andrei Malchanau, Kars Wijnhoven, and Alex Fang. 2016. The DialogBank. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3151–3158. our card
  10. Anne H Anderson, Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jacqueline Kowtko, Jan McAllister, Jim Miller, et al. 1991. The HCRC Map Task Corpus. Language and Speech, 34(4):351–366. ↗
  11. Harry Bunt, Volha Petukhova, David Traum, and Jan Alexandersson. 2017. Dialogue Act Annotation with the ISO 24617-2 Standard. In Deborah A. Dahl, editor, Multimodal Interaction with W3C Standards, pages 109–135. Springer, Cham. ↗
  12. Tatiana Anikina and Ivana Kruijff-Korbayova. 2019. Dialogue Act Classification in Team Communication for Robot Assisted Disaster Response. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 399–410. find
  13. Christophe Cerisara, Somayeh Jafaritazehjani, Adedayo Oluokun, and Hoa T Le. 2018. Multi-task Dialog Act and Sentiment Recognition on Mastodon. In Proceedings of the 27th International Conference on Computational Linguistics,, pages 745–754. ↗ our card
  14. Alessandra Cervone, Evgeny Stepanov, and Giuseppe Riccardi. 2018. Coherence Models for Dialogue. In Proceedings of Interspeech 2018, pages 1011–1015. find
  15. Role Labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2401–2411. find
  16. Zheqian Chen, Rongqin Yang, Zhou Zhao, Deng Cai, and Xiaofei He. 2018. Dialogue Act Recognition via CRF-Attentive Structured Network. In Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 225–234. our card
  17. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pre-Training Approach. arXiv preprint arXiv:1907.11692. ↗
  18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805. ↗ our card
  19. Stefano Mezza, Alessandra Cervone, Evgeny Stepanov, Giuliano Tortoreto, and Giuseppe Riccardi. 2018. ISO-Standard Domain-Independent Dialogue Act Tagging for Conversational Agents. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3539–3551. ↗ our card
  20. Alex Fang, J Cao, H Bunt, and Xiaoyue Liu. 2012. Applicability Verification of a New ISO Standard for Dialogue Act Annotation with the Switchboard Corpus. In Proceedings of the EACL 2012 Workshop on Innovative Hybrid Approaches to the Processing of Textual Data. find
  21. Emer Gilmartin, Christian Saam, Brendan Spillane, Maria O’Reilly, Ketong Su, Arturo Calvo Devesa, Loredana Cerrato, Killian Levacher, Nick Campbell, and Vincent Wade. 2018. The ADELE Corpus of Dyadic Social Text Conversations: Dialog Act Annotation with ISO 24617-2. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 4016– 4022. our card
  22. Susanne Jekat, Alexandra Klein, Elisabeth Maier, Ilona Maleck, Marion Mast, and J Joachim Quantz. 1995. Dialogue Acts in VERBMOBIL. Technical report, Universität des Saarlandes, Saarbrücken. ↗
  23. Daniel Jurafsky and Elizabeth Shriberg. 1997. Switchboard SWBD-DAMSL Shallow-Discourse-Function Annotation Coders Manual, Draft 13, University of Colorado at Boulder, SRI International. find
  24. Harshit Kumar, Arvind Agarwal, Riddhiman Dasgupta, and Sachindra Joshi. 2018. Dialogue Act Sequence Labeling using Hierarchical encoder with CRF. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pages 3440–3447. ↗ our card
  25. Shachi Paul, Rahul Goel, and Dilek Hakkani-Tür. 2019. Towards Universal Dialogue Act Tagging for Task-Oriented Dialogues. arXiv preprint arXiv:1907.03020. ↗ our card
  26. Volha Petukhova, Martin Gropp, Dietrich Klakow, Anna Schmidt, Gregor Eigner, Mario Topf, Stefan Srb, Petr Motlicek, Blaise Potard, John Dines, et al. 2014. The DBOX Corpus Collection of Spoken Human-Human and Human-Machine Dialogues. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 252–258. find
  27. Silvia Quarteroni, Alexei V Ivanov, and Giuseppe Riccardi. 2011. Simultaneous Dialog Act Segmentation and Classification from Human-Human Spoken Conversations. In Proceedings of the 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5596–5599. find
  28. Vipul Raheja and Joel Tetreault. 2019. Dialogue Act Classification with Context-Aware Self-Attention. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3727–3733. ↗ our card
  29. Eugénio Ribeiro, Ricardo Ribeiro, and David Martins de Matos. 2019. Hierarchical Multi-Label Dialog Act Recognition on Spanish Data. arXiv preprint arXiv:1907.12316. ↗
  30. Piroska Lendvai, Antal van den Bosch, and Emiel Krahmer. 2003. Machine Learning for Shallow Interpretation of User Utterances in Spoken Dialogue Systems. In Proceedings of the 2003 EACL Workshop on Dialogue Systems: Interaction, Adaptation and Styles of Management, pages 69–78. find
  31. Eugénio Ribeiro, Ricardo Ribeiro, and David Martins de Matos. 2022. Automatic Recognition of the General-Purpose Communicative Functions defined by the ISO 24617-2 Standard for Dialog Act Annotation. Journal of Artificial Intelligence Research, 73:397–436. ↗ our card
  32. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986–995. ↗ our card
  33. Eugénio Ribeiro, Ricardo Ribeiro, and David Martins de Matos. 2020. Mapping the Dialog Act Annotations of the LEGO Corpus into ISO 24617-2 Communicative Functions. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 531– 539. our card
  34. Zuchao Li, Shexia He, Jiaxun Cai, Zhuosheng Zhang, Hai Zhao, Gongshen Liu, Linlin Li, and Luo Si. 2018. A Unified Syntax-aware Framework for Semantic John R Searle. 1965. What is a Speech Act? In Maurice Black, editor, Philosophy in America, pages 221–239. Allen and Unwin, London. find
  35. Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The ICSI Meeting Recorder Dialog Act (MRDA) Corpus. Technical report, International Computer Science Institute, Berkeley, CA. ↗ our card
  36. Claudia Soria and Vito Pirrelli. 2003. A Multi-Level Annotation Meta-Scheme for Dialogue Acts. In Linguistica Computazionale, vol. XVIII-XIX, pages 865– 900. find
  37. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech. Computational Linguistics, 26(3):339–373. ↗ our card
  38. Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks. arXiv preprint arXiv:1503.00075. ↗
  39. David R Traum. 2000. 20 Questions on Dialogue Act Taxonomies. Journal of Semantics, 17(1):7–30. ↗
  40. Marilyn Walker, Jean E Fox Tree, Pranav Anand, Rob Abbott, and Joseph King. 2012. A Corpus for Research on Deliberation and Debate. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 812– 817. our card
  41. Dong Wang, Ziran Li, Dongming Sheng, Hai-Tao Zheng, and Ying Shen. 2021. Balance the Labels: Hierarchical Label Structured Network for Dialogue Act Recognition. In Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN). find
  42. Dian Yu and Zhou Yu. 2019. MIDAS: A Dialog Act Annotation Scheme for Open Domain Human Machine Spoken Conversations. arXiv preprint arXiv:1908.10023. ↗
  43. Jie Zhou, Chunping Ma, Dingkun Long, Guangwei Xu, Ning Ding, Haoyu Zhang, Pengjun Xie, and Gongshen Liu. 2020. Hierarchy-Aware Global Model for Hierarchical Text Classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1106–1117. find
  44. Yucan Zhou, Qinghua Hu, Jie Liu, and Yuan Jia. 2014. Learning Conditional Random Field with Hierarchical Representations for Dialogue Act Recognition. In Proceedings of the Fifteenth Annual Conference of the International Speech Communication Association, pages 1920–1923. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mezza-2022-multi-dimensional-cross-domain-hierarchy/