Elamur Research⌕
☰
home / papers / 2016 / Creating Annotated Dialogue Resources: Cross-domain Dialogue Act Classification

Creating Annotated Dialogue Resources: Cross-domain Dialogue Act Classification

Summary

Dialogue corpora labelled with dialogue acts are scarce and hard to combine. They differ in collection setting, modality, domain, number of participants and tagging scheme. Expert annotation is also slow, at roughly 19 hours of work per hour of dialogue. The authors test whether classifiers trained on existing resources can label new dialogue data automatically.

They use DiAML queries and a mapping onto ISO 24617-2 concepts to extract material from AMI, HCRC MapTask and Switchboard. The labels are reduced to four communicative classes (Inform, Question, Directives, Commissives) plus a negative Other class. The study then proceeds in three steps:

  • it evaluates which feature types support cross-domain prediction;
  • it compares single-corpus and combined-corpus training, testing on held-out corpora and on the unseen METALOGUE data;
  • it outlines model adaptation and late-fusion (meta-classification) designs for merging classifier decisions.

The surviving results suggest that models trained on one corpus transfer reasonably well to new domains. MapTask is the weakest link. The work offers a practical path toward ISO-compatible annotated dialogue resources with much lower annotation and training costs.

Key points

  • Instance counts: AMI 19,918; HCRC MapTask 2,920; Switchboard 12,768. The METALOGUE corpus (1,520 instances) is used only as unseen test data.
  • Class distributions vary widely: Other covers 60% of AMI and 44% of SWBD, while METALOGUE is nearly balanced at about 20% per class.
  • Cross-corpus scores (metric not named in the extracted text):
    • AMI→SWBD 0.86, AMI→METALOGUE 0.84, SWBD→METALOGUE 0.84.
    • Transfer involving MapTask is lowest, ranging from 0.53 to 0.66.
  • Earlier cross-corpus work with cue phrases reached 70.8% (Webb and Liu, 2008) and 72.34% (Webb et al., 2010). Both studies used only DAMSL-family schemes.
  • Missing details: the feature-importance and late-fusion results were in the omitted middle of the source text, so they are not covered here.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Speech act theory

Cited by

8 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{amanova2016creating,
  title = {{Creating Annotated Dialogue Resources: Cross-domain Dialogue Act Classification}},
  author = {Amanova, Dilafruz and Petukhova, Volha and Klakow, Dietrich},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  doi = {10.63317/5er6bpoj2m9k},
  url = {https://doi.org/10.63317/5er6bpoj2m9k}
}

References

34 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Blache, P., Bertrand, R.m and Ferré, G. 2009. Creating and Exploiting Multimodal Annotated Corpora: The ToMA Project. In Multimodal Corpora, Springer-Verlag. find
  2. Van den Bosch, A. 1997 Learning to pronounce written words: A study in inductive language learning. PhD thesis, Maastricht University, The Netherlands Boser, B., Guyon, I., and Vapnik, V. 1992 A Training Algorithm for Optimal Margin Classifiers. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory,Pittsburgh. ↗
  3. Bunt, H., Fang, A., Cao, J., Liu, X., and Petukhova, V. 2013 Issues in the addition of ISO standard annotations to the Switchboard corpus. In Proceedings 9th Joint ISO-ACLWorkshop on Interoperable Semantic Annotation (ISA-9), Potsdam. find
  4. Carletta, J. C., Isard, A., Isard, S., Kowtko, J., Doherty-Sneddon, G. and Anderson, A. 1996 HCRC Dialogue Structure Coding Manual. Human Communication Research Centre HCRC TR-82, University of Edinburgh. find
  5. Di Eugenio, B. et al. 1998 The COCONUT project: dialogue annotation manual. ISP Technical Report 98-1. find
  6. Dielmann, A. and Renals, S. 2008 Recognition of Dialogue Acts in Multiparty Meetings Using a Switching DBN. In Audio, Speech, and Language Processing, IEEE Transactions, pp. 1303-1314 ↗
  7. Dietterich, T. 2002 Machine learning for sequential data: a review. Proceedings of the Joint IAPR International Workshop on Structural, Syntactic, and Statistical Pattern Recognition, pp. 15-30 find
  8. Dhillon, R. et al. 2004 Meeting recorder project: dialogue labelling guide. ICSI Technical Report TR-04-002 http://www.anvil-software.org/https://tla.mpi.nl/tools/tla-tools/elan/ ↗
  9. Fang, A., Cao, J., Bunt, H., and Liu, X. 2012 The annotation of the Switchboard corpus with the new ISO standard for Dialogue Act Analysis. In Proceedings 8th Joint ISO-ACL Workshop on Interoperable Semantic Annotation (ISA-8), Pisa. find
  10. Geertzen, J., Petukhova, V., and Bunt, H. 2007. A Multidimensional Approach to Utterance Segmentation and Dialogue Act Classification. Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue, Antwerp, pages 140–149. find our card
  11. Henderson, M., Thomson, B., and Young, S. 2014. Word-Based Dialog State Tracking with Recurrent Neural Networks. Proceedings of the SIGdial, Philadelphia. find our card
  12. Jurafsky, D., Schriberg, E., and Biasca, D. 1997 Switchboard SWBD-DAMSL Shallow-Discourse-Function Annotation: Coders Manual, University of Colorado. find
  13. Jurafsky, D. et al. 1998 Lexical, prosodic, and syntactic cues for dialogue acts In Proceedings of the Discourse Relations and Discourse Markers Conference, Somerset, New Jersey, USA Fernandez, R. and Picard, R. 2002 Dialog act classification from prosodic features using Support Vector Machines In Proceedings of Speech Prosody 2002, Aix-en-Provence, France ISO. 2012 Language resource management – Semantic annotation framework – Part 2: Dialogue acts. ISO 24617-2, Geneva, ISO, Geneva. find
  14. Keizer, S. 2003 Reasoning under uncertainty in natural language dialogue using Bayesian Networks. PhD Thesis,Twente University Press,The Netherlands Lendvai, P. et al. 2004 Memory-based Robust Interpretation of Recognised Speech. In Proceedings of the SPECOM ’04, St. Petersburgh, Russia, pp. 415-422 find
  15. Lin, W.-H., and Hauptmann, A. 2002 Meta-classification: Combining multimodal classifiers. Mining Multimedia and Complex Data. Springer Berlin Heidelberg, pp. 217-231. find
  16. Morvant, E., Habrard, A., and Ayache, S. 2014 Majority vote of diverse classifiers for late fusion. Structural, Syntactic, and Statistical Pattern Recognition. Springer Berlin Heidelberg, pp. 153-162. ↗
  17. Petukhova, V. and Bunt, H. 2007 A multidimensional approach to multimodal dialogue act annotation. In Proceedings of the IWCS 2007, pp. 142-153 find
  18. Petukhova, V. 2011 Multidimensional Dialogue Modelling. PhD dissertation. find
  19. Tilburg University, The Netherlands Petukhova, V., Malchanau, A., and Bunt, H. 2014 Interoperability of dialogue corpora through ISO 24617-2-based querying. In Proceedings of the LREC 2014, Iceland. find
  20. Petukhova, V., et al. 2014 The DBOX corpus collection of spoken human-human and human-machine dialogues. In Proceedings of the LREC 2014, Iceland. find
  21. Petukhova, V., et al. 2015 Modelling argumentation in parliamentary debates. In Proceedings of the 15th Workshop on Computational Models of Natural Argument, Principles and Practice of Multi-Agent Systmes Conference (PRIMA 2015), Bertinoro, Italy Petukhova, V., et al. 2016 Modelling multi-issue bargaining dialogues: data collection, annotation design and corpus. In Proceedings of the LREC 2016, Portorož, Slovenia Popescu-Belis, A. 2003 Dialogue act tagsets for meeting understanding: an abstraction based on the DAMSL, Switchboard and ICSI-MR tagsets. Project Report IM2.MDM-09, v1.2, December 2004 find
  22. Punyakanok, V., and Roth, D. 2001 The Use of Classifiers in Sequential Inference. NIPS, pp.995–1001 ↗
  23. Reithinger, N.and Klesen, M. 1997 Dialogue act classification using language models. In Proceedings of EuroSpeech-97, pp. 2235-2238 find
  24. Samuel, K. et al. 1998 Dialogue act tagging with transformation-based learning. In Proceedings of the ACL and CoLing, Montreal, pp. 1150 - 1156 ↗ our card
  25. Shriberg, E. et al. 1998 Can prosody aid the automatic classification of dialog acts in conversational speech? In Language and Speech (Special Issue on Prosody and Conversation), 41(3-4):439-487 ↗
  26. Stolcke, A. et al. 2000 Dialogue act modeling for automatic tagging and recognition of conversational speech. In Computational Linguistics, 26(3), pp. 339-373 ↗ our card
  27. Surendran, D., and Levow, G. 2006 Dialog act tagging with Support Vector Machines and Hidden Markov Models. In Proceedings of Interspeech/ICSLP Vapnik, V.N. 1995 The Nature of Statistical Learning Theory. Springer-Verlag New York, Inc. find
  28. Webb, N. et al. 2005 Error Analysis of Dialogue Act Classification. In Proceedings of the TSD Conference, Karlovy Vary, Czech Republic, pp. 451-458 find
  29. Webb, N., and Liu, T. 2008 Investigating the Portability of Corpus-derived Cue Phrases for Dialogue Act Classification. In Proceedings of the COLING ’08, Manchester, United Kingdom Webb, N., and Ferguson, M. 2010 Automatic Extraction of Cue Phrases for Cross-Corpus Dialogue Act Classification. In Proceedings of the COLING ’10, pp. 1310-1317 find
  30. Yu, H.-F., Huang, F.-L., Lin, C.-J. 2011 Dual coordinate descent methods for logistic regression and maximum entropy models. Machine Learning 85(1-2):41-75. ↗
  31. Zhu, J., Zou, H., Rosset, S., and Hastie, T. 2009 Multi-class AdaBoost. 8. find
  32. Language Resource References AMI-Consortium. 2005 AMI Meeting Corpus. available at http://groups.inf.ed.ac.uk/ami/download/ ↗
  33. Anderson, A., et al. 1991 The HCRC Map Task Corpus. Language and Speech, 34, pp. 351-366. Available at http://groups.inf.ed.ac.uk/maptask/ ↗
  34. John, G., and Holliman, E. 1993 Switchboard-1 Release 2 LDC97S62. Web Download. Philadelphia: Linguistic Data Consortium. find

↑ back to top