Elamur Research⌕
☰
home / papers / 2016 / Transfer of Corpus-Specific Dialogue Act Annotation to ISO Standard: Is it worth it?

Transfer of Corpus-Specific Dialogue Act Annotation to ISO Standard: Is it worth it?

Summary

Many spoken dialogue corpora use their own adaptations of schemes such as DAMSL or DIT++. Their dialogue act (DA) labels are therefore incompatible across corpora, which makes the corpora hard to reuse. The authors test whether converting such legacy labels to the ISO standard DiAML (ISO 24617-2) pays off. They semi-automatically re-annotated the DA-labelled part of the Italian LUNA Human-Human corpus (help-desk dialogues). A one-to-many mapping from LUNA tags to DiAML proposed labels, and an annotator re-segmented turns into functional segments and resolved ambiguous labels. They also annotated a small out-of-domain set of call-center dialogues with DiAML. A linear SVM on bag-of-words features was trained with each scheme and evaluated in three settings: in-domain, cross-domain, and data aggregation (pooling in-domain and out-of-domain training data). In-domain results were roughly equal for both schemes. The ISO labels transferred noticeably better to the new domain and benefited from pooled data. The authors conclude that migrating legacy annotations to the standard is worthwhile, since it makes corpora interoperable without losing classification accuracy.

Key points

  • Data: LUNA has 572 dialogues, of which 50 carry DA labels (3,268 DAs, split into 40 training and 10 test dialogues). The out-of-domain set has 10 energy/utilities call-center dialogues (1,472 DAs). The legacy scheme has 15 tags; DiAML has 56 communicative functions across 9 dimensions.
  • Agreement: On 15 dialogues, the agreement between the legacy annotator and the ISO annotator was F1 = 0.68, computed after Levenshtein-based alignment of the segments.
  • Classifier: Weka SMO with a linear kernel and default parameters, using bag-of-words features of the DA span only. The authors treat this as a baseline.
  • Dimension level (micro F1): Legacy scored 0.72 in-domain, 0.60 cross-domain and 0.72 with aggregation. ISO scored 0.72, 0.67 and 0.74 respectively.
  • Communicative function level (micro F1): Legacy with 16 classes scored 0.44 in-domain and 0.30 cross-domain. ISO with 41 classes scored 0.40 and 0.37. ISO mapped back to the 16 legacy classes scored 0.47 and 0.45. Legacy models gained more from aggregation (0.49) but lost much more accuracy cross-domain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Speech act theory

Cited by

8 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 6 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{chowdhury2016transfer,
  title = {{Transfer of Corpus-Specific Dialogue Act Annotation to ISO Standard: Is it worth it?}},
  author = {Chowdhury, Shammur Absar and Stepanov, Evgeny and Riccardi, Giuseppe},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  doi = {10.63317/342s3teowe5e},
  url = {https://doi.org/10.63317/342s3teowe5e}
}

References

14 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bunt, H., Alexandersson, J., Carletta, J., Choe, J.-W., Fang, A. C., Hasida, K., Lee, K., Petukhova, V., Popescu-Belis, A., Romary, L., et al. (2010). Towards an ISO standard for dialogue act annotation. In Seventh conference on International Language Resources and Evaluation (LREC’10). find our card
  2. Bunt, H., Alexandersson, J., Choe, J.-W., Fang, A. C., Hasida, K., Petukhova, V., Popescu-Belis, A., and Traum, D. R. (2012). ISO 24617-2: A semantically-based standard for dialogue annotation. In LREC, pages 430–437. find our card
  3. Bunt, H., Fang, A. C., Liu, X., Cao, J., and Petukhova, V. (2013). Issues in the addition of ISO standard annotations to the switchboard corpus. In Workshop on Interoperable Semantic Annotation. find
  4. Bunt, H. (2005). A framework for dialogue act specification. In In Proceedings of SIGSEM WG on Representation of Multimodal Semantic Information. Chowdhury, S. A., Calvo, M., Ghosh, A., Stepanov, E. A., Bayer, A. O., Riccardi, G., Garcı́a, F., and Sanchis, E. (2015). Selection and aggregation techni… find
  5. Core, M. G. and Allen, J. F. (1997). Coding dialogs with the damsl annotation scheme. In Proceedings of AAAI Fall Symposium on Communicative Action in Humans and Machines. find
  6. Dinarelli, M., Quarteroni, S., Tonelli, S., Moschitti, A., and Riccardi, G. (2009). Annotating spoken dialogs: from speech segments to dialog acts and frame semantics. In Proceedings of EACL Workshop on the Semantic Representation of Spoken Language, Athens, Greece. find our card
  7. Fang, A. C., Cao, J., Bunt, H., and Liu, X. (2012). The annotation of the Switchboard Corpus with the new ISO standard for dialogue act analysis. In Workshop on Interoperable Semantic Annotation. find
  8. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reute-mann, P., and Witten, I. H. (2009). The weka data mining software: an update. ACM SIGKDD Explorations Newsletter, 11(1):10–18. find
  9. Joty, S., Carenini, G., and Lin, C.-Y. (2011). Unsupervised modeling of dialog acts in asynchronous conversations. In IJCAI Proceedings-International Joint Conference on Artificial Intelligence. find
  10. Petukhova, V., Malchanau, A., and Bunt, H. (2014). Interoperability of dialogue corpora through ISO 24617-2-based querying. In LREC. find
  11. Quarteroni, S., Riccardi, G., Varges, S., and Bisazza, A. (2008). An open-domain dialog act taxonomy. Technical Report DISI-08-032, University of Trento, August. find
  12. Tavafi, M., Mehdad, Y., Joty, S., Carenini, G., and Ng, R. (2013). Dialogue act recognition in synchronous and asynchronous conversations. In Proceedings of the SIG-DIAL 2013 Conference, pages 117–121. find our card
  13. Traum, D. (1996). Conversational agency: The trains-93 dialogue manager. In Proceedings of Twente Workshop on Language Technology, TWLT-II. 8. find
  14. Language Resource References Dinarelli, M., Quarteroni, S., Tonelli, S., Moschitti, A., and Riccardi, G. (2009). Annotating spoken dialogs: from speech segments to dialog acts and frame semantics. In Proceedings of EACL Workshop on the Semantic Representation of Spoken Language. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chowdhury-2016-transfer-corpus-specific-dialogue-act/