Elamur Research⌕
☰
home / papers / 2022 / An Analysis of Dialogue Act Sequence Similarity Across Multiple Domains

An Analysis of Dialogue Act Sequence Similarity Across Multiple Domains

Paper · 2022 · Ayesha Enayet and Gita Sukthankar

Summary

Models that rely on dialogue act (DA) context are often trained on one corpus and then applied to another, and they can fail when the conversational style changes. This paper asks whether that risk can be estimated before training, by looking only at the sequences of DA labels. Using a single classifier, the authors label eight English corpora: the meeting corpus AMI (DialSum), the Switchboard set SwDA, the Diplomacy Betrayal game chats, the developer discussions from GitHub, SAMsum, the Teams conversations, Military (Army) and the Hate Speech collection. The classifier is based on the Universal Sentence Encoder (USE), trained on SwDA and uses the SwDA-DAMSL tagset. They then compare the corpora in three ways: DA n-gram frequencies, a Hamming-distance overlap score on short subsequences, and Doc2Vec embeddings of whole dialogues. The embeddings are measured with cosine similarity and with pairwise binary classifiers that try to tell two datasets apart. Pairs that are hard to separate tend to come from similar communication settings. Social media corpora cluster together, while SwDA and Teams stand apart from the rest. The practical message is that a source corpus should be chosen for its discourse-pattern compatibility with the target, not assumed to transfer.

Key points

  • The DA tagger is a USE-based utterance classifier trained on SwDA, with 72% test and 70% validation accuracy. It tags each utterance alone, without neighbouring context. The resulting DA sequences are released in the DAC-USE GitHub repository (DomainShift folder).
  • The analysis uses balanced samples of 50 random dialogues per corpus. Military is the exception, with only 22 dialogues.
  • The proposed similarity score is the share of zero-Hamming-distance subsequences of a fixed window (lengths 4 and 5) shared between two datasets. Higher overlap goes with lower accuracy of an SVM (RBF kernel) trained to discriminate the pair.
  • SAMsum and Hate Speech are the most confusable pair, with a cosine similarity of about 0.53 between their Doc2Vec embeddings. SwDA and Teams are the least similar to the other corpora.
  • For robustness tests, noisy copies were made by swapping or deleting 10% of the tags, giving nine variants per dialogue and then resampling to 140 sequences per dataset. Accuracy fell most for the formal-discussion corpora (AMI, GitHub), but the overall layout of the embeddings stayed similar.

Interesting ideas

  • I1. Domain distance can be measured on an abstract label layer (DA tags) instead of raw words, so it captures how conversations flow and ignores what topics they cover.
  • I2. The paper uses "can a classifier tell these two corpora apart?" as a similarity metric, so a confusable pair signals a compatible source and target for transfer.
  • I3. Exact-match overlap of short fixed-length label windows is a cheap and transparent similarity score that anyone can inspect window by window.
  • I4. Synthetic swap and deletion noise checks whether a similarity measure is stable or merely memorizing exact sequences.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Speech-act sequences produced by prag could be compared between genres (fables, school texts, dialogues) to warn when a pragmatic rule set tuned on one genre is applied to a structurally different one.
  • I2 (learn): Before reusing silver data from one domain, Mova could train a quick discriminator on parse- or frame-label sequences, and treat high separability as a sign to expect accuracy loss.
  • I3 (tree): The share of shared UD relation or POS n-gram windows between treebanks could help rank which treebank or dialect-converted source best matches a target text type for training the perceptron parser or gparse.
  • I4 (learn): Applying controlled 10% label corruption to evaluation sets could act as a negative control, showing whether Mova's domain-similarity or quality signals stay stable under slight noise.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Central FloridaOrlando · United States13 works · 39,673 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksDialogue systems · 624 worksSpeech act theory · 229 works

Cite

Cite as Ayesha Enayet and Gita Sukthankar. 2022. An Analysis of Dialogue Act Sequence Similarity Across Multiple Domains. Proceedings of the Thirteenth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{enayet2022analysis,
  title = {{An Analysis of Dialogue Act Sequence Similarity Across Multiple Domains}},
  author = {Ayesha Enayet and Gita Sukthankar},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.63317/3ssy4aomtfbv},
  url = {https://doi.org/10.63317/3ssy4aomtfbv}
}

References

32 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aberdeen, J. and Ferro, L. (2003). Dialogue patterns and misunderstandings. In ISCA Tutorial and Research Workshop on Error Handling in Spoken Dialogue Systems. find
  2. Agrawal, S., Joshi, A., Ross, J. C., Bhattacharyya, P., and Wabgaonkar, H. M. (2017). Are word embedding and dialogue act class-based features useful for coreference resolution in dialogue? In Proceedings of PACLING. find
  3. Ahmadvand, A., Choi, J. I., and Agichtein, E. (2019). Contextual dialogue act classification for open-domain conversational agents. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1273–1276.
  4. Allen, J. and Core, M. (1997). Draft of DAMSL: Dialog act markup in several layers. find
  5. Cer, D., Yang, Y., Kong, S.-y., Hua, N., Limtiaco, N., John, R. S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al. (2018). Universal sentence encoder. arXiv preprint arXiv:1803.11175. ↗
  6. Chen, L. and Di Eugenio, B. (2013). Multimodality and dialogue act classification in the robohelper project. In Proceedings of the SIGDIAL Conference, pages 183–192.
  7. Chen, X., Campero Durand, G., Zoun, R., Broneske, D., Li, Y., and Saake, G. (2019). The best of both worlds: combining hand-tuned and word-embedding-based similarity measures for entity resolution. BTW. find
  8. Elsayed, T., Lin, J., and Oard, D. W. (2008). Pairwise document similarity in large collections with mapreduce. In Proceedings of ACL: HLT, Short Papers, pages 265–268. find
  9. Ezen-Can, A., Grafsgaard, J. F., Lester, J. C., and Boyer, K. E. (2015). Classifying student dialogue acts with multimodal learning analytics. In Proceedings of the International Conference on Learning Analytics and Knowledge, pages 280–289. find
  10. Frummet, A., Elsweiler, D., and Ludwig, B. (2019). Detecting domain-specific information needs in conversational search dialogues. In Workshop on Natural Language for Artificial Intelligence at AIIA. find
  11. Gliwa, B., Mochol, I., Biesek, M., and Wawer, A. (2019). Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. arXiv preprint arXiv:1911.12237. ↗
  12. Goo, C.-W. and Chen, Y.-N. (2018). Abstractive dialogue summarization with sentence-gated modeling optimized by dialogue acts. In IEEE Spoken Language Technology Workshop (SLT), pages 735–742. find
  13. Grau, S., Sanchis, E., Castro, M. J., and Vilar, D. (2004). Dialogue act classification using a Bayesian approach. In Conference of Speech and Computer. find
  14. Kalia, A. K., Buchler, N., DeCostanza, A., and Singh, M. P. (2017). Computing team process measures from the structure and content of broadcast collaborative communications. IEEE Transactions on Computational Social Systems, 4(2):26–39. ↗
  15. Kulkarni, R., Hanna, K., and Stanely, J. (2020). NLP generalization for QA tasks. find
  16. Kumar, H., Agarwal, A., and Joshi, S. (2018). Dialogue-act-driven conversation model: An experimental study. In Proceedings of the International Conference on Computational Linguistics, pages 1246–1256. find
  17. Lavi, O., Rabinovich, E., Shlomov, S., Boaz, D., Ronen, I., and Anaby Tavor, A. (2021). We’ve had this conversation before: A novel approach to measuring dialog similarity. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1169–1177. Association for Computational Linguistics. find
  18. Le, Q. and Mikolov, T. (2014). Distributed representations of sentences and documents. In International Conference on Machine Learning, pages 1188–1196. PMLR. ↗
  19. Lee, C., Jung, S., Eun, J., Jeong, M., and Lee, G. G. (2006). A situation-based dialogue management using dialogue examples. In IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, volume 1, pages I–I. IEEE. find
  20. Li, R., Lin, C., Collinson, M., Li, X., and Chen, G. (2018). A dual-attention hierarchical recurrent neural network for dialogue act classification. arXiv preprint arXiv:1810.09154. ↗
  21. Litman, D., Paletz, S., Rahimi, Z., Allegretti, S., and Rice, C. (2016). The Teams corpus and entrainment in multi-party spoken dialogues. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1421–1431. find
  22. Liu, Q., Huang, H., Lut, J., Gao, Y., and Zhang, G. (2017). Enhanced word embedding similarity measures using fuzzy rules for query expansion. In 2017 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), pages 1–6. find
  23. Midgley, T. D. and MacNish, C. (2003). Automatic dialogue segmentation using discourse chunking. In Australasian Joint Conference on Artificial Intelligence, pages 772–782. Springer. find
  24. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. ↗
  25. Miller, J., Krauth, K., Recht, B., and Schmidt, L. (2020). The effect of natural distribution shift on question answering models. In International Conference on Machine Learning, pages 6905–6916. PMLR. ↗
  26. Reyhani Hamedani, M. and Kim, S.-W. (2021). On investigating both effectiveness and efficiency of embedding methods in task of similarity computation of nodes in graphs. Applied Sciences, 11(1):162. ↗
  27. Ryan, J. O., Mateas, M., and Wardrip-Fruin, N. (2016). A lightweight videogame dialogue manager. In Di-GRA/FDG. find
  28. Schatzmann, J., Thomson, B., Weilhammer, K., Ye, H., and Young, S. (2007). Agenda-based user simulation for bootstrapping a POMDP dialogue system. In Human Language Technologies: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers, pages 149–152.
  29. Tran, Q. H., Haffari, G., and Zukerman, I. (2017a). A generative attentional neural network model for dialogue act classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 524–529.
  30. Tran, Q. H., Zukerman, I., and Haffari, G. (2017b). Preserving distributional information in dialogue act classification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2151–2156.
  31. Webb, N., Hepple, M., and Wilks, Y. (2005). Dialogue act classification based on intra-utterance features. In Proceedings of the AAAI Workshop on Spoken Language Understanding, volume 4, page 5. Citeseer. find
  32. Zhao, R., Romero, O. J., and Rudnicky, A. (2018). Sogo: a social intelligent negotiation dialogue system. In Proceedings of the International Conference on Intelligent Virtual Agents, pages 239–246. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/enayet-2022-analysis-dialogue-act-sequence-similarity/