Elamur Research⌕
☰
home / papers / 2008 / Investigating the Portability of Corpus-Derived Cue Phrases for Dialogue Act Classification

Investigating the Portability of Corpus-Derived Cue Phrases for Dialogue Act Classification

Paper · 2008 · Nick Webb and Ting Liu

Summary

The paper asks whether dialogue act (DA) cue phrases that are learned automatically from one corpus still work when applied unchanged to a corpus from a different domain. The classifier is deliberately simple and looks only inside each utterance. Word n-grams of length 1–4 are scored by predictivity, the conditional probability P(d|n) of a DA given the n-gram. Weak or rare candidates are removed using thresholds tuned on a validation set. Each utterance then receives the DA of its most predictive cue.

The authors train on SWITCHBOARD, which is open-ended chit-chat, and test on the AMITIÉS GE corpus of banking call-centre dialogues, and they also run the reverse direction. Results are scored under strict and lenient matching. Cues taken from SWITCHBOARD reach about 80% of the in-domain AMITIÉS score under lenient scoring, well above the majority baseline. The reverse transfer works poorly, because AMITIÉS lacks conversational categories such as backchannels.

This matters because the study suggests that much DA recognition rests on shallow, domain-independent lexical cues. Such cues can be reused across domains without retraining.

Key points

  • Corpora:
    • SWITCHBOARD has 1155 conversations tagged with 42 clustered DAMSL classes and inter-annotator kappa of .80.
    • AMITIÉS GE has 1000 call-centre dialogues annotated in XDML, with a lower kappa of .59.
    • The most frequent AMITIÉS tag covers 20% of utterances.
  • In-domain results:
    • On SWITCHBOARD the cue classifier scores 69.09% cross-validated, with a best single run of 71.29%. This is comparable to the 71% of Stolcke et al.'s more complex HMM.
    • Adding a DA sequence model gave no significant gain.
  • AMITIÉS upper bound: training on 27k utterances and testing on 3k gives 65.9% strict and 70.8% lenient.
  • SWITCHBOARD → AMITIÉS: 39.8% strict and 55.7% lenient. With SWITCHBOARD training cut to 27k utterances, this drops to 38% strict and 53.2% lenient.
  • AMITIÉS → SWITCHBOARD:
    • The initial scores are 40.0% strict and 48.3% lenient, against a 36% baseline.
    • Changing the fallback tag to <backchannel> raises them to 47.7% and 56.0%.

Interesting ideas

  • I1. Rank every n-gram by its single best conditional probability for any act, and prune by both predictivity and occurrence count, so each decision is traceable to one cue.
  • I2. When no cue fires, the fallback label should reflect the target genre rather than the training majority; switching the fallback to backchannel recovered many SWITCHBOARD errors.
  • I3. Score composite, multi-attribute tags twice: strictly on the full tag and leniently on the most salient part only.
  • I4. Bracket cross-domain results between a majority-tag floor and an in-domain ceiling, and report the transfer score as a fraction of that ceiling.
  • I5. Test transfer in both directions and with size-matched training sets, to separate the effect of data composition from that of data volume.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): crate prag could keep a transparent cue lexicon for speech-act detection, where each label cites the exact phrase that triggered it.
  • I2 (frames): the speech-act stage could use genre-aware defaults, e.g. one fallback for dialogue and another for fables or school texts, and report when a default was used.
  • I3 (learn): evaluation of frames and speech acts could report full-match and core-match accuracy separately, which shows whether errors are minor or fundamental.
  • I4 (learn): silver-data experiments on new text genres could be judged relative to in-genre ceilings and trivial baselines, a natural form of negative control.
  • I5 (learn): when adding a new training source, swap source and target and equalise sizes to find out whether gaps come from missing categories rather than too little data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University at Albany, State University of New YorkAlbany · United States5 works · 83 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 624 worksSpeech act theory · 229 works

Cite

Cite as Nick Webb and Ting Liu. 2008. Investigating the Portability of Corpus-Derived Cue Phrases for Dialogue Act Classification. Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{webb2008investigating,
  title = {{Investigating the Portability of Corpus-Derived Cue Phrases for Dialogue Act Classification}},
  author = {Nick Webb and Ting Liu},
  booktitle = {Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)},
  year = {2008},
  doi = {10.3115/1599081.1599204},
  url = {https://doi.org/10.3115/1599081.1599204}
}

References

21 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bunt, Harry. 1994. THINK, 3:19–31. Context and dialogue control. find
  2. Carletta, J. C., A. Isard, S. Isard, J. Kowtko, G. Doherty-Sneddon, and A. Anderson. 1997. The Reliability of a Dialogue Structure Coding Scheme. Computational Linguistics, 23:13–31. ↗
  3. Cohen, P. R. and C. R. Perrault. 1979. Elements of a plan based theory of speech acts. Cognitive Science, 3. ↗
  4. Core, Mark G. and James Allen. 1997. Coding dialogs with the DAMSL annotation scheme. In AAAI Fall Symposium on Communicative Action in Humans and Machines, MIT, Cambridge, MA. find
  5. Grosz, Barbara and Candace Sidner. 1986. Attention, Intentions, and the Structure of Discourse. Computational Linguistics, 19(3). ↗
  6. Hardy, Hilda, Kirk Baker, Laurence Devillers, Lori Lamel, Sophie Rosset, Tomek Strzalkowski, Cristian Ursu, and Nick Webb. 2002. Multi-layered dialogue annotation for automated multilingual customer service. In Proceedings of the ISLE workshop on Dialogue Tagging for Multimodal Human Computer Interaction, Edinburgh. find
  7. Hardy, H., K. Baker, H. Bonneau-Maynard, L. Devillers, S. Rosset, and T. Strzalkowski. 2003. Semantic and dialogic annotation for automated multilingual customer service. In Eurospeech, Geneva, Switzerland. find
  8. Hardy, H., A. Biermann, R. Bryce Inouye, A. McKenzie, T. Strzalkowski, C. Ursu, N. Webb, and M. Wu. 2005. The AMITIES System: Data-Driven Techniques for Automated Dialogue. Speech Communication, 48:354–373. ↗
  9. Hirschberg, Julia and Diane Litman. 1993. Empirical Studies on the Disambiguation of Cue Phrases. Computational Linguistics, 19(3):501–530.
  10. Jurafsky, Daniel, Rebecca Bates, Noah Coccaro, Rachel Martin, Marie Meteer, Klaus Ries, Elizabeth Shriberg, Andreas Stolcke, Paul Taylor, and Carol Van Ess-Dykema. 1998. Switchboad discourse language modeling project final report. Research Note 30, Center for Language and Speech Processing, Johns Hopkins University, Baltimore. find
  11. Prasad, Rashmi and Marilyn Walker. 2002. Training a Dialogue Act Tagger for Humna-Human and Human-Computer Travel Dialogues. In Proceedings of the 3rd SIGdial workshop on Discourse and Dialogue, Philadelphia, Pennsylvania. find
  12. Reithinger, Norbert and Martin Klesen. 1997. Dialogue act classification using language models. In Proceedings of EuroSpeech-97. Ries, Klaus. find
  13. March, 1999. Hmm and neural network based speech act classification. In Proceddings of the IEEE Conference on Acoustics, Speech and Signal Processing, volume 1, pages 497–500, Phoenix, AZ. find
  14. Samuel, Ken, Sandra Carberry, and K. Vijay-Shanker. 1998. Dialogue act tagging with transformation-based learning. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Montreal. ↗
  15. Samuel, Ken, Sandra Carberry, and K. Vijay-Shanker. 1999. Automatically selecting useful phrases for dialogue act tagging. In Proceedings of the Fourth Conference of the Pacific Association for Computational Linguistics, Waterloo, Ontario, Canada. ↗
  16. Serafin, Riccardo and Barbara Di Eugenio. 2004. FLSA: Extending Latent Semantic Analysis with features for dialogue act classification. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, Barcelona, Spain.
  17. Shriberg, E., R. Dhillon, S. Bhagat, J. Ang, and H. Carvey. 2004. The ICSI meeting recorder dialog act (MRDA) corpus. In Special Interest Group on Discourse and Dialogue (SIGdial), Boston, USA. ↗
  18. Stolcke, A., K. Ries, N. Coccaro, E. Shriberg, R. Bates, D. Jurafsky, P. Taylor, R. Martin, C. Van Ess-Dykema, and M. Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. In Computational Linguistics 26(3), 339–373. ↗
  19. Tur, Gokhan, Umit Guz, and Dilek Hakkani-Tur. 2006. Model Adaptation for Dialogue Act Tagging. In IEEE Spoken Language Technology Workshop. find
  20. Webb, Nick, Mark Hepple, and Yorick Wilks. 2005a. Dialogue Act Classification Based on Intra-Utterance Features. In Proceedings of the AAAI Workshop on Spoken Language Understanding. find
  21. Webb, Nick, Mark Hepple, and Yorick Wilks. 2005b. Empirical determination of thresholds for optimal dialogue act classification. In Proceedings of the Ninth Workshop on the Semantics and Pragmatics of Dialogue. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/webb-2008-investigating-portability-corpus-derived-cue/