Elamur Research⌕
☰
home / papers / 2002 / Training a Dialogue Act Tagger for Human-human and Human-computer Travel dialogues

Training a Dialogue Act Tagger for Human-human and Human-computer Travel dialogues

Paper · 2002 · Rashmi Prasad and Marilyn Walker

Summary

Hand-labelling dialogues with dialogue act tags is costly. Earlier, the DATE (Dialogue Act Tagging for Evaluation) scheme was applied to DARPA Communicator data with a pattern matcher built on a large hand-labelled pattern database. That matcher broke down when system prompts changed: it labelled only about 60% of the October-2001 data. Prasad and Walker try to automate DATE tagging of the information provider's turns, meaning the system in human-computer (HC) dialogues and the travel agent in human-human (HH) dialogues. They train the rule learner RIPPER on the June-2000 and October-2001 Communicator corpora and the CMU HH travel corpus, using textual and contextual features, and test several mixes of these sources. Within HC data, accuracy is very high. A tagger trained on older systems drops sharply on newer ones, but a modest amount of new labelled data recovers most of the loss. When HH labels are scarce, adding HC data raises HH accuracy. This suggests cheap machine-generated dialogues can help bootstrap annotation of harder human conversation.

Key points

  • Corpora:
    • June-2000: 663 dialogues, 9 systems, 22,930 system utterances.
    • October-2001: 1,252 dialogues, 8 systems, 69,766 utterances.
    • CMU-corpus: 38 HH dialogues, 1,062 agent utterances, segmented so that each unit carries one dialogue act.
  • DATE dimensions: DATE labels each utterance on three orthogonal axes: SPEECH ACT, CONVERSATIONAL-DOMAIN and TASK-SUBTASK. The first two are meant to be domain-general; the third is domain- and system-specific.
  • HC results:
    • Training and testing on June-2000 gives 98.5% accuracy.
    • The same tagger reaches only 71.8% on October-2001.
    • Adding 2,000 October-2001 utterances to training raises this to 93.8%.
  • HH results (speech-act tags):
    • HC-only training gives 36.7%, against a 28% majority baseline.
    • 305 HH training utterances give 48.75%; adding the HC data raises this to 55.48%.
    • 20-fold cross-validation on the CMU-corpus reaches 76.56%.
  • Why RIPPER: it accepts textual (set-valued word) features and produces ordered, readable if-then rules.

Interesting ideas

  • I1. Pair a large, uniform, easy source (machine dialogues) with a few labels from the hard target (human dialogues); the easy data helps most when target labels are scarce and its benefit fades as target data grows.
  • I2. Learn a classifier that outputs ordered if-then rules over word and context features, so each decision can be traced to the rule that fired and the features it used.
  • I3. Test a tagger trained on an older version of a source on a later version, to expose drift. A brittle exact-pattern approach lost much of its coverage when prompts changed slightly, while a learned model plus a little fresh data recovered.
  • I4. Error analysis showed that the same act surfaces differently across settings: human agents give information in small pieces that look like the system's implicit confirmations.
  • I5. Tag dialogue acts on separate orthogonal dimensions, keeping the general axes (speech act, conversational domain) apart from the domain-specific task axis.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mix plentiful silver data from simple, regular texts with small gold sets from harder genres, and measure how the gain changes as the gold set grows.
  • I2 (frames): Speech-act classification in prag could use induced, human-readable rules rather than opaque scores, which keeps each step explainable.
  • I3 (learn): Add drift checks that train on one time slice or genre and evaluate on another, flagging rule sets that only match exact surface strings.
  • I4 (frames): Speech-act rules should rely on dialogue context and function, not just utterance length or form, so that short informative turns are not mistaken for confirmations.
  • I5 (frames): Keep the speech-act label separate from the task or frame slot, so the general part carries over to new domains while only the task layer needs domain knowledge.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢AT&TBedminster · United States40 works · 75,466 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 654 worksSpeech act theory · 229 works

Cite

Cite as Rashmi Prasad and Marilyn Walker. 2002. Training a Dialogue Act Tagger for Human-human and Human-computer Travel dialogues. Proceedings of the Third SIGdial Workshop on Discourse and Dialogue.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/1118121.1118142

@inproceedings{prasad2002training,
  title = {{Training a Dialogue Act Tagger for Human-human and Human-computer Travel dialogues}},
  author = {Rashmi Prasad and Marilyn Walker},
  booktitle = {Proceedings of the Third SIGdial Workshop on Discourse and Dialogue},
  year = {2002},
  doi = {10.3115/1118121.1118142},
  url = {https://doi.org/10.3115/1118121.1118142}
}

References

19 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Allen and M. Core. 1997. Draft of DAMSL: Dialog act markup in several layers. Coding scheme developed by the MultiParty group, 1st Discourse Tagging Workshop, Univ. of Penn, March 1996. find
  2. R. Cattoni, M. Danieli, A. Panizza, V. Sandrini, and C. Soria. 2001. Building a corpus of annotated dialogues: the ADAM experience. In Proc. of the Conference Corpus-Linguistics-2001, Lancaster, U.K. find
  3. W. Cohen. 1996. Learning trees and rules with set-valued features. In 14th Conference of AAAI. find
  4. B. Di Eugenio, P. W. Jordan, J. D. Moore, and R. H. Thomason. 1998. An empirical investigation of collaborative dialogues. In ACL-COLING98, Proc. of the 36th ACL Conference. find
  5. C. Doran, J. Aberdeen, L. Damianos, and L. Hirschman. 2001. Comparing several aspects of human-computer and human-human dialogues. In SIGDIAL Workshop in conjuction with Eurospeech 2001. find
  6. M. Eskenazi, A. Rudnicky, K. Gregory, P. Constantinides, R. Brennan, K. Bennett, and J. Allen. 1999. Data collection and processing in the carnegie mellon communicator. In Proc. of Eurospeech-99, pages 2695–2698. find
  7. M. Finke, M. Lapata, A. Lavie, L. Levin, L. Mayfield Tomokiyo, T. Polzin, K. Ries, A. Waibel, and K. Zechner. 1998. Clarity: Inferring discourse structure from speech. In AAAI Symposium on Applying Machine Learning to Discourse Processing Proceedings, Stanford, California. find
  8. H. Hastie, R. Prasad, and M. A. Walker. 2002. Automatic evaluation: Using a date dialogue act tagger for user satisfaction and task completion prediction. In LREC 2002.
  9. A. Isard and J. C. Carletta. 1995. Replicability of transaction and action coding in the map task corpus. In M. A. Walker and J. Moore, eds., AAAI Spring Symposium: Empirical Methods in Discourse Interpretation and Generation, pages 60–67. find
  10. P. W. Jordan. 2000. Intentional Influences on Object Redescriptions in Dialogue: Evidence from an Empirical Study. Ph.D. thesis, Intelligent Systems Program, University of Pittsburgh. find
  11. N. Reithinger and M. Klesen. 1997. Dialogue act classification using language models. In Proc. of Eurospeech ’97, pages 2235–2238, Rhodes, Greece. find
  12. K. Samuel, S. Carberry, and K. Vijay-Shanker. 1998. Dialogue act tagging with transformation-based learning. In Proc. of COLING-ACL, pages 1150–1156. ↗
  13. K. Scheffler and S. Young. 2002. Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning. In HLT Conference. find
  14. E. Shriberg, P. Taylor, R. Bates, A. Stolcke, K. Ries, D. Jurafsky, N. Coccaro, R. Martin, M. Meteer, and C. Van Ess-Dykema. 2000. Can prosody aid the automatic classification of dialog acts in conversational speech. Language and Speech: Special Issue on Prosody and Conversation.
  15. M. A. Walker, D. Litman, C. A. Kamm, and A. Abella. 1997. PARADISE: A general framework for evaluating spoken dialogue agents. In Proc. of ACL/EACL 97, pages 271–280. find
  16. M. A. Walker, J. Aberdeen, J. Boland, E. Bratt, J. Garofolo, L. Hirschman, A. Le, S. Lee, S. Narayanan, K. Papineni, B. Pellom, J. Polifroni, A. Potamianos, P. Prabhu, A. Rudnicky, G. Sanders, S. Seneff, D. Stallard, and S. Whittaker. 2001a. Darpa communicator dialog travel planning systems: The june 2000 data collection. In Proc. of EUROSPEECH 2001. find
  17. M. A. Walker, O. Rambow, and M. Rogati. 2001b. Spot: A trainable sentence planner. In Proc. of NAACL. find
  18. M. A. Walker, R. Passonneau, and J. E. Boland. 2001c. Quantitative and qualitative evaluation of DARPA communicator spoken dialogue systems. In Proc. of ACL 2001. find
  19. M. A. Walker. 2000. An application of reinforcement learning to dialogue strategy selection in a spoken dialogue system for email. Journal of Artificial Intelligence Research, 12:387–416. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/prasad-2002-training-dialogue-act-tagger-human/