Elamur Research⌕
☰
home / papers / 2001 / DATE: A Dialogue Act Tagging Scheme for Evaluation of Spoken Dialogue Systems

DATE: A Dialogue Act Tagging Scheme for Evaluation of Spoken Dialogue Systems

Summary

Standard logfile metrics for spoken dialogue systems, such as turn counts, latency and ASR accuracy, say little about what systems actually do in conversation. Comparing systems with different dialogue strategies needs a uniform way to describe their behaviour. Walker and Passonneau address this with DATE (Dialogue Act Tagging for Evaluation). It labels every system utterance along three independent dimensions: speech act, task/subtask, and conversational domain. The conversational-domain dimension separates task talk from two kinds of maintenance work. One is managing the communication channel (confirmations, apologies, greetings). The other is managing the situation frame, meaning instructions and guidance on what users should expect when talking to a machine and how they should go about it. The authors applied the scheme to the DARPA COMMUNICATOR June 2000 corpus of travel-planning dialogues, using an automatic dialogue parser to assign the labels. They then added the resulting counts as predictors in PARADISE models of user satisfaction, which raised explained variance from 38% to 45%. This shows that fine-grained dialogue-act metrics capture differences between systems that matter to users. It also offers a portable basis for comparing strategies such as confirmation and initiative.

Key points

  • The data is the DARPA COMMUNICATOR June 2000 collection: 662 dialogues from 72 users calling nine different travel-planning systems.
  • DATE uses three orthogonal dimensions: SPEECH-ACT, TASK-SUBTASK, and CONVERSATIONAL-DOMAIN. The last has three values: ABOUT-TASK, ABOUT-COMMUNICATION, and ABOUT-SITUATION-FRAME.
  • ABOUT-SITUATION-FRAME is a new category. Such utterances were common in the human-computer dialogues but rare in CMU human-human travel dialogues, and sites differed in placing them up-front versus mid-dialogue.
  • Adding DATE metrics to PARADISE improved the user-satisfaction model by an absolute 7% (38% → 45%). The most useful predictors were meta-dialogue acts, apologies, and acts marking subtask landmarks.
  • The authors treat the scheme as an evolving, partial model of a class of dialogues. Possible extensions include separating price from schedule information, distinguishing summaries from first presentations, and adding politeness ("face") and rhetorical relations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Speech act theory

Cited by

14 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 13 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{walker2001date,
  title = {{DATE: A Dialogue Act Tagging Scheme for Evaluation of Spoken Dialogue Systems}},
  author = {Walker, Marilyn and Passonneau, Rebecca},
  booktitle = {Proceedings of the First International Conference on Human Language Technology Research},
  year = {2001},
  doi = {10.3115/1072133.1072148},
  url = {https://doi.org/10.3115/1072133.1072148}
}

References

31 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Aberdeen and C. Doran. Human-computer and human-human dialogues. DARPA Communicator Principle Investigators Meeting (Philadelphia, PA USA). http://www.dsic-web.net/ito/meetings/communicator sep2000/, September, 2000. ↗
  2. J. Allen and M. Core. Draft of DAMSL: Dialog act markup in several layers. Coding scheme developed by the MultiParty group, 1st Discourse Tagging Workshop, University of Pennsylvania, March 1996, 1997. find
  3. J. F. Allen. Recognizing intentions from natural language utterances. In M. Brady and R. Berwick, editors, Computational Models of Discourse. MIT Press, 1983. find
  4. S. Bird and M. Liberman. A formal framework for linguistic annotation. Speech Communication, 33(1,2):23–60, 2001. ↗
  5. P. Brown and S. Levinson. Politeness: Some universals in language usage. Cambridge University Press, 1987. find our card
  6. J. C. Carletta, A. Isard, S. Isard, J. C. Kowtko, G. Dowerty-Sneddon, and A. H. Anderson. The reliability of a dialogue structure coding scheme. Computational Linguistics, 23-1:13–33, 1997. ↗
  7. R. Cattoni, M. Danieli, A. Panizza, V. Sandrini, and C. Soria. Building a corpus of annotated dialogues: the ADAM experience. In Proc. of the Conference Corpus-Linguistics-2001, Lancaster, U.K., 2001. find
  8. H. H. Clark and E. F. Schaefer. Contributing to discourse. Cognitive Science, 13:259–294, 1989. ↗
  9. S. L. Condon and C. G. Cech. Functional comparison of face-to-face and computer-mediated decision-making interactions. In S. Herring, editor, Computer-Mediated Converstaion. John Benjamins, 1995. find
  10. M. Danieli and E. Gerbino. Metrics for evaluating dialogue strategies in a spoken language system. In Proceedings of the 1995 AAAI Spring Symposium on Empirical Methods in Discourse Interpretation and Generation, pages 34–39, 1995. ↗
  11. B. Di Eugenio, P. W. Jordan, J. D. Moore, and R. H. Thomason. An empirical investigation of collaborative dialogues. In ACL-COLING98, Proceedings of the Thirty-sixth Conference of the Association for Computational Linguistics, 1998. find
  12. M. Finke, M. Lapata, A. Lavie, L. Levin, L. M. Tomokiyo, T. Polzin, K. Ries, A. Waibel, and K. Zechner. Clarity: Inferring discourse structure from speech. In American Association for Artificial Intelligence (AAAI) Symposium on Applying Machine Learning to Discourse Processing Proceedings, Stanford, California, March 1998. find
  13. E. Goffman. Frame Analysis: An Essay on the Organization of Experience. Harper and Row, New York, 1974. find
  14. E. Goffman. Forms of Talk. University of Pennsylvania Press, Philadelphia, Pennsylvania, USA, 1981. find
  15. B. J. Grosz. The representation and use of focus in dialogue understanding. Technical Report 151, SRI International, 333 Ravenswood Ave, Menlo Park, Ca. 94025, 1977. find
  16. A. Isard and J. C. Carletta. Replicability of transaction and action coding in the map task corpus. In M. Walker and J. Moore, editors, AAAI Spring Symposium: Empirical Methods in Discourse Interpretation and Generation, pages 60–67, 1995. find
  17. P. W. Jordan. Intentional Influences on Object Redescriptions in Dialogue: Evidence from an Empirical Study. PhD thesis, Intelligent Systems Program, University of Pittsburgh, 2000. find
  18. D. Jurafsky, E. Shriberg, and D. Biasca. Swbd-damsl labeling project coder’s manual. Technical report, University of Colorado, 1997. available as http://stripe.colorado.edu/ jurafsky/manual.august1.html. ↗
  19. D. Litman. Plan recognition and discourse analysis: An integrated approach for understanding dialogues. Technical Report 170, University of Rochester, 1985. find
  20. D. Marcu. Perlocutions: The achilles’ heel of speech act theory. Journal of Pragmatics, 1999. ↗
  21. M. G. Moser, J. Moore, and E. Glendening. Instructions for coding explanations: Identifying segments, relations and minimal units. Technical Report 96-17, University of Pittsburgh, Department of Computer Science, 1996. find
  22. R. Perrault and J. Allen. A plan-based analysis of indirect speech acts. American Journal of Computational Linguistics, 6:167–182, 1980. ↗ our card
  23. R. Power. A Computer Model of Conversation. PhD thesis, University of Edinburgh, 1974. find
  24. N. Reithinger and E. Maier. Utilizing statistical speech act processing in verbmobil. In ACL 95, 1995. find
  25. D. R.Traum and E. A. Hinkelman. Conversation acts in task-oriented spoken dialogue. Computational Intelligence, 8(3):575–599, 1992. ↗
  26. E. A. Schegloff and H. Sacks. Opening up closings. Semiotica, 8:289–327, 1977. find
  27. E. Shriberg, P. Taylor, R. Bates, A. Stolcke, K. Ries, D. Jurafsky, N. Coccaro, R. Martin, M. Meteer, and C. V. Ess-Dykema. Can prosody aid the automatic classification of dialog acts in conversational speech. Language and Speech: Special Issue on Prosody and Conversation, 2000. find
  28. M. Walker, J. Aberdeen, J. Boland, E. Bratt, J. Garofolo, L. Hirschman, A. Le, S. Lee, S. Narayanan, K. Papineni, B. Pellom, J. Polifroni, A. Potamianos, P. Prabhu, A. Rudnicky, G. Sanders, S. Seneff, D. Stallard, and S. Whittaker. Darpa communicator dialog travel planning systems: The june 2000 data collection. In Submitted to EUROSPEECH 2001, 2001. find
  29. M. A. Walker. Redundancy in collaborative dialogue. In Fourteenth International Conference on Computational Linguistics, pages 345–351, 1992. find
  30. M. A. Walker. Limited attention and discourse structure. Computational Linguistics, 22-2:255–264, 1996. ↗
  31. M. A. Walker, C. A. Kamm, and D. J. Litman. Towards developing general models of usability with PARADISE. Natural Language Engineering: Special Issue on Best Practice in Spoken Dialogue Systems, 2000. Speaker Utterance SYS Welcome. You are logged in as a guest user. You may say repeat, help me out, start over, or, that’s wrong, you can also correct and interrupt the system at any time. What airport woodja like to fly out of? USER MIAMI FLORIDA SYS Leaving from Miami, And, what city are you flying to? USER MINNEAPOLIS MINNESOTA SYS Flying from Miami to Minneapolis, What date would you like to fly? USER UH MONDAY OCTOBER TWENTY THIRD SYS Leaving on the 23rd of October, And what time didja wanna leave? USER UH LATE MORNING SYS From Minneapolis. Are you returning to Miami? USER UM SYS Are you going back to Miami? Please say yes or no. USER NO SYS Okay, it will just take a moment while I look. SYS I found 3 flights from Miami to Minneapolis on October 23rd . For option one I have a flight with United departing at 10 A M and arriving at 3 P M with one stopover and Ticket price is 390 dollars. Please say next option, or, flight details, or, I’ll take it. USER NEXT OPTION SYS For option two I have a flight with Delta departing at 12 and arriving at 4 P M non-stop. Ticket price is 450 dollars. Would you like me to hold this option? USER NEXT OPTION SYS For the last option I have a flight with Northwest departing at 2 P M and arriving at 7 P M with 1 stopover and Ticket price is 470 dollars. say next option, or, flight details, or, I’ll take it. USER NEXT OPTION SYS Here is your third option again. a flight with Northwest departing at 2 P M and arriving at 7 P M with 1 stopover and Ticket price is 470 dollars. say next option, or, flight details, or, I’ll take it. USER I’LL TAKE IT SYS Great! I am adding this flight to your itinerary. Okay, Is Minneapolis your final destination? USER YES IT IS SYS Would you like to make any ground arrangements? USER NO SYS Now, would you like to hear a summary of your itinerary? USER YES PLEASE SYS Okay, Here’s your itinerary. On October 23rd Northwest Flight 123 departs Miami at 2 P M, arrives Pittsburgh at 4 30 P M connecting to Northwest Flight 146 that departs Pittsburgh at 5 P M and arrives Minneapolis at 7 P M. Figure 1: Dialogue Illustrating the Speech Act, Task-Subtask and Conversational Domain Dimensions of DATE Speech Act opening/closing instruct instruct instruct request-info Task nil nil nil nil origin C-Domain task frame frame frame task implicit-conf request-info origin dest communication task implicit-conf request-info origin, dest date communication task implicit-conf request-info date time communication task implicit-conf request-info origin trip-type communication task request-info instruct trip-type nil task frame status-report present-info retrieval itinerary task task present-info offer price itinerary task task present-info present-info offer itinerary itinerary itinerary task task task present-info itinerary task present-info offer itinerary itinerary task task present-info itinerary task present-info offer itinerary itinerary task task acknowledge request-info booking trip-type task task offer ground task offer itinerary task present-info itinerary task ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/walker-2001-date-dialogue-act-tagging-scheme/