Elamur Research⌕
☰
home / papers / 1997 / PARADISE: A Framework for Evaluating Spoken Dialogue Agents

PARADISE: A Framework for Evaluating Spoken Dialogue Agents

Summary

Comparing spoken dialogue agents is difficult. Reference-answer evaluation is tied to one particular dialogue strategy. Metrics such as transaction success or dialogue length cannot say which factor matters more, and they do not carry over to other tasks. Walker et al. propose PARADISE, a decision-theoretic framework in which the top-level goal is maximizing user satisfaction. Satisfaction is driven by task success and by dialogue costs, which cover both efficiency and naturalness.

The task is described as an attribute value matrix (AVM) of the information the agent and user must exchange. This separates what has to be achieved from how a strategy achieves it. Task success is computed with the Kappa coefficient over a confusion matrix that compares dialogue AVMs with scenario keys. Linear regression against user satisfaction then sets the weights that combine success and cost measures into one performance function.

The framework is illustrated with hypothetical data. It works on whole dialogues and on subdialogues, credits partial success, and normalizes for task complexity, so agents doing different tasks can be compared.

Key points

  • The running example is Danieli and Gerbino's train timetable agents. Agent A confirms explicitly, while Agent B confirms implicitly. Agent A was judged more robust, but its dialogues ran about twice as long, and earlier methods could not weigh those two facts against each other.
  • The simplified timetable AVM has four attributes: depart-city, arrival-city, depart-range and depart-time. Two very different dialogues that correctly convey all values produce the same AVM as the scenario key.
  • Off-diagonal cells in the confusion matrix capture misunderstandings that were never corrected. Corrected misunderstandings show up as extra dialogue costs instead. The illustration uses 100 hypothetical dialogues per agent.
  • The framework is also applied to Smith and Gordon's circuit-repair domain, whose subtasks are Introduction, Assessment, Diagnosis, Repair and Test. The AVM attributes map closely onto these subtasks, and Smith and Gordon report a reliability of .82 for their tagging scheme.
  • Stated limitation: Kappa cannot rate some correct solutions as better than others, for example suggesting express rather than local trains. The authors suggest an interval-scaled Kappa or a domain-specific success measure as possible fixes.

Interesting ideas

  • I1. Describe the task separately from the strategy. Defining success only by the information that must end up exchanged makes very different interaction policies directly comparable. This could transfer to evaluating chatbots or form-filling assistants.
  • I2. Use a chance-corrected agreement statistic as the success score. Applying Kappa to task outcomes accounts for how easy a task is by chance. That allows fair comparisons across tasks of different complexity and also gives credit for partial success.
  • I3. Learn metric weights from user judgments. Instead of fixing by hand how much success versus cost matters, regress user satisfaction on the measures. This is a reusable recipe whenever several objective metrics must be merged into one score.
  • I4. Score subdialogues as well as whole dialogues. Computing success and costs for each subtask makes it possible to link performance to individual local strategies, such as a particular confirmation style, rather than only to the whole system.
  • I5. Assign errors to the right bucket. Uncorrected errors lower the success score, while corrected ones count as cost. This cleanly separates final outcome from the effort spent getting there, which is useful in any interactive evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

120 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 113 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/976909.979652

@inproceedings{walker1997paradise,
  title = {{PARADISE: A Framework for Evaluating Spoken Dialogue Agents}},
  author = {Walker, Marilyn A. and Litman, Diane J. and Kamm, Candace A. and Abella, Alicia},
  booktitle = {35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {1997},
  doi = {10.3115/976909.979652},
  url = {https://doi.org/10.3115/976909.979652}
}

References

30 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abella, Alicia, Michael K Brown, and Bruce Buntschuh. 1996. Development principles for dialog-based interfaces. In ECAI-96 Spoken Dialog Processing Workshop, Budapest, Hungary. find
  2. Bates, Madeleine and Damaris Ayuso. 1993. A proposal for incremental dialogue evaluation. In Proceedings of the DARPA Speech and NL Workshop, pages 319-322. find
  3. Carberry, S. 1989. Plan recognition and its use in understanding dialogue. In A. Kobsa and W. Wahlster, editors, User Models in Dialogue Systems. Springer Verlag, Berlin, pages 133-162. find
  4. Carletta, Jean C. 1996. Assessing the reliability of subjective codings. Computational Linguistics, 22(2):249-254. find
  5. Chu-Carrol, Jennifer and Sandra Carberry. 1995. Response generation in collaborative negotiation. In Proceedings of the Conference of the 33rd Annual Meeting of the Association for Computational Linguistics, pages 136-143. find
  6. Cohen, Paul. R. 1995. Empirical Methods for Artificial Intelligence. MIT Press, Boston. find
  7. Danieli, M., W. Eckert, N. Fraser, N. Gilbert, M. Guy-omard, P. Heisterkam p, M. Kharoune, J. Magadur, S. McGlashan, D. Sadek, J. Siroux, and N. Youd. 1992. Dialogue manager design evaluation. Technical Report Project Esprit 2218 SUNDIAL, WP6000-D3. find
  8. Danieli, Morena and Elisabetta Gerbino. 1995. Metrics for evaluating dialogue strategies in a spoken language system. In Proceedings of the 1995 AAAI Spring Symposium on Empirical Methods in Discourse Interpretation and Generation, pages 34-39. find
  9. Doyle, Jon. 1992. Rationality and its roles in reasoning. Computational Intelligence, 8(2):376--409. find
  10. Fraser, Norman M. 1995. Quality standards for spoken dialogue systems: a report on progress in EAGLES. In ESCA Workshop on Spoken Dialogue Systems Vigso, Denmark, pages 157-160. find
  11. Gale, William, Ken W. Church, and David Yarowsky. 1992. Estimating upper and lower bounds on the performance of word-sense disambiguation programs. In Proc. of3Oth ACL, pages 249-256, Newark, Delaware. find
  12. Grosz, Barbara J. and Candace L. Sidner. 1986. Attentions, intentions and the structure of discourse. Computational Linguistics, 12:175-204. find
  13. Hirschberg, Julia and Christine Nakatani. 1996. A prosodic analysis of discourse segments in direction-giving monologues. In 34th Annual Meeting of the Association for Computational Linguistics, pages 286-- 293. find
  14. Hirschman, Lynette, Deborah A. Dahl, Donald P. McKay, Lewis M. Norton, and Marcia C. Linebarger. 1990. Beyond class A: A proposal for automatic evaluation of discourse. In Proceedings of the Speech and Natural Language Workshop, pages 109-113. find
  15. Hirschman, Lynette and Christine Pao. 1993. The cost of errors in a spoken language system. In Proceedings of the Third European Conference on Speech Communication and Technology, pages 1419-1422. find
  16. Joshi, Aravind K., Bonnie L. Webber, and Ralph M. Weischedel. 1984. Preventing false inferences. In COLING84: Proc. lOth International Conference on Computational Linguistics., pages 134-138. find
  17. Kamm, Candace. 1995. User interfaces for voice applications. In David Roe and Jay Wilpon, editors, Voice Communication between Humans and Machines. National Academy Press, pages 422--442. find
  18. Keeney, Ralph and Howard Raiffa. 1976. Decisions with Multiple Objectives: Preferences and Value Tradeoffs. find
  19. Krippendorf, Klaus. 1980. Content Analysis: An Introduction to its Methodology. Sage Publications, Beverly Hills, Ca. find
  20. Litman, Diane and James Allen. 1990. Recognizing and relating discourse intentions and task-oriented plans. In Philip Cohen, Jerry Morgan, and Martha Pollack, editors, Intentions in Communication. MIT Press. find
  21. Passonneau, Rebecca J. and Diane Litman. 1997. Discourse segmentation by human and automated means. Computational Linguistics, 23(1). find
  22. Polifroni, Joseph, Lynette Hirschman, Stephanie Seneff, and Victor Zue. 1992. Experiments in evaluating interactive spoken language systems. In Proceedings of the DARPA Speech and NL Workshop, pages 28-33. find
  23. Pollack, Martha, Julia Hirschberg, and Bonnie Webber. 1982. User participation in the reasoning process of expert systems. In Proceedings First National Conference on Artificial Intelligence, pages pp. 358-361. find
  24. Shriberg, Elizabeth, Elizabeth Wade, and Patti Price. 1992. Human-machine problem solving using spoken language systems (SLS): Factors affecting performance and user satisfaction. In Proceedings of the DARPA Speech and NL Workshop, pages 49-54. find
  25. Siegel, Sidney and N. J. Castellan. 1988. Nonparametric Statistics for the Behavioral Sciences. McGraw Hill. find
  26. Simpson, A. and N. A. Fraser. 1993. Black box and glass box evaluation of the SUNDIAL system. In Proceedings of the Third European Conference on Speech Communication and Technology, pages 1423-1426. find
  27. Smith, Ronnie W. and Steven A. Gordon. 1997. Effects of variable initiative on linguistic behavior in human-computer spoken natural language dialog. Computational Linguistics, 23(1). find
  28. Sparck-Jones, Karen and Julia R. Galliers. 1996. Evaluating Natural Language Processing Systems. Springer. find
  29. Walker, Marilyn A. 1996. The Effect of Resource Limits and Task Complexity on Collaborative Planning in Dialogue. Artificial Intelligence Journal, 85(1-2): 181-243. find
  30. Webber, Bonnie and Aravind Joshi. 1982. Taking the initiative in natural language database interaction: Justifying why. In Coling 82, pages 413--419. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/walker-1997-paradise-framework-evaluating-spoken-dialogue/