Elamur Research⌕
☰
home / papers / 2017 / Frames: a corpus for adding memory to goal-oriented dialogue systems

Frames: a corpus for adding memory to goal-oriented dialogue systems

Summary

Most goal-oriented dialogue datasets reflect linear slot-filling: the system gathers constraints, queries a database, and returns results. Real users often behave differently. They weigh alternatives, revisit earlier options, and compare items, so a system needs memory of several distinct preference sets. To support research on such behaviour, the authors built Frames, a corpus of 1369 text-based human-human travel-booking dialogues. The dialogues were gathered in a Wizard-of-Oz setup on Slack. Users received templated trip constraints, and wizards searched a hand-built database of flight-plus-hotel packages. The corpus is fully annotated with dialogue acts, slots and values, the active frame, and cross-frame references. Building on it, the paper defines frame tracking, which extends dialogue state tracking by maintaining multiple semantic frames simultaneously rather than one compressed state. A rule-based tracker barely beats a random baseline and fails especially on slot-less frame switches and on new frames triggered by changed values. This shows the task needs richer modelling, such as text understanding and anaphora resolution.

Key points

  • Corpus scale: Frames has 1369 dialogues averaging 15 turns, for 19986 turns in total. It was collected over 20 days by 12 participants who rotated daily between the user and wizard roles.
  • Task templates: The authors wrote 38 task templates, 14 generic and 24 with role-play backstories, and participants contributed 13 more.
  • Database behaviour: Each task had a set probability that the database would return no matches. In those cases, wizards were shown suggestions produced by relaxing constraints.
  • Annotation and ratings:
    • About 25% of turns carry more than one dialogue act.
    • New acts such as switch frame and request compare were introduced for frame tracking.
    • Over 70% of dialogues received the top user rating of 5.
  • Evaluation setup: The baselines use 10 leave-one-user-out splits. One is an NLU tagger built on character trigrams with separate act and slot heads; the other is the rule-based frame tracker.
  • Frame tracker accuracy (one fold):
    • Frame switches: 44.9% with slots vs 16.3% without, and 54.3% after an offer vs 16.5% otherwise.
    • Frame creation caused by a changed value: only 5.5%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

114 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 108 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-5526

@inproceedings{asri2017frames,
  title = {{Frames: a corpus for adding memory to goal-oriented dialogue systems}},
  author = {Asri, Layla El and Schulz, Hannes and Sharma, Shikhar and Zumer, Jeremie and Harris, Justin and Fine, Emery and Mehrotra, Rahul and Suleman, Kaheer},
  booktitle = {Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue},
  year = {2017},
  doi = {10.18653/v1/w17-5526},
  url = {https://doi.org/10.18653/v1/w17-5526}
}

References

34 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Arnold, F. A. Gers, T. Kilias, and A. Löser. 2016. Robust named entity recognition in idiosyncratic domains. arXiv:1608.06757 [cs.CL] . — [arXiv:1608.06757](https://arxiv.org/abs/1608.06757) find
  2. Christina L. Bennett and Alexander I. Rudnicky. 2002. The carnegie mellon communicator corpus. In Proc. of Interspeech. find
  3. Paul A Crook and Oliver Lemon. 2010. Representing uncertainty about complex user goals in statistical dialogue systems. In Proc. of SIGDIAL. pages 209– 212. find
  4. Paul A Crook, Zhuoran Wang, Xingkun Liu, and Oliver Lemon. 2012. A statistical spoken dialogue system using complex user goals and value directed compression. In Proc. of EACL. Association for Computational Linguistics, pages 46–50. find
  5. Layla El Asri, Rémi Lemonnier, Romain Laroche, Olivier Pietquin, and Hatim Khouzaimi. 2014. NAS-TIA: Negotiating Appointment Setting Interface. In Proc. of LREC. find
  6. Mehdi Fatemi, Layla El Asri, Hannes Schulz, Jing He, and Kaheer Suleman. 2016. Policy networks with two-stage training for dialogue systems. In Proc. of SIGDIAL. find
  7. M. Gašić, M. Henderson, B. Thomson, P. Tsiakoulis, and S. Young. 2012. Policy optimisation of POMDP-based dialogue systems without state space compression. In Proc. of SLT. find
  8. M. Henderson, B. Thomson, and J. Williams. 2014a. The Third Dialog State Tracking Challenge. In Proceedings of IEEE Spoken Language Technology. find
  9. Matthew Henderson. 2015. Machine learning for dialog state tracking: A review. In First International Workshop on Machine Learning in Spoken Language Processing. find
  10. Matthew Henderson, Blaise Thomson, and Jason D. Williams. 2014b. The second dialog state tracking challenge. In Proc. of SIGDIAL. find
  11. John F. Kelley. 1984. An iterative design methodology for user-friendly natural language office information applications. ACM Transaction on Information Systems . find
  12. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. Proc. of ICLR . find
  13. Romain Laroche, Ghislain Putois, Philippe Bretier, Martin Aranguren, Julia Velkovska, Helen Hastie, Simon Keizer, Kai Yu, Filip Jurčı́ček, Oliver Lemon, and Steve Young. 2011. Report D6.4 : Final evaluation of classic towninfo and appointment scheduling systems. Technical report, CLASSIC Project. find
  14. Romain Laroche, Ghislain Putois, Philippe Bretier, and Bernadette Bouchon-Meunier. 2009. Hybridisation of expertise and reinforcement learning in dialogue systems. In Proc. of Interspeech. find
  15. Sungjin Lee and Amanda Stent. 2016. Task lineages: Dialog state tracking for flexible interaction. In Proc. of SIGDIAL. find
  16. Grégoire Mesnil, Xiaodong He, Li Deng, and Yoshua Bengio. 2013. Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding. In Proc. of Interspeech. find
  17. Teruhisa Misu, Komei Sugiura, Tatsuya Kawahara, Kiyonori Ohtake, Chiori Hori, Hideki Kashioka, Hisashi Kawai, and Satoshi Nakamura. 2011. Modeling spoken decision support dialogue and optimization of its dialogue strategy. Transactions on Speech and Language Processing 7(3):10. find
  18. Wendy W. Moe and Peter S. Fader. 2001. Uncovering patterns in cybershopping. California Management Review . find
  19. Alice H. Oh and Alexander I. Rudnicky. 2000. Stochastic language generation for spoken dialogue systems. In Proc. of ANLP/NAACL workshop on Conversational Systems. find
  20. Julien Perez and Fei Liu. 2016. Dialog state tracking, a machine reading approach using memory network. In Proc. of EACL. find
  21. Olivier Pietquin, Matthieu Geist, Senthilkumar Chandramohan, and Hervé Frezza-Buet. 2011. Sample-efficient batch reinforcement learning for dialogue management optimization. ACM Transaction on Speech and Language Processing 7(3):1–21. find
  22. Antoine Raux, Brian Langner, Allan Black, and Maxine Eskenazi. 2003. LET’S GO: Improving Spoken Dialog Systems for the Elderly and Non-natives. In Proc. of Eurospeech. find
  23. Verena Rieser, Ivana Kruijff-Korbayov, and Oliver Lemon. 2005. A corpus collection and annotation framework for learning multimodal clarification strategies. In Proc. of SIGDIAL. find
  24. Verena Rieser and Oliver Lemon. 2009. Natural language generation as planning under uncertainty for spoken dialogue systems. In Proc. of EACL. pages 683–691. find
  25. Verena Rieser, Oliver Lemon, and Simon Keizer. 2014. Natural language generation as incremental planning under uncertainty: adaptive information presentation for statistical dialogue systems. Transactions on Audio, Speech and Language Processing 22(5):979–994. find
  26. A. Saha, M. Khapra, and K. Sankaranarayanan. 2017. Multimodal Dialogs (MMD): A large-scale dataset for studying multimodal domain-aware conversations. arXiv:1704.00200 [cs.CL] . — [arXiv:1704.00200](https://arxiv.org/abs/1704.00200) find
  27. Jost Schatzmann, Karl Weilhammer, Matt Stuttle, and Steve Young. 2006. A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. The knowledge engineering review 21(2):97–126. find
  28. Shikhar Sharma, Jing He, Kaheer Suleman, Hannes Schulz, and Philip Bachman. 2017. Natural language generation in dialogue using lexicalized and delexicalized data. Proc. of ICLR Workshop . find
  29. Satinder Singh, Michael Kearns, Diane Litman, and Marilyn Walker. 2002. Optimizing dialogue management with reinforcement learning: experiments with the njfun system. Journal of Artificial Intelligence Research 16:105–133. find
  30. M.a. Walker, S.j. Whittaker, A. Stent, P. Maloor, J. Moore, M. Johnston, and G. Vasireddy. 2004. Generation and evaluation of user tailored responses in multimodal dialogue. Cognitive Science 28(5):811–840. find
  31. Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Lina Maria Rojas-Barahona, Pei-Hao Su, Stefan Ultes, David Vandyke, and Steve J. Young. 2016. A network-based end-to-end trainable task-oriented dialogue system. In Proc. of EACL. find
  32. Tsung-Hsien Wen, Milica Gasic, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. In Proc. of EMNLP. find our card
  33. Jason Weston, Sumit Chopra, and Antoine Bordes. 2015. Memory networks. Proc. of ICLR . find
  34. Jason D. Williams, Antoine Raux, and Matthew Henderson. 2016. The dialog state tracking challenge series: A review. Dialogue and Discourse . find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/asri-2017-frames-corpus-adding-memory-goal/