Elamur Research⌕
☰
home / papers / 2020 / CIMA: A Large Open Access Dialogue Dataset for Tutoring

CIMA: A Large Open Access Dialogue Dataset for Tutoring

Paper · 2020 · Katherine Stasaski, Kimberly Kao and Marti A. Hearst

Summary

Conversational tutors can be effective learning aids, but there are few large, openly licensed collections of human tutoring dialogue for training neural models. Existing tutoring systems often rely on hand-built rules and assume a single correct reply. The authors present CIMA, a public corpus built by having Amazon Mechanical Turk workers role-play students and tutors in English-to-Italian translation exercises. Each exercise is grounded in an image and a concept description, and related exercises share vocabulary. Student and tutor turns are collected asynchronously, so each student turn gets three independent tutor replies, each labelled with tutoring actions such as hints or questions.

The analysis shows that students need fewer turns as sessions progress. Tutors mostly behave in pedagogically sensible ways, for example answering student questions with hints. They also often choose different strategies, which suggests the data captures more than one valid way to respond. A neural model trained on the data can produce tutor replies that follow a requested action strategy. CIMA is therefore a useful starting point for data-driven, multi-response tutoring agents.

Key points

  • CIMA has two sub-datasets of different difficulty, linked by shared colour words:
    • Shape follows the BURCHAK format, with 6 colours and 5 shapes.
    • Prepositional Phrase has 6 prepositions and 10 coloured objects.
  • Every student turn receives responses from three different crowdworker tutors. Overall tutor agreement on a single action set is only 18.1%.
  • An Action Classifier reached a weighted F1 of 0.72 overall. Per action, scores were 0.93 for Hint, 0.85 for Question, 0.67 for Correction and 0.61 for Confirmation.
  • Utterances from the action-conditioned Generation model matched the requested action 89.8% of the time. Generation quality was also compared with a rule-based baseline using BLEU and BERT F1.
  • The data and code are released at github.com/kstats/CIMA. The model experiments used a 2296/217/107 train/dev/test split.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, BerkeleyBerkeley · United States243 works · 251,774 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 664 works

Cite

Cite as Katherine Stasaski, Kimberly Kao and Marti A. Hearst. 2020. CIMA: A Large Open Access Dialogue Dataset for Tutoring. Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.bea-1.5

@inproceedings{stasaski2020cima,
  title = {{CIMA: A Large Open Access Dialogue Dataset for Tutoring}},
  author = {Katherine Stasaski, Kimberly Kao and Marti A. Hearst},
  booktitle = {Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications},
  year = {2020},
  doi = {10.18653/v1/2020.bea-1.5},
  url = {https://doi.org/10.18653/v1/2020.bea-1.5}
}

References

35 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kate Forbes-Riley and Diane Litman. 2013. When does disengagement correlate with performance in spoken dialog computer tutoring? Int. J. Artif. Intell. Ed., 22(1–2):39–58. find
  2. Vincent Aleven, Octav Popescu, and Kenneth R Koedinger. 2001. A tutorial dialogue system with knowledge-based understanding and classification of student explanations. In Working Notes of 2nd IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems. find
  3. Arthur C. Graesser. 2015. Conversations with autotutor help students learn. International Journal of Artificial Intelligence in Education, 26:124–132. find
  4. John R Anderson, Albert T Corbett, Kenneth R Koedinger, and Ray Pelletier. 1995. Cognitive tutors: Lessons learned. The journal of the learning sciences, 4(2):167–207. find
  5. Arthur C Graesser, Sidney D’Mello, and Natalie Person. 2009. Meta-knowledge in tutoring. The educational psychology series. Handbook of metacognition in education, pages 361–382. find
  6. Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017. Learning end-to-end goal-oriented dialog. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net. find
  7. Arthur C. Graesser, Kurt VanLehn, Carolyn Penstein Rosé, Pamela W. Jordan, and Derek Harter. 2001. Intelligent tutoring systems with conversational dialogue. AI Magazine, 22:39–52. find
  8. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1631–1640, Berlin, Germany. Association for Computat…
  9. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz - a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language…
  10. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
  11. Jean Chandler. 2003. The efficacy of various kinds of error feedback for improvement in the accuracy and fluency of l2 student writing. Journal of Second Language Writing, 12:267–296. find
  12. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. In Proc. ACL.
  13. Guanliang Chen, David Lang, Rafael Ferreira, and Dragan Gasevic. 2019. Predictors of student satisfaction: A large-scale study of human-human online tutorial dialogues. In Proceedings of the 12th International Conference on Educational Data Mining, EDM 2019, Montréal, Canada, July 2-5, 2019. Intern… find
  14. Ben Krause, Marco Damonte, Mihai Dobre, Daniel Duma, Joachim Fainberg, Federico Fancellu, Emmanuel Kahembwe, Jianpeng Cheng, and Bonnie Webber. 2017. Edina: Building an open domain socialbot with self-dialogues. Alexa Prize Proceedings. find
  15. D. Coetzee, Seongtaek Lim, Armando Fox, Bjorn Hartmann, and Marti A. Hearst. 2015. Structuring interactions for large-scale synchronous peer learning. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work and Social Computing, CSCW ’15, page 1139–1152, New York, NY, USA. A… find
  16. Huan Ling and Sanja Fidler. 2017. Teaching machines to describe images via natural language feedback. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 5075–5085, Red Hook, NY, USA. Curran Associates Inc. find
  17. Cristian Danescu-Niculescu-Mizil and Lillian Lee. 2011. Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. In Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics, pages 76–87, Portland, Oregon, USA. Assoc…
  18. Nobal Niraula, Vasile Rus, Rajendra Banjade, Dan Stefanescu, William Baggett, and Brent Morgan. 2014. The DARE corpus: A resource for anaphora resolution in dialogue based intelligent tutoring systems. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14… find
  19. Myroslava O. Dzikovska, Natalie B. Steinhauser, Elaine Farrow, Johanna D. Moore, and Gwen-dolyn E. Campbell. 2014. Beetle ii: Deep natural language understanding and automatic feedback generation for intelligent tutoring in basic electricity and electronics. International Journal of Artificial Intel… find
  20. Benjamin D Nye, Arthur C Graesser, and Xiangen Hu. 2014. Autotutor and family: A review of 17 years of natural language tutoring. International Journal of Artificial Intelligence in Education, 24(4):427–469. find
  21. Kohei Ono, Ryu Takeda, Eric Nichols, Mikio Nakano, and Kazunori Komatani. 2017. Lexical acquisition through implicit confirmations over multiple dialogues. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 50– 59, Saarbrücken, Germany. Association for Computational… find
  22. Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computati…
  23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  24. Yanchao Yu, Arash Eshghi, and Oliver Lemon. 2016. Training an adaptive dialogue policy for interactive learning of visually grounded word meanings. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 339–349, Los Angeles. Association for Computati… find
  25. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  26. Yanchao Yu, Arash Eshghi, and Oliver Lemon. 2017a. Learning how to learn: An adaptive dialogue agent for incrementally learning visually grounded word meanings. In Proceedings of the First Workshop on Language Grounding for Robotics, pages 10–19, Vancouver, Canada. Association for Computational Ling… find
  27. Sherry Ruan, Liwei Jiang, Justin Xu, Bryce Joe-Kun Tham, Zhengneng Qiu, Yeshuang Zhu, Elizabeth L. Murnane, Emma Brunskill, and James A. Landay. 2019. Quizbot: A dialogue-based adaptive learning system for factual knowledge. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Sys… find
  28. Yanchao Yu, Arash Eshghi, Gregory Mills, and Oliver Lemon. 2017b. The BURCHAK corpus: a challenge data set for interactive learning of visually grounded word meanings. In Proceedings of the Sixth Workshop on Vision and Language, pages 1– 10, Valencia, Spain. Association for Computational Linguistics… find
  29. Vasile Rus, Nabin Maharjan, and Rajendra Banjade. 2015a. Unsupervised discovery of tutorial dialogue modes in human-to-human tutorial data. In Proceedings of the Third Annual GIFT Users Symposium, pages 63–80. find
  30. Vasile Rus, Nobal B. Niraula, and Rajendra Banjade. 2015b. Deeptutor: An effective, online intelligent tutoring system that promotes deep learning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI’15, page 4294–4295. AAAI Press. find
  31. Zhou Yu, Alan W. Black, and Alexander I. Rudnicky. 2017c. Learning conversational systems that interleave task and non-task content. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI’17, page 4214–4220. AAAI Press. find
  32. Kurt VanLehn, Arthur C Graesser, G Tanner Jackson, Pamela Jordan, Andrew Olney, and Carolyn P Rosé. 2007. When are tutorial dialogues more effective than reading? Cognitive science, 31(1):3–62.
  33. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find
  34. Kurt VanLehn, Pamela W. Jordan, Carolyn Penstein Rosé, Dumisizwe Bhembe, Michael Böttner, Andy Gaydos, Maxim Makatchev, Umarani Pappuswamy, Michael A. Ringenberg, Antonio Roque, Stephanie Siler, and Ramesh Srivastava. 2002. The architecture of why2-atlas: A coach for qualitative physics essay writ… find
  35. Zhongyu Wei, Qianlong Liu, Baolin Peng, Huaixiao Tou, Ting Chen, Xuanjing Huang, Kam-Fai Wong, and Xiangying Dai. 2018. Task-oriented dialogue system for automatic diagnosis. In Proceedings of

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/stasaski-2020-cima-large-open-access-dialogue/