Elamur Research⌕
☰
home / papers / 2007 / When Are Tutorial Dialogues More Effective Than Reading?

When Are Tutorial Dialogues More Effective Than Reading?

Paper · 2007 · Kurt VanLehn, Arthur C. Graesser, G. Tanner Jackson et al.

Summary

The paper tests the "interaction hypothesis": the claim that one-on-one natural-language tutoring, by a human or a computer, produces more learning than a less interactive condition covering identical content. A literature review shows mixed evidence. Tutoring usually beats passive reading or lecturing, but it often fails to beat controls in which students also solve problems and get feedback. The authors ran seven experiments in qualitative (conceptual) physics. They compared human tutors, the dialogue systems Why2-Atlas and Why2-AutoTutor (the latter based on latent semantic analysis), and reading-based conditions. The main control was "canned text remediation": students worked the same problems and then read text matching the tutorial content. When students already had the prerequisite knowledge for the material, tutoring and canned text produced roughly equal gains. Tutoring had a clear advantage only when novices studied content pitched above their current level. The work matters because it narrows when costly dialogue technology actually pays off. It also shows that controlling for content is essential when evaluating tutoring.

Key points

  • The review groups prior studies by outcome. Studies supporting the hypothesis (e.g., Wood, Wood & Middleton 1978; Swanson 1992; Evens & Michael) mostly used passive reading or lecture controls. Null results (e.g., Chi et al. 2001; Rosé et al. 2001) came from controls that also involved active work.
  • Experiments 1 and 3 used intermediate students on intermediate content and found tutoring = canned text remediation. In Exp. 3, immediate multiple-choice scores were .725 for Why2-AutoTutor vs. .650 for canned text (p = .084, not significant).
  • In Experiment 2, tutoring outperformed both a textbook condition and no instruction, and the textbook and no-instruction conditions did not differ.
  • Experiments 4 and 5 used novices on intermediate content, and tutoring beat canned text. In Exp. 4, adjusted multiple-choice posttest scores were .740 for spoken human tutoring, .660 for typed human tutoring, and .567 for canned text.
  • Experiments 6 and 7 used novices on novice-level content. Why2-Atlas, Why2-AutoTutor, canned text remediation, and text-only reading all yielded similar learning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States633 works · 195,218 citations🎓University of PittsburghPittsburgh · United States68 works · 25,442 citations🎓University of MemphisMemphis · United States5 works · 713 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksPsychology · 1,673 worksExperimental and Cognitive Psychology · 950 worksDevelopmental and Educational Psychology · 533 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 664 works

Cite

Cite as Kurt VanLehn, Arthur C. Graesser, G. Tanner Jackson, Pamela Jordan, Andrew Olney and Carolyn P. Rosé. 2007. When Are Tutorial Dialogues More Effective Than Reading?. Cognitive Science 31(1).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1080/03640210709336984

@inproceedings{vanlehn2007when,
  title = {{When Are Tutorial Dialogues More Effective Than Reading?}},
  author = {Kurt VanLehn, Arthur C. Graesser, G. Tanner Jackson, Pamela Jordan, Andrew Olney and Carolyn P. Rosé},
  booktitle = {Cognitive Science 31(1)},
  year = {2007},
  doi = {10.1080/03640210709336984},
  url = {https://doi.org/10.1080/03640210709336984}
}

References

79 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aleven, V., Ogden, A., Popescu, O., Torrey, C., & Koedinger, K. (2004). Evaluating the effectiveness of a tutorial dialogue system for self-explanation. In J. Lester, R. M. Vicari & F. Paraguaca (Eds.), Intelligent Tutoring Systems: Seventh International Conference, ITS 2004 (pp. 443-454). Berlin: S… find
  2. Bransford, J. D., Brown, A. L., & Cocking, R. R. (2000). How people learn: Brain, mind, experience and school. Washington, DC: National Academy Press. find
  3. Charney, D. H., Reder, L. M., & Wells, G. W. (1988). Studies of Elaboration in Instructional Texts. In S. find
  4. Doheny-Farina (Ed.), Effective Documentation: What We Have Learned From Research. Cambridge, MA: MIT Press. find
  5. Chi, M. T. H., Bassok, M., Lewis, M., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 15, 145-182. find
  6. Tutoring = Canned Text Remediation = Text Only Tutoring \> Textbook = No instruction Tutoring \> Canned Text Remediation Chi, M. T. H., Siler, S., Jeong, H., Yamauchi, T., & Hausmann, R. G. (2001). Learning from human tutoring. Cognitive Science, 25, 471-533.
  7. Chi, M. T. H., Slotta, J. D., & de Leeuw, N. (1994). From things to processes: A theory of conceptual change for learning science concepts. Learning and Instruction, 4, 27-43. find
  8. Cohen, P. A., Kulik, J. A., & Kulik, C.-L. C. (1982). Educational outcomes of tutoring: A meta-analysis of findings. American Educational Research Journal, 19(2), 237-248. find
  9. Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the craft of reading, writing and mathematics. In L. B. Resnick (Ed.), Knowing, learning and instruction: Essays in honor of Robert Glaser (pp. 543-494). Hillsdale, NJ: Lawrence Erlbaum Associates. find
  10. Core, M. G., Moore, J. D., & Zinn, C. (2003). The role of initiative in tutorial dialogue. In Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics (EACL) (pp. 67-74). find
  11. Craig, S. D., Driscoll, D. M., & Gholson, B. (2004). Constructing knowledge from dialog in an intelligent tutoring system: Interactive learning, vicarious learning and pedagogical agents. Journal of Educational Multimedia and Hypermedia, 13(2), 163-183. find
  12. Craig, S. D., Sullins, J., Witherspoon, A., & Gholson, B. (in press). The deep-level reasoning questions effect: The role of dialog and deep-level reasoning questions during vicarious learning. Cognition and Instruction. find
  13. Cronback, L. J., & Snow, R. E. (1977). Aptitudes and instructional methods: A handbook for research on interactions. find
  14. New York: Irvington. de Kleer, J. (1977). Multiple representations of knowledge in a mechanics problem-solver. In International Joint Conference on Artificial Intelligence (pp. 299-304). find
  15. Cambridge, MA: MIT Press. di Sessa, A. A. (1993). Towards an epistemology of physics. Cognition and Instruction, 10(2 & 3), 105-225. find
  16. Evens, M., & Michael, J. (in press). One-on-one Tutoring By Humans and Machines. Mahwah, NJ: Erlbaum. find
  17. Foltz, P. W., Gilliam, S., & Kendall, S. (2000). Supporting content-based feedback in on-line writing evaluation with LSA. Interactive Learning Environments, 8(2), 111-127. find
  18. Freedman, R., Rose, C. P., Ringenberg, M. A., & VanLehn, K. (2000). ITS Tools for natural language dialogue: A domain-independent parser and planner. In G. Gauthier, C. Frasson & K. VanLehn (Eds.), Intelligent Tutoring Systems: 5th International Conference, ITS 2000 (pp. 433-442). Berlin: Springer. find
  19. Graesser, A. C., Lu, S., Jackson, G. T., Mitchell, H. H., Ventura, M., Olney, A., et al. (2004). AutoTutor: A tutor with dialogue in natural language. Behavioral Reseach Methods, Instruments and Computers, 36(180-193). find
  20. Graesser, A. C., McNamara, D. S., Louwerse, M. M., & Cai, Z. (2004). Coh-Metrix: Analysis of text on cohension and language. Behavioral Reseach Methods, Instruments and Computers, 36, 193-202. find
  21. Graesser, A. C., Moreno, K., Marineau, J., Adcock, A., Olney, A., & Person, N. (2003). AutoTutor improves deep learning of computer literacy: Is it the dialog or the talking head? In U. Hoppe, F. find
  22. Verdejo & J. Kay (Eds.), Proceedings of Artificial Intelligence in Education (pp. 47-54). Amsterdam: IOS. find
  23. Graesser, A. C., & Olde, B. A. (2003). How does one know whether a person understands a device? The quality of questions the person asks when the device breaks down. Journal of Educational Psychology, 95, 524-536. find
  24. Graesser, A. C., & Person, N. (1994). Question asking during tutoring. American Educational Research Journal, 31, 104-137. find
  25. Graesser, A. C., Person, N., Harter, D., & TRG. (2001). Teaching tactics and dialog in AutoTutor. find
  26. International Journal of Artificial Intelligence and Education, 12, 257-279. find
  27. Graesser, A. C., Person, N., & Magliano, J. (1995). Collaborative dialog patterns in naturalistic one-on-one tutoring. Applied Cognitive Psychology, 9, 359-387. find
  28. Graesser, A. C., Wiemer-Hastings, K., Wiemer-Hastings, P., & Kreuz, R. (1999). AutoTutor: A simulation of a human tutor. Journal of Cognitive Systems Research, 1, 15-25. find
  29. Graesser, A. C., Wiemer-Hastings, P., Wiemer-Hastings, K., Harter, D., Person, N., & TRG. (2000). find
  30. Using latent semantic analysis to evaluate the contributions of students in AutoTutor. Interactive Learning Environments. find
  31. Heller, J. I., & Reif, F. (1984). Prescribing effective human problem-solving processes: Problem descriptions in physics. Cognition and Instruction, 1(2), 177-216. find
  32. Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force concept inventory. The Physics Teacher, 30, 141-158. find
  33. Hewitt, P. G. (1987). Conceptual Physcis. Menlo Park, CA: Adison-Wesley. find
  34. Hu, X., Cai, Z., Louwerse, M. M., Olney, A., Penumatsa, P., Graesser, A. C., et al. (2003). A revised algorithm for latent semantic analysis. In G. Gottlob & T. Walsh (Eds.), Proceedings of the 2003 find
  35. International Joint Conference on Artificial Intelligence (pp. 1489-1491). Menlo Park, CA: AAAI Press. Jordan, P., Makatchev, M., Pappuswamy, U., VanLehn, K., & Albacete, P. (in press). A natural language tutorial dialogue system for physics. In Proceedings of FLAIRS 2006. Menlo Park, CA: AAAI Press… find
  36. Jordan, P., Makatchev, M., & Vanlehn, K. (2004). Combining competing language understanding approaches in an intelligent tutoring system. In J. C. Lester, R. M. Vicari & F. Praguacu (Eds.), Intelligent Tutoring Systems: 7th International Conference, ITS 2004 (pp. 346-357). Berlin: Springer-Verlag. find
  37. Jordan, P., Rose, C. P., & VanLehn, K. (2001). Tools for authoring tutorial dialogue knowledge. In J. D. find
  38. Moore, C. Redfield & W. L. Johnson (Eds.), Artificial Intelligence in Education: AI-Ed in the Wired and Wireless future (pp. 222-233). Washington, DC: IOS. find
  39. Jordan, P., & Vanlehn, K. (2002). Discourse processing for explanatory essyas in tutorial applications. In Proceedings of the third SIGdial Workshop on Discourse and Dialogue (pp. 74-83): Association for Computational Linguistics. find
  40. Katz, S., Connelly, J., & Allbritton, D. (2003). Going beyond the problem given: How human tutors use post-solution discussions to support transfer. International Journal of Artificial Intelligence in Education, 13, 79-116. find
  41. Landauer, T. K., Foltz, P. W., & Laham, D. (1998). An introduction to latent semantic analysis. find
  42. Discourse Processes, 25, 259-284. Lane, H. C., & VanLehn, K. (in press). Teaching the tacit knowledge of programming to novices with natural language tutoring. Computer Science Education. find
  43. Lemke, J. L. (1990). Talking science: Language, learning and values. Norwood, NJ: Ablex. find
  44. Litman, D., Rose, C., Forbes-Riley, K., VanLehn, K., Bhembe, D., & Silliman, S. (in press). Spoken versus typed human and computer dialogue. International Journal of Artificial Intelligence and Education. find
  45. Makatchev, M., Hall, B. S., Jordan, P., Papuswamy, U., & VanLehn, K. (2005). Mixed language processing in the Why2-Atlas tutoring system. Paper presented at the Proceedings of the Workshop on Mixed Language Explanations in Learning Environments, AIED2005, Amsterdam,NL. find
  46. Makatchev, M., Jordan, P., & VanLehn, K. (2004). Abductive theorem proving for analyzing student explanations and guiding feedback in intelligent tutoring systems. Journal of Automated Reasoning, 32(3), 187-226. find
  47. McNamara, D. S. (2001). Reading both high-coherence and low-coherence texts: Effects of text sequence and prior knowledge. Canadian Journal of Experimental Psychology, 55(1), 51-62. find
  48. McNamara, D. S., Kintsch, E., Songer, N., & Kintsch, W. (1996). Are good texts always better? find
  49. Interactions of text coherence, background knowledge, and levels of understanding in learning from texts. Cognition and Instruction, 14(1), 1-43. find
  50. Mehan, H. (1979). Learning lessons: Social organization in the classroom. Cambridge, MA: Harvard University Press. find
  51. Michael, J., Rovick, A., Glass, M. S., Zhou, Y., & Evens, M. (2003). Learning from a computer tutor with natural language capabilities. Interactive Learning Environments, 11(3), 233-262. find
  52. Olde, B. A., Franceschetti, D., Karnavat, A., Graesser, A. C., & TRG. (2002). The right stuff: Do you need to sanitize your corpus when using latent semantic analysis? In W. D. Gray & C. D. Schunn (Eds.), Proceedings of the 24th Annual Conference of the Cognitive Science Society (pp. 708-713). find
  53. Olney, A., Louwerse, M. M., Mathews, E. C., Marineau, J., Hite-Mitchell, H., & Graesser, A. C. (2003). utterance classification in AutoTutor. In Proceedings of the HLT-NAACL 03 Workshop on Building Educational Applications Using Natural Language Processing (pp. 1-8). Philadephia, PA: Association for… find
  54. Otero, J., & Graesser, A. C. (2001). PREG: Elements of a model of question asking. Cognition and Instruction, 19, 143-175. find
  55. Person, N., Graesser, A. C., Bautista, L., Mathews, E. C., & TRG. (2001). Evaluating student learning gains in two version of AutoTutor. In J. D. Moore, C. Redfield & W. L. Johnson (Eds.), Artificial Intelligence in Education: AI-ED in the Wired and Wireless future (pp. 268-293). Amsterdam: IOS. find
  56. Person, N., Graesser, A. C., Kreuz, R. J., Pomeroy, V., & TRG. (2001). Simulating human tutor dialog moves in AutoTutor. International Journal of Artificial Intelligence in Education, 12, 23-39. find
  57. Person, N., Graesser, A. C., & TRG. (2002). Human or computer?: AutoTutor in a bystander Turing test. find
  58. In S. A. Cerri, G. Gouarderes & F. Paraguaca (Eds.), Intelligent Tutoring Systems, ITS 2002 (pp. 821-830). Berlin: Springer. find
  59. Ploetzner, R., & VanLehn, K. (1997). The acquisition of informal physics knowledge during formal physics training. Cognition and Instruction, 15(2), 169-206. find
  60. Ranney, M., & Thagard, P. (1988). Explanatory coherence and belief revision in naive physics. In V. find
  61. Patel & G. J. Groen (Eds.), The Tenth Annual Conference of the Cognitive Science Society (pp. 426-432). Mahwah, NJ: Erlbaum. find
  62. Reif, F., & Scott, L. A. (1999). Teaching scientific thinking skills: Students and computers coaching each other. American Journal of Physics, 67(9), 819-831. find
  63. Reiner, M., Slotta, J. D., Chi, M. T. H., & Resnick, L. B. (2000). Naive physics reasoning: A commitment to substance-based conceptions. Cognition and Instruction, 18(1), 1-34. find
  64. Renkl, A. (2002). Worked-out examples: Instructional explanations support learning by self-explanations. find
  65. Renkl, A., & Atkinson, R. K. (2003). Structuring the transition from example study to problem solving in cognitive skill acquisition: A cognitive load perspective. Educational Psychologist, 38(1), 15-22. find
  66. Rose, C. P., Bhembe, D., Siler, S., Srivastava, R., & Vanlehn, K. (2003). Exploring the effectiveness of knowledge construction dialogues. In U. Hoppe, F. Verdejo & J. Kay (Eds.), Artificial Intelligence in Education: Shaping the Future of Learning through Intelligent Technologies (pp. 497-499). find
  67. Rose, C. P., Jordan, P. W., Ringenberg, M., Siler, S., Vanlehn, K., & Weinstein, A. (2001). Interactive conceptual tutoring in Atlas-Andes. In J. D. Moore, C. Redfield & W. L. Johnson (Eds.), Artificial Intelligence in Education: AI-Ed in the Wired and Wireless future (pp. 256-266). Washington, DC:… find
  68. Rose, C. P., Moore, J. D., VanLehn, K., & Allbritton, D. (2001). A comparative evaluation of Socratic versus didactic tutoring. In J. D. Moore & K. Stenning (Eds.), Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society (pp. 897-902). Mahwah, NJ: Erlbaum. find
  69. Rose, C. P., Roque, A., Bhembe, D., & VanLehn, K. (2002). A hybrid language understanding approach for robust selection of tutoring goals. In S. A. Cerri, G. Gouarderes & F. Paraguacu (Eds.), Intelligent Tutoring Systems, 2002: 6th International Conference (pp. 552-561). Berlin: Springer. find
  70. Sinclair, J. M., & Coulthard, R. M. (1975). Towards an analysis of discourse: The English used by teachers and pupils. London: Oxford University Press. find
  71. Slamecka, N. J., & Graf, P. (1978). The generation effect: Delineation of a phenomenon. Journal of Experimental Psychology: Human Learning and Memory, 4(6), 592-604. find
  72. Slotta, J. D., Chi, M. T. H., & Joram, E. (1995). Assessing students' misclassifications of physics concepts: An ontological basis for conceptual change. Cognition and Instruction, 13(3), 373-400. find
  73. Swanson, J. H. (1992). What Does it Take to Adapt Instruction to the Individual? A Case Study of Oneto-One Tutoring. Paper presented at the American Education Research Association, San Francisco, CA. find
  74. VanLehn, K. (in prep.). When is tutoring more effective than less interactive instruction? In M. T. H. Chi (Ed.), Festshrift for Lauren Resnick. find
  75. VanLehn, K., Jordan, P., Rose, C. P., Bhembe, D., Bottner, M., Gaydos, A., et al. (2002). The architecture of Why2-Atlas: A coach for qualitative physics essay writing. In S. A. Cerri, G. Gouarderes & F. find
  76. Paraguacu (Eds.), Intelligent Tutoring Systems, 2002, 6th International Conference (pp. 158-167). find
  77. Vygotsky, L. S. (1978). Mind in Society: The Development of Higher Psychological Processes. find
  78. Wood, D. J., & Middleton, D. (1975). A study of assisted problem-solving. British Journal of Psychology, 66(2), 181-191. find
  79. Wood, D. J., Wood, H., & Middleton, D. (1978). An experimental evaluation of four face-to-face teaching strategies. International Journal of Behavioral Development, 1, 131-147. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vanlehn-2007-tutorial-dialogues-more-effective-than/