Elamur Research⌕
☰
home / papers / 2011 / Exploring Effective Dialogue Act Sequences in One-on-one Computer Science Tutoring Dialogues

Exploring Effective Dialogue Act Sequences in One-on-one Computer Science Tutoring Dialogues

Paper · 2011 · Lin Chen, Barbara Di Eugenio, Davide Fossati et al.

Summary

The paper asks which patterns of tutor and student moves actually help students learn in one-on-one Computer Science tutoring. It also aims to turn those patterns into design guidance for iList, an intelligent tutoring system for linked-list problems. The authors recorded and transcribed 54 sessions on Lists, Stacks and Trees. Each session had one of two tutors: an experienced professor (LOW) or an undergraduate (JAC). The tutors turned out to be equally effective, so the authors set aside expert-versus-novice comparisons. Instead, they annotated a small set of dialogue acts and used multiple linear regression to predict learning gain from pre-test score, session length and counts of DA sequences.

The key methodological step is "flexible" bigrams and trigrams that may contain gaps, so that filler turns such as a student's "ok" do not break a pattern. Several sequences correlate significantly with learning, and the best trigram models beat models built from single DAs. These sequences are cognitively plausible and directly usable as rules when designing tutoring systems.

Key points

  • Corpus and conditions.
    • 54 sessions were collected over 3 semesters, with 30 students tutored by LOW and 24 by JAC.
    • A control group of 53 students attended an unrelated 40-minute lecture instead of tutoring.
    • Tutored students gained significantly more than the controls on every topic (ANOVA, p < 0.01).
    • The two tutors did not differ significantly in learning gain.
  • Dialogue act inventory. The DA set is minimalist and grounded in skill-acquisition theory:
    • DPI (procedural instruction) and DDI (declarative instruction);
    • +FB and -FB (positive and negative feedback);
    • Prompt (PT) and Student Initiative (SI).
    • Tutors were highly directive, producing 93.5% of all words.
  • Flexible matching. Sequences with gaps of 0–3 intervening DAs were matched using fuzzy matching in Apache Lucene. Transcription followed the CHILDES conventions.
  • Best models (Table 8).
    • Lists: {PT, DPI, FB}, R² = .415.
    • Stacks: {DDI, FB, PT}, R² = .416.
    • Trees: {FB, SI, DDI}, R² = .732.
    • R² gains over the best simpler models were 0.038, 0.12 and 0.056 respectively.
  • Other findings.
    • The trigram {DPI, FB, DDI} was significant for Lists.
    • Models combining two different sequences as predictors were never significant.

Interesting ideas

  • I1. Allowing gaps inside n-grams lets a pattern survive interruptions by meaningless backchannel turns, which keeps short exchanges from dominating sequence statistics.
  • I2. When the "expert vs. novice" contrast showed no difference, the authors switched to explaining outcomes with regression over behaviours rather than over group labels.
  • I3. A deliberately tiny, theory-motivated act inventory (tell how, tell what, confirm, correct, prompt, initiative) made the annotation tractable and every coefficient interpretable.
  • I4. A rare category such as student initiative can still surface as a significant predictor in regression, even though frequency counting would overlook it.
  • I5. Controlling for pre-test score separates the effect of a pattern from what the learner already knew.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): When chaining speech acts into exchanges such as instruction → feedback → explanation, skip acknowledgement turns within a bounded window of up to 3 acts, so dialogue structure reflects real moves.
  • I2 (learn): Evaluate repair rules or parser variants by regressing downstream accuracy on how often each rule fires, not only by comparing system A with system B.
  • I3 (frames): A compact, explainable speech-act set for instructional texts in the prag crate could label teacher-style sentences in school-course material.
  • I4 (learn): In error analysis, test rare constructions as explicit predictors of failure instead of discarding them as low-count noise.
  • I5 (answer): When scoring student-reader improvements on tests, condition on baseline per-item difficulty so that gains are not credited to easy items.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations🎓University of Illinois ChicagoChicago · United States38 works · 19,844 citations🎓Carnegie Mellon University QatarDoha · Qatar13 works · 1,947 citations

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Psychology · 798 works Developmental and Educational Psychology · 260 works

🧭 Topics 2 Dialogue systems · 401 works Speech act theory · 160 works

Cite

Cite as Lin Chen, Barbara Di Eugenio, Davide Fossati, Stellan Ohlsson and David Cosejo. 2011. Exploring Effective Dialogue Act Sequences in One-on-one Computer Science Tutoring Dialogues. Proceedings of the Sixth Workshop on Innovative Use of NLP for Building Educational Applications.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{chen2011exploring,
  title = {{Exploring Effective Dialogue Act Sequences in One-on-one Computer Science Tutoring Dialogues}},
  author = {Lin Chen, Barbara {Di Eugenio}, Davide Fossati, Stellan Ohlsson and David Cosejo},
  booktitle = {Proceedings of the Sixth Workshop on Innovative Use of NLP for Building Educational Applications},
  year = {2011},
  url = {https://aclanthology.org/W11-1408/}
}

References

27 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. John R. Anderson. 1986. Knowledge compilation: The general learning mechanism. Machine learning: An artificial intelligence approach, 2:289–310. find
  2. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. Survey Article. ↗ our card
  3. Kristy Elizabeth Boyer, Robert Phillips, Amy Ingram, Eun Young Ha, Michael Wallis, Mladen Vouk, and James Lester. 2010. Characterizing the effectiveness of tutorial dialogue with Hidden Markov Models. In Intelligent Tutoring Systems, pages 55–64. Springer. find
  4. Whitney L. Cade, Jessica L. Copeland, Natalie K. Person, and Sidney K. D’Mello. 2008. Dialogue modes in expert tutoring. In Intelligent Tutoring Systems, volume 5091 of Lecture Notes in Computer Science, pages 470–479. Springer Berlin / Heidelberg. find
  5. Lin Chen and Barbara Di Eugenio. 2010. A lucene and maximum entropy model based hedge detection system. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 114–119, Uppsala, Sweden, July. Association for Computational Linguistics. our card
  6. Michelene T. H. Chi, Nicholas de Leeuw, Mei-Hung Chiu, and Christian LaVancher. 1994. Eliciting self-explanations improves understanding. Cognitive Science, 18(3):439–477. ↗
  7. Michelene T. H. Chi, Stephanie A. Siler, Takashi Yamauchi, and Robert G. Hausmann. 2001. Learning from human tutoring. Cognitive Science, 25:471–533. ↗
  8. Min Chi, Kurt VanLehn, and Diane Litman. 2010. The more the merrier? Examining three interaction hypotheses. In Proceedings of the 32nd Annual Conference of the Cognitive Science Society (CogSci2010), Portland,OR. find
  9. Michelene T.H. Chi. 1997. Quantifying qualitative analyses of verbal data: A practical guide. Journal of the Learning Sciences, 6(3):271–315. ↗
  10. Herbert H. Clark. 1992. Arenas of Language Use. The University of Chicago Press, Chicago, IL. find
  11. Barbara Di Eugenio and Michael Glass. 2004. The Kappa statistic: a second look. Computational Linguistics, 30(1):95–101. Squib. ↗
  12. Barbara Di Eugenio, Trina C. Kershaw, Xin Lu, Andrew Corrigan-Halpern, and Stellan Ohlsson. 2006. Toward a computational model of expert tutoring: a first report. In FLAIRS06, the 19th International Florida AI Research Symposium, Melbourne Beach, FL. find
  13. Barbara Di Eugenio, Davide Fossati, Stellan Ohlsson, and David Cosejo. 2009. Towards explaining effective tutorial dialogues. In Annual Meeting of the Cognitive Science Society, pages 1430–1435, Amsterdam, July. find
  14. Martha W. Evens and Joel A. Michael. 2006. One-on-one Tutoring by Humans and Machines. Mahwah, NJ: Lawrence Erlbaum Associates. find
  15. Davide Fossati, Barbara Di Eugenio, Christopher Brown, Stellan Ohlsson, David Cosejo, and Lin Chen. 2009. Supporting Computer Science curriculum: Exploring and learning linked lists with iList. IEEE Transactions on Learning Technologies, Special Issue on Real-World Applications of Intelligent Tutoring Systems, 2(2):107–120, April-June. ↗
  16. Davide Fossati, Barbara Di Eugenio, Stellan Ohlsson, Christopher Brown, and Lin Chen. 2010. Generating proactive feedback to help students stay on track. In ITS 2010, 10th International Conference on Intelligent Tutoring Systems. Poster. find
  17. Barbara A. Fox. 1993. The Human Tutorial Dialogue Project: Issues in the design of instructional systems. Lawrence Erlbaum Associates, Hillsdale, NJ. find
  18. Arthur C. Graesser, Natalie K. Person, and Joseph P. Magliano. 1995. Collaborative dialogue patterns in naturalistic one-to-one tutoring. Applied Cognitive Psychology, 9:495–522. ↗
  19. Arthur C. Graesser, Shulan Lu, George Tanner Jackson, Heather Hite Mitchell, Mathew Ventura, Andrew Olney, and Max M. Louwerse. 2004. AutoTutor: A tutor with dialogue in natural language. Behavioral Research Methods, Instruments, and Computers, 36:180–193. ↗
  20. Mark R. Lepper, Michael F. Drake, and Teresa O’Donnell-Johnson. 1997. Scaffolding techniques of expert human tutors. In K. Hogan and M. Pressley, editors, Scaffolding student learning: Instructional approaches and issues. Cambridge, MA: Brookline. find
  21. Diane Litman and Kate Forbes-Riley. 2006. Correlations between dialogue acts and learning in spoken tutoring dialogues. Natural Language Engineering, 12(02):161–176. ↗
  22. Brian MacWhinney. 2000. The Childes Project: Tools for Analyzing Talk: Transcription format and programs, volume 1. Psychology Press, 3 edition. find
  23. Johanna D. Moore, Kaska Porayska-Pomsta, Sebastian Varges, and Claus Zinn. 2004. Generating Tutorial Feedback with Affect. In FLAIRS04, Proceedings of the Seventeenth International Florida Artificial Intelligence Research Society Conference. find
  24. Stellan Ohlsson, Barbara Di Eugenio, Bettina Chow, Davide Fossati, Xin Lu, and Trina C. Kershaw. 2007. Beyond the code-and-count analysis of tutoring dialogues. In Proceedings of the 13th International Conference on Artificial Intelligence in Education, pages 349–356, Los Angeles, CA, July. IOS Press. find
  25. Stellan Ohlsson. 2008. Computational models of skill acquisition. The Cambridge handbook of computational psychology, pages 359–395. find
  26. Ron Sun, Paul Slusarz, and Chris Terry. 2005. The Interaction of the Explicit and the Implicit in Skill Learning: A Dual-Process Approach. Psychological Review, 112:159–192. ↗
  27. Kurt VanLehn, Arthur C. Graesser, G. Tanner Jackson, Pamela W. Jordan, Andrew Olney, and Carolyn P. Rosé. 2007. When are tutorial dialogues more effective than reading? Cognitive Science, 31(1):3–62. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2011-exploring-effective-dialogue-act-sequences/