Elamur Research⌕
☰
home / papers / 2021 / Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions

Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions

Paper · 2021 · Dorottya Demszky, Jing Liu, Zid Mancenido et al.

Summary

Uptake is when a speaker builds on what their partner just said, for example by acknowledging, repeating, rephrasing, or asking a follow-up question. Teachers who take up student ideas tend to see better learning outcomes, but expert coding of uptake is too costly to scale. The authors release 2246 student–teacher exchanges from US elementary math lesson transcripts (NCTE). Three trained experts rated each exchange. The authors then compare unsupervised ways to score uptake automatically.

Simple word-overlap measures, especially %-IN-T, capture much of uptake and beat sentence-level neural similarity. To go beyond repetition, the authors define uptake as how much a reply depends on the preceding utterance, measured as pointwise Jensen-Shannon divergence (PJSD). They estimate PJSD with a BERT-base model fine-tuned for next utterance classification. PJSD outperforms the overlap baselines and better detects question answering, reformulation and collaborative completion. Unlike the baselines, it shows a significant relationship with instructional quality across each of the three educational datasets tested. This makes it a plausible basis for automated, low-cost feedback to teachers.

Key points

  • Dataset construction: 2246 (student, teacher) pairs were drawn from about 55k candidates in NCTE 4th–5th grade math transcripts (2010–2013, 317 teachers, 4 New England districts).
    • Student turns had to contain at least 5 tokens.
    • Pairs with [Inaudible] markers were dropped.
  • Annotation: 13 expert raters labeled uptake as low, mid or high.
    • Agreement was Spearman ρ = .474 (Fleiss κ = .286), similar to classroom observation protocols such as MQI and CLASS.
  • Model: The PJSD measure comes from BERT-base fine-tuned via next utterance classification on combined classroom data and Switchboard.
    • Switchboard dialog act labels show that PJSD captures more uptake types than %-IN-T.
  • Validation: The measure was checked against outcomes on NCTE, an online one-on-one tutoring dataset (108 tutors, 1821 students) and SimTeacher.
  • Release: Code and annotations are available at github.com/ddemszky/conversational-uptake.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States567 works · 335,190 citations🎓University of Maryland, College ParkCollege Park · United States149 works · 29,191 citations🎓Harvard UniversityCambridge · United States137 works · 45,850 citations🎓University of VirginiaCharlottesville · United States41 works · 10,139 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Science Applications · 184 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 664 works

Cite

Cite as Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather Hill, Dan Jurafsky and Tatsunori Hashimoto. 2021. Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions. arXiv:2106.03873.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2106.03873 · arXiv 2106.03873

@misc{demszky2021measuring,
  title = {{Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions}},
  author = {Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather Hill, Dan Jurafsky and Tatsunori Hashimoto},
  year = {2021},
  doi = {10.48550/arxiv.2106.03873},
  eprint = {2106.03873},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2106.03873}
}

References

55 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. M. Bakhtin. 1981. The dialogic imagination: four essays. University of Texas Press. find
  2. Steven Bird. 2006. NLTK: The Natural Language Toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 69–72, Sydney, Australia. Association for Computational Linguistics.
  3. Jere E Brophy. 1984. Teacher behavior and student achievement. 73. Institute for Research on Teaching, Michigan State University. find
  4. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder for english. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demo…
  5. Guanliang Chen, Rafael Ferreira, David Lang, and Dragan Gasevic. 2019. Predictors of student satisfaction: A large-scale study of human-human online tutorial dialogues. International Educational Data Mining Society. find
  6. Herbert H Clark and Edward F Schaefer. 1989. Contributing to discourse. Cognitive science, 13(2):259– 294. find
  7. David K Cohen. 1988. Teaching practice: Plus ça change. National Center for Research on Teacher Education East Lansing, MI. find
  8. David K Cohen and Jal D Mehta. 2017. Why reform sometimes succeeds: Understanding the conditions that produce reforms that last. American Educational Research Journal, 54(4):644–690. find
  9. Julie Cohen and Dan Goldhaber. 2016. Building a more complete understanding of teacher evaluation using classroom observations. Educational Researcher, 45(6):378–387. find
  10. Julie Cohen, Vivian Wong, Anandita Krishnamachari, and Rebekah Berlin. 2020. Teacher coaching in a simulated environment. Educational Evaluation and Policy Analysis, 42(2):208–231. find
  11. James Collins. 1982. Discourse style, classroom interaction and differential treatment. Journal of Reading Behavior, 14(4):429–437. find
  12. Larry Cuban. 1993. How teachers taught: Constancy and change in American classrooms, 1890-1990. Teachers College Press. find
  13. Cristian Danescu-Niculescu-Mizil and Lillian Lee. 2011. Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs.
  14. Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A computational approach to politeness with application to social factors. In 51st Annual Meeting of the Association for Computational Linguistics, pages 250–259. ACL.
  15. Rosario Delgado and Xavier-Andoni Tibau. 2019. Why cohen’s kappa should be avoided as performance measure in classification. PloS one, 14(9):e0222916. find
  16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  17. Patrick J Donnelly, Nathaniel Blanchard, Andrew M Olney, Sean Kelly, Martin Nystrand, and Sidney K D’Mello. 2017. Words matter: automatic detection of teacher questions in live classroom discourse using linguistics, acoustics, and context. In Proceedings of the Seventh International Learning Analyti… find
  18. Ellice A Forman, Dawn E McCormick, and Richard Donato. 1997. Learning what counts as a mathematical explanation. Linguistics and Education, 9(4):313–339. find
  19. John J Godfrey and Edward Holliman. 1997. Switchboard-1 release 2. Linguistic Data Consortium, Philadelphia, 926:927. find
  20. Barbara J Grosz et al. 1977. The representation and use of focus in a system for understanding dialogs. In IJCAI, volume 67, page 76. Citeseer. find
  21. Michael Alexander Kirkwood Halliday and Ruqaiya Hasan. 1976. Cohesion in English. London: Longmans. find
  22. Jerry R Hobbs. 1979. Coherence and coreference. Cognitive Science, 3(1):67–90. find
  23. Dan Iter, Kelvin Guu, Larry Lansing, and Dan Jurafsky. 2020. Pretraining with contrastive sentence objectives improves discourse performance of language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4859–4870. find
  24. Emily Jensen, Meghan Dale, Patrick J Donnelly, Cath-lyn Stone, Sean Kelly, Amanda Godley, and Sidney K D’Mello. 2020. Toward automated feedback on teacher discourse to enhance teacher learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–13. find
  25. Daniel Jurafsky, Elizabeth Shriberg, and Debra Biasca. 1997. Switchboard SWBD-DAMSL Labeling Project Coder’s Manual, Draft 13. Technical Report 97-02, University of Colorado Institute of Cognitive Science. find
  26. T Kane, H Hill, and D Staiger. 2015. National center for teacher effectiveness main study. icpsr36095-v2. find
  27. Sean Kelly, Robert Bringe, Esteban Aucejo, and Jane Cooley Fruehwirth. 2020. Using global observation protocols to inform research on teaching effectiveness and school improvement: Strengths and emerging limitations. Education Policy Analysis Archives, 28:62. find
  28. Sean Kelly, Andrew M Olney, Patrick Donnelly, Martin Nystrand, and Sidney K D’Mello. 2018. Automatically measuring question authenticity in real-world classrooms. Educational Researcher, 47(7):451– 464. find
  29. Wei-Jen Ko, Greg Durrett, and Junyi Jessy Li. 2019. Linguistically-informed specificity and semantic plausibility for dialogue generation. In Proceedings of NAACL 2019, pages 3456–3466. find
  30. Matthew A Kraft. 2020. Interpreting effect sizes of education interventions. Educational Researcher, 49(4):241–253. find
  31. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
  32. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. find
  33. Ryan Lowe, Iulian Vlad Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. On the evaluation of dialogue systems with next utterance classification. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 264–269. find
  34. George A. Miller and J. G. Beebe-Center. 1956. Some psychological methods for evaluating the quality of translations. Mechanical Translation, 3:73–80. find
  35. Vlad Niculae and Cristian Danescu-Niculescu-Mizil. 2016. Conversational markers of constructive discussions. In Proceedings of NAACL-HLT, pages 568–578. find
  36. Suphakit Niwattanakul, Jatsada Singthongchai, Ekkachai Naenudorn, and Supachanun Wanapu. 2013. Using of Jaccard Coefficient for Keywords Similarity. In Proceedings of the International MultiConference of Engineers and Computer Scientists, volume 1. find
  37. Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. 2016. f-gan: training generative neural samplers using variational divergence minimization. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 271–279. find
  38. Martin Nystrand, Adam Gamoran, Robert Kachur, and Catherine Prendergast. 1997. Opening dialogue. New York: Teachers College Press. find
  39. Martin Nystrand, Lawrence L Wu, Adam Gamoran, Susie Zeiser, and Daniel A Long. 2003. Questions in time: Investigating the structure and dynamics of unfolding classroom discourse. Discourse processes, 35(2):135–198. find
  40. Mary C O’Connor and Sarah Michaels. 1993. Aligning academic task and participation status through revoicing: Analysis of a classroom discourse strategy. Anthropology & Education Quarterly, 24(4):318–335. find
  41. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  42. Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2011. English gigaword fifth edition ldc2011t07, 2011. URL https://catalog.ldc. upenn. edu/LDC2011T07.[Online]. ↗
  43. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
  44. Robert C Pianta, Karen M La Paro, and Bridget K Hamre. 2008. Classroom Assessment Scoring System: Manual K-3. Paul H Brookes Publishing. find
  45. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3973–3983.
  46. Borhan Samei, Andrew M Olney, Sean Kelly, Martin Nystrand, Sidney D’Mello, Nathan Blanchard, Xiaoyi Sun, Marcy Glaus, and Art Graesser. 2014. Domain independent assessment of dialogic properties of classroom discourse. Grantee Submission. find
  47. Cathlyn Stone, Patrick J Donnelly, Meghan Dale, Sarah Capello, Sean Kelly, Amanda Godley, and Sidney K D’Mello. 2019. Utterance-level modeling of indicators of engaging classroom discourse. International Educational Data Mining Society. find
  48. Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016. Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions. In Proceedings of the 25th international conference on world wide web, pages 613–624. find
  49. Deborah Tannen. 1987. Repetition in conversation: Toward a poetics of talk. Language, pages 574–605. find
  50. Learning Mathematics for Teaching Project. 2011. Measuring the mathematical quality of instruction. Journal of Mathematics Teacher Education, 14:25– 47. find
  51. Gordon Wells. 1999. Dialogic inquiry: Towards a socio-cultural practice and theory of education. Cambridge University Press. find
  52. Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019. Transfertransfo: A transfer learning approach for neural network based conversational agents. CoRR, abs/1901.08149. find
  53. Peng Xu, Hamidreza Saghir, Jin Sung Kang, Teng Long, Avishek Joey Bose, Yanshuai Cao, and Jackie Chi Kit Cheung. 2019. A cross-domain transferable neural coherence model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 678–687. find
  54. Xinnuo Xu, Ondřej Dušek, Ioannis Konstas, and Verena Rieser. 2018. Better conversations by modeling, filtering, and optimizing for coherence and diversity. In Proceedings of EMNLP 2018, pages 3981–3991. find
  55. Justine Zhang and Cristian Danescu-Niculescu-Mizil. 2020. Balancing objectives in counseling conversations: Advancing forwards or looking backwards. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5276– 5289. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/demszky-2021-measuring-conversational-uptake-case-study/