Elamur Research⌕
☰
home / papers / 2022 / The TalkMoves Dataset: K-12 Mathematics Lesson Transcripts Annotated for Teacher and Student Discursive Moves

The TalkMoves Dataset: K-12 Mathematics Lesson Transcripts Annotated for Teacher and Student Discursive Moves

Paper · 2022 · Abhijit Suresh, Jennifer Jacobs, Charis Harty et al.

Summary

Privacy and practical constraints mean classroom recordings are rarely shared, so researchers building NLP tools for education have little public data. Existing dialogue-act corpora (Switchboard, ICSI/MRDA, MultiWOZ, EMOTyDA) also do not capture multi-party talk between teachers and students. The authors release the TalkMoves dataset to fill this gap. It contains 567 human-transcribed K-12 mathematics lesson transcripts. Most are complete in-person lessons with whole-class or small-group discussion, and a few are online sessions. The material comes from Inside Mathematics, the TIMSS 1999 video study, Video Mosaic, and anonymized transcripts gathered through the TalkMoves application. Each sentence is segmented by speaker role (teacher or student). Annotators hand-coded each one as one of ten accountable talk moves or as "none." Utterances also carry automatically assigned dialogue acts based on the Switchboard scheme. Two annotators reached high Cohen's kappa agreement, measured before coding and again about halfway through. The corpus and its coding protocol give educators and NLP researchers a theory-grounded resource for studying classroom discourse and training classifiers. Parts of it already power the TalkMoves feedback tool for teachers.

Key points

  • Size: 567 transcripts, 174,186 teacher utterances, 59,874 student utterances, about 1.8M words (15,830 unique). Lessons typically run about 55 minutes.
  • Label set: six teacher talk moves (e.g., Keeping everyone together, Revoicing, Pressing for reasoning) and four student moves (e.g., Making a claim, Providing evidence or reasoning). These were chosen for their frequency, reliability and expert guidance.
  • Sources: Inside Mathematics (Dana Center, UT Austin), TIMSS 1999 video study, Video Mosaic (Rutgers), and the TalkBack studies run through talkmoves.com.
  • Preprocessing: raw transcripts were converted to standardized CSV files, and transcription artifacts such as "[background noise]" were stripped.
  • The closest related resource is the Educational Uptake dataset (Demszky et al., 2021). The released corpus supports the deployed TalkMoves application, which gives teachers automated feedback.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Colorado BoulderBoulder · United States76 works · 51,396 citations🎓University of Colorado SystemBoulder · United States11 works · 1,747 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksSocial Sciences · 2,009 worksEducation · 221 works

Cite

Cite as Abhijit Suresh, Jennifer Jacobs, Charis Harty, Margaret Perkoff, James H. Martin and Tamara Sumner. 2022. The TalkMoves Dataset: K-12 Mathematics Lesson Transcripts Annotated for Teacher and Student Discursive Moves. Proceedings of the Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2azjpibjyj5e · arXiv 2204.09652

@inproceedings{suresh2022talkmoves,
  title = {{The TalkMoves Dataset: K-12 Mathematics Lesson Transcripts Annotated for Teacher and Student Discursive Moves}},
  author = {Abhijit Suresh, Jennifer Jacobs, Charis Harty, Margaret Perkoff, James H. Martin and Tamara Sumner},
  booktitle = {Proceedings of the Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.63317/2azjpibjyj5e},
  eprint = {2204.09652},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.63317/2azjpibjyj5e}
}

References

32 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Association, N. G. et al. (2010). Common core state standards. Washington, DC. find
  2. Boston, M. (2012). Assessing instructional quality in mathematics. The Elementary School Journal, 113(1):76–104. find
  3. Budzianowski, P., Wen, T.-H., Tseng, B.-H., Casanueva, I., Ultes, S., Ramadan, O., and Gašić, M. (2018). Multiwoz–a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. arXiv preprint arXiv:1810.00278.
  4. Candela, A. G., Boston, M. D., and Dixon, J. K. (2020). Discourse actions to promote student access. Mathematics Teacher: Learning and Teaching PK-12, 113(4):266–277. find
  5. Clark, K., Luong, M.-T., Le, Q. V., and Manning, C. D. (2020). Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. find
  6. Demszky, D., Liu, J., Mancenido, Z., Cohen, J., Hill, H., Jurafsky, D., and Hashimoto, T. (2021). Measuring conversational uptake: A case study on student-teacher interactions. arXiv preprint arXiv:2106.03873.
  7. Derry, S. J., Pea, R. D., Barron, B., Engle, R. A., Erickson, F., Goldman, R., Hall, R., Koschmann, T., Lemke, J. L., Sherin, M. G., et al. (2010). Conducting video research in the learning sciences: Guidance on selection, analysis, technology, and ethics. The journal of the learning sciences, 19(1)… find
  8. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  9. Franke, M. L., Turrou, A. C., Webb, N. M., Ing, M., Wong, J., Shin, N., and Fernandez, C. (2015). Student engagement with others’ mathematical ideas: The role of teacher invitation and support moves. The Elementary School Journal, 116(1):126–148. find
  10. Ganesh, A., Palmer, M., and Kann, K. (2021). What would a teacher do? predicting future talk moves. arXiv preprint arXiv:2106.05249. find
  11. Janin, A., Baron, D., Edwards, J., Ellis, D., Gelbart, D., Morgan, N., Peskin, B., Pfau, T., Shriberg, E., Stolcke, A., et al. (2003). The icsi meeting corpus. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03)., volume 1, pages I–I. IEEE… find
  12. Jennifer, J., Scornavacco, K., Harty, C., Suresh, A., Lai, V., and Sumner, T. (in press). Promoting rich discussions in mathematics classrooms: Using personalized, automated feedback to support reflection and instructional change. Teaching and Teacher Education. find
  13. Jurafsky, D. (1997). Switchboard swbd-damsl shallow-discourse-function annotation coders manual. Institute of Cognitive Science Technical Report. find
  14. Krawczyk, B. (2016). Learning from imbalanced data: open challenges and future directions. Progress in Artificial Intelligence, 5(4):221–232. find
  15. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  16. Major, L. and Watson, S. (2018). Using video to support in-service teacher professional development: the state of the field, limitations and possibilities. Technology, Pedagogy and Education, 27(1):49–68. find
  17. McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276–282. find
  18. Michaels, S., O’Connor, M. C., Hall, M. W., and Resnick, L. B. (2010). Accountable talk® sourcebook. Pittsburg, PA: Institute for Learning University of Pittsburgh. Murphy, PK, Wilkinson, IAG, Soter, AO, Hennessey, MN, & Alexander, JF. find
  19. O’Connor, C. and Michaels, S. (2019). Supporting teachers in taking up productive talk moves: The long road to professional learning at scale. International Journal of Educational Research, 97:166– 175. find
  20. O’Connor, C., Michaels, S., and Chapin, S. (2015). Scaling down” to explore the role of talk in learning: From district intervention to controlled classroom study. Socializing intelligence through academic talk and dialogue, pages 111–126. find
  21. Ravi, S. and Kozareva, Z. (2018). Self-governing neural networks for on-device short text classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 887–893. find
  22. Resnick, L. B., Asterhan, C. S., and Clarke, S. N. (2018). Accountable talk: Instructional dialogue that builds the mind. Geneva, Switzerland: The International Academy of Education (IAE) and the International Bureau of Education (IBE) of the United Nations Educational, Scientific and Cultural Organ… find
  23. Saha, T., Patra, A., Saha, S., and Bhattacharyya, P. (2020). Towards emotion-aided multi-modal dialogue act classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4361–4372.
  24. Scornavacco, K., Jacobs, J., and Harty, C. (accepted). Automated feedback on discourse moves teachers’ perceived utility of a big data tool. Annual conference of the American Educational Research Association. find
  25. Shriberg, E., Dhillon, R., Bhagat, S., Ang, J., and Carvey, H. (2004). The icsi meeting recorder dialog act (mrda) corpus. Technical report, INTERNA-TIONAL COMPUTER SCIENCE INST BERKE-LEY CA.
  26. Stolcke, A., Ries, K., Coccaro, N., Shriberg, E., Bates, R., Jurafsky, D., Taylor, P., Martin, R., Ess-Dykema, C. V., and Meteer, M. (2000). Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339–373.
  27. Suresh, A., Sumner, T., Huang, I., Jacobs, J., Foland, B., and Ward, W. (2018). Using deep learning to automatically detect talk moves in teachers’ mathematics lessons. In 2018 IEEE International Conference on Big Data (Big Data), pages 5445–5447. IEEE. find
  28. Suresh, A., Sumner, T., Jacobs, J., Foland, B., and Ward, W. (2019). Automating analysis and feedback to improve mathematics teachers’ classroom discourse. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 9721–9728. find
  29. Suresh, A., Jacobs, J., Clevenger, C., Lai, V., Tan, C., Martin, J. H., and Sumner, T. (2021a). Using ai to promote equitable classroom discussions: The talkmoves application. In International Conference on Artificial Intelligence in Education, pages 344–348. Springer. find
  30. Suresh, A., Jacobs, J., Lai, V., Tan, C., Ward, W., Martin, J. H., and Sumner, T. (2021b). Using transformers to provide teachers with personalized feedback on their classroom discourse: The talkmoves application. find
  31. AAAI 2021 Spring Symposium on Artificial Intelligence for K-12 Education. find
  32. Xu, L., Aranda, G., Widjaja, W., and Clarke, D. (2018). Video-based Research in Education: Cross-disciplinary Perspectives. Routledge. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/suresh-2022-talkmoves-dataset-k-12-mathematics/