Elamur Research⌕
☰
home / papers / 2023 / The NCTE Transcripts: A Dataset of Elementary Math Classroom Transcripts

The NCTE Transcripts: A Dataset of Elementary Math Classroom Transcripts

Paper · 2023 · Dorottya Demszky and Heather Hill

Summary

NLP models trained on classroom transcripts can measure effective math teaching at scale. Progress is limited, however, because few transcript collections are connected to outside variables such as demographics or learning outcomes. Demszky and Hill fill this gap with the NCTE Transcripts. The collection contains 1,660 anonymized full-lesson transcripts from U.S. 4th and 5th grade math classes, recorded by the National Center for Teacher Effectiveness between 2010 and 2013 in four districts that mostly serve historically marginalized students.

Each transcript links to several kinds of metadata:

  • turn-level discourse-move labels
  • CLASS and MQI observation scores
  • teacher and student surveys
  • student administrative records
  • teacher value-added scores

To demonstrate the dataset's value, the authors fine-tune RoBERTa classifiers to detect four moves: on- vs off-task talk, teacher uptake, focusing questions and student reasoning. The classifiers reach moderate to high accuracy. After controlling for covariates, predictions for all four moves correlate significantly with observation scores. The three dialogic moves also correlate positively with value-added scores. This makes the corpus a practical foundation for instructional measures and automated feedback tools for teachers.

Key points

  • Scale: The corpus covers 1,660 lessons (45–60 min each) from 317 teachers and about 10,817 students. A lesson averages roughly 350 turns and 5,733 words, and teachers produce about 88% of the words.
  • Demographics: Teachers are mostly white (65%) and female (84%). Students are mostly of color (43% African American, 23% Hispanic/Latinx, 8% Asian), and 67% qualify for free or reduced lunch.
  • Comparison with other datasets:
    • TalkMoves is public with 567 transcripts but has fewer linked outcomes than NCTE.
    • MET has outcomes, but access is paid and remote-only, and it lacks shared transcripts.
  • Data collection and privacy: Lessons were recorded with the ThereNow Iris system and professionally transcribed. Names were replaced with placeholders such as "Student J", and consent was obtained under Harvard IRB #17768.
  • Availability: The data is released on GitHub (ddemszky/classroom-transcript-analysis). Users must agree not to use it to discriminate against, surveil or penalize teachers or students.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States567 works · 335,190 citations🎓Harvard UniversityCambridge · United States137 works · 45,850 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksSocial Sciences · 2,009 worksEducation · 221 works

Cite

Cite as Dorottya Demszky and Heather Hill. 2023. The NCTE Transcripts: A Dataset of Elementary Math Classroom Transcripts. Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.bea-1.44 · arXiv 2211.11772

@inproceedings{demszky2023ncte,
  title = {{The NCTE Transcripts: A Dataset of Elementary Math Classroom Transcripts}},
  author = {Dorottya Demszky and Heather Hill},
  booktitle = {Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)},
  year = {2023},
  doi = {10.18653/v1/2023.bea-1.44},
  eprint = {2211.11772},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.bea-1.44}
}

References

44 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Robin John Alexander. 2008. Towards Dialogic Teaching: rethinking classroom talk (4th Edition). Dialo-gos. find
  2. Sterling Alic, Dorottya Demszky, Zid Mancenido, Jing Liu, Heather Hill, and Dan Jurafsky. 2022. Computationally identifying funneling and focusing questions in classroom discourse. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), pages 22… find
  3. M. M. Bakhtin. 1981. The dialogic imagination: four essays. University of Texas Press. find
  4. McKenzie Braley and Gabriel Murray. 2018. The group affect and performance (gap) corpus. In Proceedings of the ICMI 2018 Workshop on Group Interaction Frontiers in Technology (GIFT). find
  5. Jere E Brophy. 1984. Teacher behavior and student achievement. 73. Institute for Research on Teaching, Michigan State University. find
  6. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language P…
  7. Shanyce L Campbell and Matthew Ronfeldt. 2018. Observational evaluation of teachers: Measuring more than we bargained for? American Educational Research Journal, 55(6):1233–1267. find
  8. David K Cohen. 1988. Teaching practice: Plus ça change. National Center for Research on Teacher Education East Lansing, MI. find
  9. Daniel Jurafsky, Elizabeth Shriberg, and Debra Biasca. 1997. Switchboard SWBD-DAMSL Labeling Project Coder’s Manual, Draft 13. Technical Report 97-02, University of Colorado Institute of Cognitive Science. find
  10. David K Cohen and Jal D Mehta. 2017. Why reform sometimes succeeds: Understanding the conditions that produce reforms that last. American Educational Research Journal, 54(4):644–690. find
  11. T Kane, H Hill, and D Staiger. 2015. National center for teacher effectiveness main study. icpsr36095-v2. find
  12. Larry Cuban. 1993. How teachers taught: Constancy and change in American classrooms, 1890-1990. Teachers College Press. find
  13. Dorottya Demszky, Jing Liu, Heather C Hill, Dan Jurafsky, and Chris Piech. 2021a. Can automated feedback improve teachers’ uptake of student ideas? evidence from a randomized controlled trial in a large-scale online course. edworkingpaper no. 21-483. Annenberg Institute for School Reform at Brown Un… find
  14. Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather Hill, Dan Jurafsky, and Tatsunori B Hashimoto. 2021b. Measuring conversational uptake: A case study on student-teacher interactions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th…
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  16. John J Godfrey and Edward Holliman. 1997. Switchboard-1 release 2. Linguistic Data Consortium, Philadelphia, 926:927. find
  17. A Gregory, E Ruzek, CA Hafen, A Yee Mikami, JP Allen, and RC Pianta. 2017. My teaching partner-secondary: A video-based coaching model. Theory into practice, 56(1):38–45. find
  18. Heather C Hill, Merrie L Blunk, Charalambos Y Charalambous, Jennifer M Lewis, Geoffrey C Phelps, Laurie Sleep, and Deborah Loewenberg Ball. 2008. Mathematical knowledge for teaching and the mathematical quality of instruction: An exploratory study. Cognition and instruction, 26(4):430–511. find
  19. Nicholas Hunkins, Sean Kelly, and Sidney D’Mello. 2022. “beautiful work, you’re rock stars!”: Teacher analytics to uncover discourse that supports or undermines student motivation, identity, and belonging in classrooms. In LAK22: 12th International Learning Analytics and Knowledge Conference, pages… find
  20. Thomas J Kane, Daniel F McCaffrey, Trey Miller, and Douglas O Staiger. 2013. Have we identified effective teachers? validating measures of effective teaching using random assignment. research paper. met project. Bill & Melinda Gates Foundation. find
  21. Sean Kelly, Robert Bringe, Esteban Aucejo, and Jane Cooley Fruehwirth. 2020. Using global observation protocols to inform research on teaching effectiveness and school improvement: Strengths and emerging limitations. Education Policy Analysis Archives, 28:62. find
  22. Sean Kelly, Andrew M Olney, Patrick Donnelly, Martin Nystrand, and Sidney K D’Mello. 2018. Automatically measuring question authenticity in real-world classrooms. Educational Researcher, 47(7):451– 464. find
  23. Jing Liu and Julie Cohen. 2020. Measuring teaching practices at scale: A novel application of text-as-data methods. EdWorking PaperNo, pages 20–239. find
  24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
  25. Michael Madaio, Su Lin Blodgett, Elijah Mayfield, and Ezekiel Dixon-Román. 2022. Beyond “fairness”: Structural (in) justice lenses on ai for education. In The Ethics of Artificial Intelligence in Education, pages 203–239. Routledge. find
  26. Sarah Michaels and Catherine O’Connor. 2015. Conceptualizing talk moves as tools: Professional development approaches for academically productive discussion. Socializing intelligence through talk and dialogue, 347:362. find
  27. Sarah Michaels, Mary Catherine O’Connor, Megan Williams Hall, and Lauren B Resnick. 2010. Accountable talk sourcebook: For classroom conversation that works. Pittsburgh, PA: University of Pittsburgh Institute for Learning. find
  28. Martin Nystrand, Adam Gamoran, Robert Kachur, and Catherine Prendergast. 1997. Opening dialogue. New York: Teachers College Press. find
  29. Mary C O’Connor and Sarah Michaels. 1993. Aligning academic task and participation status through revoicing: Analysis of a classroom discourse strategy. Anthropology & Education Quarterly, 24(4):318–335. find
  30. James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: Liwc 2001. Mahway: Lawrence Erlbaum Associates, 71(2001):2001. find
  31. Robert C Pianta, Karen M La Paro, and Bridget K Hamre. 2008. Classroom Assessment Scoring System: Manual K-3. Paul H Brookes Publishing. find
  32. Rebecca Powell, Susan Chambers Cantrell, Victor Malo-Juvera, and Pamela Correll. 2016. Operationalizing culturally responsive instruction: Preliminary findings of criop research. Teachers College Record, 118(1):1–46. find
  33. Lynn C Reimer, Katerina Schenke, Tutrang Nguyen, Diane K O’dowd, Thurston Domina, and Mark Warschauer. 2016. Evaluating promising practices in undergraduate stem lecture courses. RSF: The Russell Sage Foundation Journal of the Social Sciences, 2(1):212–233. find
  34. Lorenzo A. Rossi and Omprakash Gnawali. 2014. Language Independent Analysis and Classification of Discussion Threads in Coursera MOOC Forums. In Proceedings of the IEEE International Conference on Information Reuse and Integration (IRI 2014). find
  35. Abhijit Suresh, Jennifer Jacobs, Charis Harty, Margaret Perkoff, James H Martin, and Tamara Sumner. 2022. The talkmoves dataset: K-12 mathematics lesson transcripts annotated for teacher and student discursive moves. arXiv preprint arXiv:2204.09652.
  36. Abhijit Suresh, Tamara Sumner, Jennifer Jacobs, Bill Foland, and Wayne Ward. 2019. Automating analysis and feedback to improve mathematics teachers’ classroom discourse. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 9721–9728. find
  37. Joshua Sussman and Mark R Wilson. 2019. The use and validity of standardized achievement tests for evaluating new curricular interventions in mathematics and science. American Journal of Evaluation, 40(2):190–213. find
  38. Gordon Wells. 1999. Dialogic inquiry: Towards a sociocultural practice and theory of education. Cambridge University Press. find
  39. Katherine Schaeffer. 2021. America’s public school teachers are far less racially and ethnically diverse than their students. find
  40. Daniel L Schwartz, Catherine C Chase, Marily A Oppezzo, and Doris B Chin. 2011. Practicing versus inventing with contrasting cases: The effects of telling first on learning and transfer. Journal of educational psychology, 103(4):759. find
  41. Klara Sedova, Martin Sedlacek, Roman Svaricek, Martin Majcik, Jana Navratilova, Anna Drexlerova, Jakub Kychler, and Zuzana Salamounova. 2019. Do those who talk more learn more? the relationship between student classroom talk and student achievement. Learning and instruction, 63:101217. find
  42. Pooja G Sidney, Shanta Hattikudur, and Martha W Alibali. 2015. How do contrasting cases and self-explanation promote learning? evidence from fraction division. Learning and Instruction, 40:29–38. find
  43. Katherine Stasaski, Kimberly Kao, and Marti A. Hearst. 2020. CIMA: A large open access dialogue dataset for tutoring. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52–64, Seattle, WA, USA → Online. Association for Computational Linguis…
  44. Abhijit Suresh, Jennifer Jacobs, Charis Clevenger, Vivian Lai, Chenhao Tan, James H Martin, and Tamara Sumner. 2021. Using ai to promote equitable classroom discussions: The talkmoves application. In International Conference on Artificial Intelligence in Education, pages 344–348. Springer. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/demszky-2023-ncte-transcripts-dataset-elementary-math/