Elamur Research⌕
☰
home / papers / 2026 / Modeling Turn-Taking with Semantically Informed Gestures

Modeling Turn-Taking with Semantically Informed Gestures

Paper · 2026 · Varsha Suresh, M. Hamza Mughal, Christian Theobalt et al.

Summary

Computational turn-taking models usually rely on speech content and prosody and ignore body movement. This paper asks whether meaning-bearing gestures add useful signal for predicting whether a speaker will keep or give up the floor in multi-party conversation. The authors extend the DnD Gesture corpus, which contains six hours of Dungeons & Dragons sessions with five English speakers, 3D motion capture, audio and transcripts. They manually label semantic gesture types following McNeill's taxonomy to create DnD Gesture++. They then segment the conversations into IPUs and label each one as hold or yield.

Their model is a Mixture-of-Experts with three experts: text, audio and gesture. A gating network weights the experts. The gesture expert uses VQ-VAE motion tokens, which are passed to a transformer and supervised with the gesture-type labels.

Adding gestures improves on text and audio alone. The semantically supervised gesture embeddings beat raw-motion ones, and the yield class gains the most. The resource is also useful for gesture generation and multi-party interaction research.

Key points

  • DnD Gesture++ annotations: 2,663 semantic gesture labels.
    • Counts by type: 1,155 deictic, 724 iconic, 633 discourse and 151 metaphoric.
    • Density is about 444 labels per hour, compared with roughly 288 in BEAT.
    • Beat gestures were deliberately excluded.
  • Annotator agreement: average Cohen's κ is 0.52. Agreement on discourse gestures is very low (κ ≈ 0.08).
  • Turn-taking data: about 12k turns (around 7k hold and 5k yield), defined with a 200 ms silence threshold. The data is split 70/10/20 into train, validation and test.
  • Main result: Text+Audio+Gesture reaches macro-F1 69.9, compared with 67.9 for Text+Audio and 68.7 when gestures lack semantic supervision.
    • Yield F1 rises to 63.1.
    • MoE fusion also outperforms ConcatFusion (68.2) and LMF (67.9).
  • Analysis: discourse gestures are associated mainly with yields, and metaphoric gestures with holds. Semantic supervision produces clearer clusters in the gesture embedding space and raises the gating weights for both gesture and text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Saarland UniversitySaarbrücken · Germany148 works · 18,287 citations🔬Max Planck Institute for InformaticsSaarbrücken · Germany16 works · 4,453 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksHuman-Computer Interaction · 141 works

Cite

Cite as Varsha Suresh, M. Hamza Mughal, Christian Theobalt and Vera Demberg. 2026. Modeling Turn-Taking with Semantically Informed Gestures. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.106

@inproceedings{suresh2026modeling,
  title = {{Modeling Turn-Taking with Semantically Informed Gestures}},
  author = {Varsha Suresh, M. Hamza Mughal, Christian Theobalt and Vera Demberg},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.106},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.106}
}

References

39 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Janet Beavin Bavelas, Nicole Chovil, Linda Coates, and Lori Roe. 1995. Gestures specialized for dialogue. Personality and social psychology bulletin, 21(4):394–405. find
  2. Starkey Duncan, Lawrence J Brunner, and Donald W Fiske. 1979. Strategy signals in face-to-face interaction. Journal of Personality and Social Psychology, 37(2):301. find
  3. Erik Ekstedt and Gabriel Skantze. 2020. Turngpt: a transformer-based language model for predicting turn-taking in spoken dialog. Findings of the Association for Computational Linguistics: EMNLP 2020. find
  4. Erik Ekstedt and Gabriel Skantze. 2022. How much does prosody help turn-taking? investigations using voice activity projection models. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 541–551. Edinburgh UK: Association for Computational Linguis… find
  5. Erik Ekstedt, Siyang Wang, Éva Székely, Joakim Gustafson, and Gabriel Skantze. 2023. Automatic evaluation of turn-taking cues in conversational speech synthesis. In Proc. find
  6. Interspeech 2023, pages 5481–5485. find
  7. Esam Ghaleb, Bulat Khaertdinov, Asli Özyürek, and Raquel Fernández. 2025. I see what you mean: Co-speech gestures for reference resolution in multimodal dialogue. In The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025). find
  8. Susan Goldin-Meadow. 2014. How gesture works to change our minds. Trends in neuroscience and education, 3(1):4–6. find
  9. Susan Goldin-Meadow, Martha Wagner Alibali, and R Breckinridge Church. 1993. Transitions in concept acquisition: using the hand to read the mind. Psychological review, 100(2):279. find
  10. Kohei Hara, Koji Inoue, Katsuya Takanashi, and Tatsuya Kawahara. 2019. Turn-taking prediction based on detection of transition relevance place. In Interspeech, pages 4170–4174. find
  11. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778.
  12. Emily Hofstetter. 2021. Achieving preallocation: Turn transition practices in board games. Discourse processes, 58(2):113–133. find
  13. Judith Holler, Kobin H Kendrick, and Stephen C Levinson. 2018. Processing language in face-to-face conversation: Questions with gestures get faster responses. Psychonomic bulletin & review, 25(5):1900– 1908. find
  14. Judith Holler and Stephen C Levinson. 2019. Multimodal language processing in human communication. Trends in cognitive sciences, 23(8):639–652. find
  15. Kristiina Jokinen, Hirohisa Furukawa, Masafumi Nishida, and Seiichi Yamamoto. 2013. Gaze and turn-taking behavior in casual conversational interactions. ACM Transactions on Interactive Intelligent Systems (TiiS), 3(2):1–30. find
  16. Anneliese Kelterer and Barbara Schuppler. 2025. Turn-taking annotation for quantitative and qualitative analyses of conversation. arXiv preprint arXiv:2504.09980. — [arXiv:2504.09980](https://arxiv.org/abs/2504.09980) find
  17. Adam Kendon. 1995. Gestures as illocutionary and discourse structure markers in southern italian conversation. Journal of pragmatics, 23(3):247–279. find
  18. Kobin H Kendrick, Judith Holler, and Stephen C Levinson. 2023. Turn-taking in human face-to-face interaction is multimodal: gaze direction and manual gestures aid the coordination of turn transitions. Philosophical transactions of the royal society B, 378(1875):20210473. find
  19. Zohreh Khosrobeigi, Maria Koutsombogera, and Carl Vogel. 2022. Gesture and part-of-speech alignment in dialogues. In Proceedings of the 26th Workshop on the Semantics and Pragmatics of Dialogue, pages 172–182. find
  20. Taras Kucherenko, Rajmund Nagy, Patrik Jonell, Michael Neff, Hedvig Kjellström, and Gustav Eje Henter. 2021. Speech2Properties2Gestures: Gesture-property prediction as a tool for generating representational gestures from speech. In Proceedings of the 21th ACM International Conference on Intelligent… find
  21. Fuma Kurata, Mao Saeki, Shinya Fujie, and Yoichi Matsuyama. 2023. Multimodal turn-taking model using visual cues for end-of-utterance prediction in spoken dialogue systems. In Proc. Interspeech, pages 2658–2662. find
  22. Meng-Chen Lee, Mai Trinh, and Zhigang Deng. 2023. Multimodal turn analysis and prediction for multi-party conversations. In Proceedings of the 25th International Conference on Multimodal Interaction, pages 436–444. find
  23. Yuxin Lin, Yinglin Zheng, Ming Zeng, and Wangzheng Shi. 2025. Predicting turn-taking and backchannel in human-machine conversations using linguistic, acoustic, and visual signals. arXiv preprint arXiv:2505.12654. find
  24. Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, and Michael J Black. 2024. Emage: Towards unified holistic co-speech gesture generation via expressive masked audio gesture modeling. find
  25. Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. 2022. Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis. European Conference on Computer Vision. find
  26. Andy Lücking, Kirsten Bergman, Florian Hahn, Stefan Kopp, and Hannes Rieser. 2013. Data-based analysis of speech and gesture: The bielefeld speech and gesture alignment corpus (saga) and its applications. Journal on Multimodal User Interfaces, 7:5–18. find
  27. David McNeill. 1992. Hand and mind: What gestures reveal about thought. find
  28. David McNeill. 2005. Gesture and Thought. University of Chicago Press. find
  29. M. Hamza Mughal, Rishabh Dabral, Merel C. J. Scholman, Vera Demberg, and Christian Theobalt. 2025. Retrieving semantics from the deep: an rag solution for gesture synthesis. In Computer Vision and Pattern Recognition (CVPR). find
  30. Muhammad Hamza Mughal, Rishabh Dabral, Ikhsanul Habibie, Lucia Donatelli, Marc Habermann, and Christian Theobalt. 2024. Convofusion: Multi-modal conversational diffusion for co-speech gesture synthesis. In Computer Vision and Pattern Recognition (CVPR). find
  31. Simbarashe Nyatsanga, Taras Kucherenko, Chaitanya Ahuja, Gustav Eje Henter, and Michael Neff. 2023. A comprehensive review of data-driven co-speech gesture generation. In Computer Graphics Forum, volume 42, pages 569–596. Wiley Online Library. find
  32. Ashwin Ram, Varsha Suresh, Artin Saberpour Abadian, Vera Demberg, and Jürgen Steimle. 2025. Gesture-coach: Rehearsing for engaging talks with llm-driven gesture recommendations. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology, UIST ’25, New York, NY, USA. As… find
  33. Sam O’Connor Russell and Naomi Harte. 2025. Visual cues enhance predictive turn-taking for two-party human interaction. arXiv preprint arXiv:2505.21043. find
  34. Harvey Sacks, Emanuel A Schegloff, and Gail Jefferson. 1974. A simplest systematics for the organization of turn-taking for conversation. language, 50(4):696– 735. find
  35. Gabriel Skantze. 2021. Turn-taking in conversational systems and human-robot interaction: a review. Computer Speech & Language, 67:101178. find
  36. Varsha Suresh, M Hamza Mughal, Christian Theobalt, and Vera Demberg. 2025. Enhancing spoken discourse modeling in language models using gestural cues. arXiv preprint arXiv:2503.03474. find
  37. Marlijn Ter Bekke, Linda Drijvers, and Judith Holler. 2024. Gestures speed up responses to questions. Language, Cognition and Neuroscience, 39(4):423–430. find
  38. Aaron Van Den Oord, Oriol Vinyals, and 1 others. 2017. Neural discrete representation learning. Advances in neural information processing systems, 30. find
  39. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/suresh-2026-modeling-turn-taking-semantically-informed/