Elamur Research⌕
☰
home / papers / 2026 / Analysing Next Speaker Prediction in Multi-Party Conversation Using Multimodal Large Language Models

Analysing Next Speaker Prediction in Multi-Party Conversation Using Multimodal Large Language Models

Paper · 2026 · Taiga Mori, Koji Inoue, Divesh Lala et al.

Summary

The paper asks how well current multimodal large language models (MLLMs) can predict who will speak next in three-person conversation. Turn-taking among three or more people depends on subtle cues like addressing and gaze, so it is still hard for dialogue systems. The authors use the Teidan corpus of Japanese triadic discussions with turn annotations. They prompt three GPT-5 series models with the transcript context, and optionally with a short summary of the Sacks et al. (1974) turn-taking rules and a video frame from the end of the turn. The models can answer "O" when the next speaker cannot be determined. They vary three factors: model size, whether the rules are given, and whether the image is included.

The models can guess a plausible next speaker from language and built-in knowledge alone. However, they tend to name one specific participant even when the turn is open to anyone. Giving the rules explicitly reduces this bias. Visual input helps in some cases and misleads in others, with no consistent overall gain. The work links classical conversation analysis with foundation models for multi-party agents.

Key points

  • Data: 12 groups from the Teidan corpus, each recorded in three topic sessions of about 5–7 minutes. In total there are 3,121 turns over 3 h 38 min 27 s, averaging 86.7 turns per dialogue.
  • Conditions are labelled Model–Rule–Image:
    • Models: G5, G5M, G5N from the GPT-5 series, accessed via the Python API with temperature fixed at 1.
    • Rules: provided (R) or not (NR).
    • Image: one screenshot near the end of the turn, included (I) or not (NI).
  • Without the rules, the models overpredict a single next speaker. Adding the rules helps them recognise when no participant is selected.
  • Case studies show both sides of visual input:
    • Gaze and pointing toward a participant led to a correct prediction.
    • In another case, gaze led the model to wrongly assume a particular addressee.
  • Limitations: only GPT-5 models were tested, and a single static frame was used. Future work proposes Gemini 2.5, Qwen2-VL, multi-frame input, and gaze information in text form.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksDialogue systems · 624 worksVision & language · 657 works

Cite

Cite as Taiga Mori, Koji Inoue, Divesh Lala, Keiko Ochi and Tatsuya Kawahara. 2026. Analysing Next Speaker Prediction in Multi-Party Conversation Using Multimodal Large Language Models. Proceedings of the 16th International Workshop on Spoken Dialogue System Technology.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mori2026analysing,
  title = {{Analysing Next Speaker Prediction in Multi-Party Conversation Using Multimodal Large Language Models}},
  author = {Taiga Mori, Koji Inoue, Divesh Lala, Keiko Ochi and Tatsuya Kawahara},
  booktitle = {Proceedings of the 16th International Workshop on Spoken Dialogue System Technology},
  year = {2026},
  url = {https://aclanthology.org/2026.iwsds-1.8/}
}

References

12 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  2. Harvey Sacks, Emanuel A Schegloff, and Gail Jefferson. 1974. A simplest systematics for the organization of turn-taking for conversation. language, 50(4):696– 735. find
  3. Gabriel Skantze, Martin Johansson, and Jonas Beskow. 2015. Exploring turn-taking cues in multi-party human-robot discussions about objects. In Proceedings of the 2015 ACM on international conference on multimodal interaction, pages 67–74. find
  4. Mikey Elmers, Koji Inoue, Divesh Lala, and Tatsuya Kawahara. 2025. Triadic multi-party voice activity projection for turn-taking in spoken dialogue systems. arXiv preprint arXiv:2507.07518. — [arXiv:2507.07518](https://arxiv.org/abs/2507.07518) find
  5. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.
  6. Mika Enomoto, Yasuharu Den, and Yuichi Ishimoto. 2020. A conversation-analytic annotation of turn-taking behavior in japanese multi-party conversation and its preliminary analysis. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 644–652. find
  7. Koji Inoue, Divesh Lala, Mikey Elmers, Keiko Ochi, and Tatsuya Kawahara. 2025. An llm benchmark for addressee recognition in multi-modal multi-party dialogue. arXiv preprint arXiv:2501.16643. find
  8. Yuriko Iseki, Keisuke Kadota, and Yasuharu Den. 2019. Characteristics of everyday conversation derived from the analysis of dialog act annotation. In 2019 22nd conference of the oriental cocosda international committee for the co-ordination and standardisation of speech databases and assessment tech… find
  9. Kristiina Jokinen, Hirohisa Furukawa, Masafumi Nishida, and Seiichi Yamamoto. 2013. Gaze and turn-taking behavior in casual conversational interactions. ACM Transactions on Interactive Intelligent Systems (TiiS), 3(2):1–30. find
  10. Keisuke Kadota, Seima Oyama, and Yasuharu Den. 2024. Annotation of addressing behavior in multi-party conversation. In 2024 27th Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardisation of Speech Databases and Assessment Techniques (O-COCOSDA), pages 1–6.… find
  11. Adam Kendon. 1967. Some functions of gaze-direction in social interaction. Acta psychologica, 26:22–63. find
  12. Meng-Chen Lee and Zhigang Deng. 2024. Online multimodal end-of-turn prediction for three-party conversations. In Proceedings of the 26th International Conference on Multimodal Interaction, pages 57–65. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mori-2026-analysing-next-speaker-prediction-multi/