Elamur Research⌕
☰
home / papers / 2026 / K-MIND: Korean Multimodal INteraction Data for Dyadic Conversation Analysis

K-MIND: Korean Multimodal INteraction Data for Dyadic Conversation Analysis

Summary

Most multimodal dialogue corpora are scripted, task-driven or segment-based, and Korean has very little data that records spontaneous face-to-face conversation across all channels. Existing Korean sets are small: one has about 35 minutes of cartoon narration, and KEMDy20 has roughly 7 hours of emotion-elicited talk. K-MIND fills this gap. It is a large corpus of unscripted Korean dyadic conversations between pairs of participants in many kinds of relationships, real or simulated, talking about everyday topics. Synchronized video, audio and transcripts are annotated in three aligned layers:

  • verbal: communicative intents, including fine-grained backchannel types;
  • paraverbal: prosodic functions;
  • nonverbal: gesture and behavior functions.

These layers feed into higher-level affective labels. Full double annotation was too costly, so quality comes from a staged workflow with dictionary-constrained descriptions and sequential expert review. The authors also report label distributions and analyses within and across layers. The resource supports Korean and cross-linguistic research on socially aware, culturally adaptive conversational AI. A triadic extension (K-MIND Triadic) is announced for future releases.

Key points

  • Scale: 292 native Korean speakers (165 female, 127 male, aged 10–79), 200 sessions split into 935 clips. The total is about 115.5 hours; the abstract and the body differ by 5 minutes. Average clip length is 7:02, ranging from 2:01 to 17:04.
  • Recording (2022–2024):
    • five cameras: two face close-ups, two waist shots and one full scene;
    • three microphones: one boom mic and two pin mics;
    • files stored as MP4 (1920×1080, ~30 fps) and 48 kHz WAV.
  • Annotation scheme:
    • 21 verbal intent categories, such as Req.Subj (split into Closed and Open), Elab.Obj, several BC subtypes, Reflection and Recall;
    • 4 paraverbal functions: Amplification, Expressivity, Grammar and Others, with multiple labels allowed per utterance;
    • 7 nonverbal functions: Representational, Highlighting, Pointing, Focusing, Responsive, Attitudinal and Conventional;
    • an emotion label set ranging from Happiness to Neutral, with entries such as Hope, Thinking and Humour.
  • Annotation process: ran in three stages from 2023 to 2025 with a custom tool. There were 21 contributors: 19 annotators and 5 reviewers, with 3 people doing both. Team meetings were held weekly, later monthly, and random spot-checks were added.
  • Availability: the corpus is released at github.com/AIRC-KETI/K-MIND (Korea Electronics Technology Institute and MaumAI).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3mz4q73vpu6q

@inproceedings{yang2026k,
  title = {{K-MIND: Korean Multimodal INteraction Data for Dyadic Conversation Analysis}},
  author = {Yang, Jae Hee and Shin, Yuha and Shin, Saim and Kim, Je Woo and Jang, Jin Yea},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3mz4q73vpu6q},
  url = {https://doi.org/10.63317/3mz4q73vpu6q}
}

References

44 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vasu Agrawal, Akinniyi Akinyemi, Kathryn Alvero, Morteza Behrooz, Julia Buffalini, Fabio Maria Carlucci, Joy Chen, Junming Chen, Zhang Chen, Shiyang Cheng, Praveen Chowdary, Joe Chuang, Antony D’Avirro, Jon Daly, Ning Dong, Mark Duppenthaler, Cynthia Gao, Jeff Girard, Martin Gleize, Sahir Gomez, Hon… find
  2. Andrew J Aubrey, David Marshall, Paul L Rosin, Jason Vendeventer, Douglas W Cunningham, and Christian Wallraven. 2013. Cardiff conversation database (ccdb): A database of natural dyadic conversations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 2… find
  3. Pawel Bujnowski, AmirAli Zadeh, Kwan-Yee Lee, Louis-Philippe Morency, and Taehwan Kim. 2024. Samsemo: A multilingual multimodal dataset for emotion recognition. In Proceedings of the Language Resources and Evaluation Conference (LREC-COLING 2024), Torino, Italy. European Language Resources Associati… find
  4. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N. Chang, Sungbok Lee, and Shrikanth S. Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42(4):335– 359. find
  5. Alexandra Canavan, David Graff, and George Zipperlen. 1997. Callhome american english speech. Linguistic Data Consortium. find
  6. Dorothy M. Chun. 2002. Discourse intonation in L2: from theory and research to practice. J. Benjamins. find
  7. H.H. Clark and S.E. Haviland. 1975. Comprehension and the Given-new Contract. Social sciences working paper. School of Social Sciences, University of California, Irvine. find
  8. Justine Coupland. 2014. Small talk. Routledge. find
  9. S. Eggins and D. Slade. 2004. Analysing Casual Conversation. Equinox Textbooks and Surveys in Linguistics. University of Toronto Press. find
  10. Paul Ekman. 1992. An argument for basic emotions. Cognition & emotion, 6(3-4):169–200. find our card
  11. Paul Ekman and Wallace V. Friesen. 2015. Felt, false, and miserable smiles. Nonverbal Behavior, 6:238–252. find
  12. Florian Eyben, Felix Weninger, Florian Gross, and Björn Schuller. 2013. Recent developments in openSMILE, the munich open-source multimedia feature extractor. In Proceedings of the 21st ACM International Conference on Multimedia (MM ’13), pages 835–838. ACM. find
  13. Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. openSMILE: The munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM International Conference on Multimedia (MM ’10), pages 1459–1462. ACM. find
  14. Eric N Forsythand and Craig H Martell. 2007. Lexical and discourse analysis of online chat dialog. In International Conference on Semantic Computing (ICSC 2007), pages 19–26. IEEE. find
  15. John J Godfrey, Edward C Holliman, and Jane Mc-Daniel. 1992. Switchboard: Telephone speech corpus for research and development. In Acoustics, speech, and signal processing, ieee international conference on, volume 1, pages 517–520. IEEE Computer Society. find
  16. Koichiro Ito, Masaki Murata, Tomohiro Ohno, and Shigeki Matsubara. 2020. Relation between degree of empathy for narrative speech and type of responsive utterance in attentive listening. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 696–701. find
  17. Adam Kendon. 1967. Some functions of gaze-direction in social interaction. Acta psychologica, 26:22–63. find
  18. Haeyeon Kim. 2002. Collaborative turn completion in korean conversation. Language Research, pages 1,281–1,316. find
  19. Hansaem Kim. 2015. A research on non-verbal behavior of korean: Focused on gestures accompanied by interjection [in korean]. Journal of Bangyo Language and Literature, 65:5–28. find
  20. Jinsung Kim et al. 2020. Kemdy20: Korean multimodal emotion dialog dataset. find
  21. Satwik Kottur, Seungwhan Moon, Alborz Geramifard, and Babak Damavandi. 2021. SIMMC 2.0: A task-oriented dialog dataset for immersive multimodal conversations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4903–4912, Online and Punta Cana, Dominican… find
  22. Wessel Kraaij, Thomas Hain, Mike Lincoln, and Wilfried Post. 2005. The ami meeting corpus. In Proc. International Conference on Methods and Techniques in Behavioral Research, pages 1–4. find
  23. Petri Laukka, Patrik N. Juslin, and Roberto Bresin. 2005. A dimensional approach to vocal expression of emotion. Cognition & Emotion, 19(5):633– 653. find
  24. Hyewon Lee. 2016. A study on empathic listener response strategies for promoting communication: focusing on korean language learners. find
  25. Stephen C. Levinson and Francisco Torreira. 2015. Timing in turn-taking and its implications for processing models of language. Frontiers in Psychology, Volume 6 - 2015. find
  26. Andrew David Marshall, Paul L Rosin, Jason Vandeventer, and Andrew Aubrey. 2015. 4d cardiff conversation database (4d ccdb): A 4d database of natural, dyadic conversations. Auditory-Visual Speech Processing,{AVSP} 2015, pages 157– 162. find
  27. James R Martin and Peter R White. 2003. The language of evaluation, volume 2. Springer. find
  28. Albert Mehrabian and James A Russell. 1974. An approach to environmental psychology. the MIT Press. find
  29. Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. 2019. OpenDialKG: Explainable conversational reasoning with attention-based walks over knowledge graphs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 845–854, Florence, Italy. Association… find our card
  30. Paula M. Niedenthal, Martial Mermillod, Marcus Maringer, and Ursula Hess. 2010. The simulation of smiles (sims) model: Embodied simulation and the meaning of facial expression. Behavioral and Brain Sciences, 33(6):417–433. find
  31. Catharine Oertel, Fred Cummins, Jens Edlund, Petra Wagner, and Nick Campbell. 2013. D64: A corpus of richly recorded conversational interaction. Journal on Multimodal User Interfaces, 7(1):19–28. find
  32. Emily Öhman. 2020. Emotion annotation: Rethinking emotion categorization. In DHN post-proceedings, pages 134–144. find
  33. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 527–536, Floren… find our card
  34. Andrew Reece, Gus Cooney, Peter Bull, Christine Chung, Bryn Dawson, Casey Fitzpatrick, Tamara Glazer, Dean Knox, Alex Liebscher, and Sebastian Marin. 2023. The candor corpus: Insights from a large multimodal dataset of naturalistic conversation. Science Advances, 9(13):eadf3197. find
  35. Steve Renals, Thomas Hain, and Hervé Bourlard. 2007. Recognition and understanding of meetings the ami and amida projects. In 2007 IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU), pages 238–247. IEEE. find
  36. James A Russell. 1980. A circumplex model of affect. Journal of personality and social psychology, 39(6):1161. find
  37. James A Russell and Lisa Feldman Barrett. 1999. Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant. Journal of personality and social psychology, 76(5):805. find
  38. Fiona Ryan, Ajay Bati, Sangmin Lee, Daniel Bolya, Judy Hoffman, and James M Rehg. 2025. Gaze-lle: Gaze target estimation via large-scale learned encoders. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 28874–28884. find
  39. Iulian Vlad Serban, Ryan Lowe, Peter Henderson, Laurent Charlin, and Joelle Pineau. 2018. A survey of available corpora for building data-driven dialogue systems: The journal version. Dialogue and Discourse, 9(1):1–49. Publisher Copyright: © 2018 Iulian Vlad Serban, Ryan Lowe, Peter Henderson, Laure… find
  40. Pranaydeep Singh, Luna De Bruyne, Orphée De Clercq, and Els Lefever. 2023. Misery loves complexity: Exploring linguistic complexity in the context of emotion detection. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12871–12880. Association for Computational Linguist… find
  41. Hyunjung Son. 2012. Construction of a Korean multimodal corpus. Language Facts and Perspectives, 30:55–79. find
  42. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Proceedings of the 2015 Conference of the North American Chapte… find our card
  43. Tanya Stivers, Nicholas J Enfield, Penelope Brown, Christina Englert, Makoto Hayashi, Trine Heinemann, Gertie Hoymann, Federico Rossano, Jan Peter De Ruiter, Kyung-Eun Yoon, et al. 2009. Universals and cultural variation in turn-taking in conversation. Proceedings of the National Academy of Sciences… find
  44. Ning Tan, Gaëlle Ferré, Marion Tellier, Edlira Cela, Mary-Annick Morel, Jean-Claude Martin, and Philippe Blache. 2010. Multi-level annotations of nonverbal behaviors in french spontaneous conversation. In International Conference for Language Resources and Evaluation, pages 74– 79. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-k-mind-korean-multimodal-interaction/