Elamur Research⌕
☰
home / papers / 2026 / Predicting States of Understanding in Explanatory Interactions Using Cognitive Load-Related Linguistic Cues

Predicting States of Understanding in Explanatory Interactions Using Cognitive Load-Related Linguistic Cues

Summary

The paper asks whether moment-by-moment signals of cognitive load can help a system tell how well a listener is following an explanation. The authors use 21 German board-game explanation dialogues from the MUNDEX corpus. In these dialogues, explainees labelled their own understanding afterwards by reviewing video of the interaction.

Three cue types are quantified:

  • the explainer's information value, measured as surprisal;
  • the syntactic complexity of the explainer's utterances;
  • variation in the explainee's gaze.

Statistical tests show that most of the derived measures differ across four understanding states. A classification experiment then compares two off-the-shelf classifiers with a fine-tuned German BERT-based multimodal model. All three beat chance at the four-way task, and adding the load-related cues to textual input improves results. The BERT-based model proves the most robust, although performance varies across labels.

Earlier work treated this as a binary understood/not-understood problem. This study moves to a finer, multi-class view, which is relevant for adaptive conversational agents and "Social XAI" systems that must monitor and react to a user's comprehension.

Key points

  • Data: 21 dialogues from MUNDEX (Türk et al., 2023), using MUNDEX annotations v0.7 on Zenodo. The dialogues yield 642 labelled segments:

    • Understanding
    • Partial Understanding
    • Non-Understanding
    • Misunderstanding (smallest class, 113 segments, 17.6%)

    The paper's text and its Table 1 disagree on which of Understanding and Non-Understanding is larger.

  • Labelling: labels come from retrospective video-recall self-annotation. Agreement for the earlier binary understanding annotation was Cohen's κ = 0.90.

  • Cue analysis: four cognitive-load values were derived from the three cue types, and three of them varied significantly with understanding state.

  • Tools: the reference list cites German GPT-2 (dbmdz/german-gpt2), a Universal Dependencies parser (Qi et al., 2018) and scikit-learn. These were likely used for surprisal, syntactic measures and the baseline classifiers; the methods section was not available to confirm this.

  • Results: four-class prediction is feasible, and the fine-tuned German BERT multimodal classifier is the most robust of the three. Exact scores were not available in the provided text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4tsmsshhd3ad

@inproceedings{wang2026predicting,
  title = {{Predicting States of Understanding in Explanatory Interactions Using Cognitive Load-Related Linguistic Cues}},
  author = {Wang, Yu and Türk, Olcay and Grimminger, Angela and Buschmeier, Hendrik},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4tsmsshhd3ad},
  url = {https://doi.org/10.63317/4tsmsshhd3ad}
}

References

56 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jens Allwood, Stefan Kopp, Karl Grammer, Elisabeth Ahlsén, Elisabeth Oberzaucher, and Markus Koppensteiner. 2007. The analysis of embodied communicative feedback in multimodal corpora: A prerequisite for behaviour simulation. Language Resources and Evaluation, 41:255–272. find
  2. Jens Allwood, Joakim Nivre, and Elisabeth Ahlsén. 1992. On the semantics and pragmatics of linguistic feedback. Journal of Semantics, 9:1–26. find
  3. Agnes Axelsson and Gabriel Skantze. 2022. Multimodal user feedback during adaptive robot-human presentations. Frontiers in Computer Science, 3:741148:22. find
  4. Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. 2018. OpenFace 2.0: Facial behavior analysis toolkit. In Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition, pages 59–66, Xi’an, China. find
  5. Janet Bavelas and Nicole Chovil. 2018. Some pragmatic functions of conversational facial gestures. Gesture, 17(1):98–127. find
  6. Simon Betz, Nataliya Bryhadyr, Olcay Türk, and Petra Wagner. 2023. Cognitive load increases spoken and gestural hesitation frequency. Languages, 8(1). find
  7. Mathias Rose Bjare, Stefan Lattner, and Gerhard Widmer. 2024. Controlling surprisal in music generation via information content curve matching. In Proceedings of the 25th International Society for Music Information Retrieval Conference, pages 922–929, San Francisco, CA, USA. ISMIR. find
  8. Hendrik Buschmeier and Stefan Kopp. 2018. Communicative listener feedback in human–agent interaction: Artificial speakers need to be attentive and adaptive. In Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems, pages 1213–1221, Stockholm, Sweden. IFAAMAS. find
  9. Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794, San Francisco, CA, USA. ACM. find
  10. Herbert H. Clark and Susan E. Brennan. 1991. Grounding in communication. In Lauren B. Resnick, John M. Levine, and Stephanie D. Teasley, editors, Perspectives on Socially Shared Cognition, pages 222–233. American Psychological Association, Washington, DC, USA. find
  11. Herbert H. Clark and Meredyth Krych. 2004. Speaking while monitoring addressees for understanding. Journal of Memory and Language, 50:62– 81. find
  12. Vera Demberg and Frank Keller. 2008. Data from eye-tracking corpora as evidence for theories of syntactic processing complexity. Cognition, 109(2):193–210. find
  13. Vera Demberg and Frank Keller. 2009. A computational model of prediction in human parsing: Unifying locality and surprisal effects. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 31. find
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  15. Stefan L. Frank. 2013. Uncertainty reduction as a measure of cognitive load in sentence comprehension. Topics in Cognitive Science, 5(3):475– 494. find
  16. Richard Futrell, Kyle Mahowald, and Edward Gibson. 2015. Large-scale evidence of dependency length minimization in 37 languages. Proceedings of the National Academy of Sciences, 112(33):10336–10341. find
  17. Edward Gibson. 1998. Linguistic complexity: Locality of syntactic dependencies. Cognition, 68(1):1– 76. find
  18. Edward Gibson. 2000. The dependency locality theory: A distance-based theory of linguistic complexity. In Image, Language, Brain: Papers from the First Mind Articulation Project Symposium, pages 95–126. The MIT Press. find
  19. Mario Giulianelli, Arabella Sinclair, and Raquel Fernández. 2021. Is information density uniform in task-oriented dialogues? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8271–8283, Online and Punta Cana, Dominican Republic. find
  20. Arthur M. Glenberg, Jennifer L. Schroeder, and David A. Robertson. 1998. Averting the gaze disengages the environment and facilitates remembering. Memory & Cognition, 26:651–658. find
  21. John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Proceedings of the 2nd Meeting of the North American Chapter of the Association for Computational Linguistics, pages 159–166, Pittsburgh, PA, USA. ACL. find our card
  22. Christine Howes and Arash Eshghi. 2021. Feedback relevance spaces: Interactional constraints on processing contexts in dynamic syntax. Journal of Logic, Language and Information, 30:331– 362. find
  23. Marcel A. Just and Patricia A. Carpenter. 1992. A capacity theory of comprehension: Individual differences in working memory. Psychological Review, 99(1):122–149. find
  24. Adam Kendon. 1967. Some functions of gaze-direction in social attention. Acta Psychologica, 26:22–63. find
  25. Shunichi Kinoshita, Toshiki Onishi, Naoki Azuma, Ryo Ishii, Atsushi Fukayama, Takao Nakamura, and Akihiro Miyata. 2023. A study of prediction of listener’s comprehension based on multimodal information. In Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents, pages 30:1… find
  26. Stefan Lazarov, Michael Schaffer, Viviane Gladow, Hendrik Buschmeier, Angela Grimminger, and Heike M. Buhl. 2025. Applications of video-recall for the assessment of understanding and knowledge in explanatory contexts. OSFPreprints. find
  27. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. find
  28. Dekang Lin. 1996. On the structural complexity of natural language sentences. In Proceedings of the 16th International Conference on Computational Linguistics, pages 729–733, Copenhagen, Denmark. find
  29. Haitao Liu. 2008. Dependency distance as a metric of language comprehension difficulty. Journal of Cognitive Science, 9(2):159–191. find
  30. Eliot Maës, Philippe Blache, and Leonor Becerra. 2022. Shared knowledge in natural conversations: Can entropy metrics shed light on information transfers? In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 213–227, Abu Dhabi, UAE. ACL. find
  31. Dimitry. Mindlin, Amelie S. Robrecht, Michael Morasch, and Philipp Cimiano. 2024. Measuring user understanding in dialogue-based xAI systems. In ECAI 2024: 27th European Conference on Artificial Intelligence, pages 1148–1155, Santiago de Compostela, Spain. IOS Press. find
  32. Benjamin Minixhofer, Thomas Winkler, Martin Schmitt, Hinrich Schütze, and Markus Leuck. 2019. German BERT models (bert-base-german-dbmdz-uncased). Hugging Face. find
  33. Jeff Mitchell, Mirella Lapata, Vera Demberg, and Frank Keller. 2010. Syntactic and semantic factors in processing difficulty: An integrated measure. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 196–206, Uppsala, Sweden. ACL. find
  34. Louis-Philippe Morency, C. Mario Christoudias, and Trevor Darrell. 2006. Recognizing gaze aversion gestures in embodied conversational discourse. In Proceedings of the 8th International Conference on Multimodal Interfaces, page 287–294, Banff, Canada. find
  35. Fred Paas, Alexander Renkl, and John Sweller. 2003. Cognitive load theory and instructional design: Recent developments. Educational Psychologist, 38(1):1–4. find
  36. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. scikit-learn:… find
  37. Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher D. Manning. 2018. Universal Dependency parsing from scratch. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 160–170, Brussels, Belgium. find
  38. Dennis Reidsma, Iwan de Kok, Daniel Neiberg, Sathish Pammi, Bart van Straalen, Khiet Truong, and Herwin van Welbergen. 2011. Continuous interaction with a virtual human. Journal on Multimodal User Interfaces, 4:97–118. find
  39. Amelie Robrecht and Stefan Kopp. 2023. SNAPE: A sequential non-stationary decision process model for adaptive explanation generation. In Proceedings of the 15th International Conference on Agents and Artificial Intelligence, pages 48– 58, Lisbon, Portugal. SciTePress. find
  40. Katharina Rohlfing, Philipp Cimiano, Ingrid Scharlau, Tobias Matzner, Heike Buhl, Hendrik Buschmeier, Angela Grimminger, Barbara Hammer, Reinhold Häb-Umbach, Ilona Horwath, Eyke Hüllermeier, Friederike Kern, Stefan Kopp, Kirsten Thommes, Axel-Cyrille Ngonga Ngomo, Carsten Schulte, Henning Wachsmuth,… find
  41. Katharina Rohlfing, Kary Främling, Brian Lim, Kirsten Thommes, and Suzana Alpsancar, editors. 2026. Social Explainable AI. Communications of NII Shonan Meetings. Springer, Singapore. find
  42. Stefan Schweter. 2020. German GPT-2 model (dbmdz/german-gpt2). Hugging Face. find
  43. Cory Shain. 2021. CDRNN: Discovering complex dynamics in human language processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3718–3734, Online. ACL. find
  44. Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):302–319. find
  45. John Sweller. 1988. Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2):257–285. find
  46. Benedikt M. Szmrecsányi. 2004. On operationalizing syntactic complexity. In Le poids des mots. Proceedings of the 7th International Conference on Textual Data Statistical Analysis, volume 2, pages 1031–1038, Louvain-la-Neuve, Belgium. find
  47. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? Probing for sentence structure in contextualized word representations. In International Conference on Lear… find our card
  48. Eleftheria Tsipidi, Franz Nowak, Ryan Cotterell, Ethan Wilcox, Mario Giulianelli, and Alex Warstadt. 2024. Surprise! Uniform Information Density isn’t the whole story: Predicting surprisal contours in long-form discourse. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language… find
  49. Olcay Türk, Stefan Lazarov, Yu Wang, Hendrik Buschmeier, Angela Grimminger, and Petra Wagner. 2024. Predictability of understanding in explanatory interactions based on multimodal cues. In Proceedings of the 26th ACM International Conference on Multimodal Interaction, pages 449–458, San José, Costa… find
  50. Yu Wang and Hendrik Buschmeier. 2023. Does listener gaze in face-to-face interaction follow the entropy rate constancy principle: An empirical study. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15372– 15379, Singapore. ACL. find
  51. Yu Wang and Hendrik Buschmeier. 2024. Revisiting the phenomenon of syntactic complexity convergence on German dialogue data. In Proceedings of the 20th Conference on Natural Language Processing, pages 75–80, Vienna, Austria. ACL. find
  52. Yu Wang, Yang Xu, Gabriel Skantze, and Hendrik Buschmeier. 2024. How much does non-verbal communication conform to entropy rate constancy?: A case study on listener gaze in interaction. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3533– 3545, Bangkok, Thailand. ACL. find
  53. Ethan G. Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P. Levy. 2023. Testing the predictions of surprisal theory in 11 languages. Transactions of the Association for Computational Linguistics, 11:1451–1470. find our card
  54. Yang Xu and David Reitter. 2018. Information density converges in dialogue: Towards an information-theoretic model. Cognition, 170:147– 163. 11. find
  55. Language Resource References Hendrik Buschmeier, Angela Grimminger, Petra Wagner, Stefan Lazarov, Olcay Türk, and Yu Wang. 2025. MUNDEX annotations (version 0.7). Zenodo. find
  56. Olcay Türk, Petra Wagner, Hendrik Buschmeier, Angela Grimminger, Yu Wang, and Stefan Lazarov. 2023. MUNDEX: A multimodal corpus for the study of the understanding of explanations. In Proceedings of the 1st International Multimodal Communication Symposium, pages 63– 64, Barcelona, Spain. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-predicting-states-understanding-explanatory-interactions/