Elamur Research⌕
☰
home / papers / 2026 / Aligning Backchannel and Dialogue Context Representations via Contrastive LLM Fine-Tuning

Aligning Backchannel and Dialogue Context Representations via Contrastive LLM Fine-Tuning

Paper · 2026 · Livia Qian and Gabriel Skantze

Summary

Backchannels such as 'mhm', 'yeah' or 'wow' are brief listener responses. Their meaning depends on both the word chosen and how it is said. Most computational work predicts when they occur, not which form fits a given moment. Qian and Skantze build a two-stage system for that second problem.

First, they adapt open-weight LLMs to Fisher telephone-call transcripts. The hidden state just before a backchannel then serves as a summary of the conversation so far. Second, they combine this text vector with WavLM features from the final second of context speech. A contrastive InfoNCE objective places it in a shared space with WavLM encodings of backchannel audio.

For evaluation, they draw on human perception data, using three-way similarity comparisons that cover both prosody and differences across lexical forms, along with a task rating how well a backchannel suits its context. Compared with earlier approaches, retrieval improves considerably, and longer context helps. The learned backchannel embeddings also match human judgments better than raw WavLM features. This gives spoken dialogue systems a lightweight, interpretable way to choose backchannels and interpret them.

Key points

  • Data: Fisher Part 1 (5,850 US English phone calls) is used for training and evaluation. FiCa, a single-speaker collection of backchannels, is used for comparison. The study covers 18 lexical forms, leaving out 'no' and clarification-type tokens.
  • Stage 1 models: Gemma 3 4B, LLaMA 3.1 8B, Qwen2.5 7B and Mistral 7B were fine-tuned with QLoRA on 83,047 transcript snippets, split evenly into training and test sets.
  • Context length matters: Perplexity on backchannels drops as more preceding turns are given (1, 3, then 5). For LLaMA 3.1 8B it falls from 10.19 to 8.19. Pre-trained models without fine-tuning score far worse, roughly 31–55.
  • Interpretability: Linear probes recover three affective dimensions from the embedding space: Energy, Surprisal and Polarity.
  • Setup and resources: WavLM stays frozen throughout. Each LLM takes 1–2 days to fine-tune on 8 RTX 3090 GPUs. Code, perception-study results and links to model weights are released on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓KTH Royal Institute of TechnologyStockholm · Sweden18 works · 788 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksDialogue systems · 624 works

Cite

Cite as Livia Qian and Gabriel Skantze. 2026. Aligning Backchannel and Dialogue Context Representations via Contrastive LLM Fine-Tuning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.629

@inproceedings{qian2026aligning,
  title = {{Aligning Backchannel and Dialogue Context Representations via Contrastive LLM Fine-Tuning}},
  author = {Livia Qian and Gabriel Skantze},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.629},
  url = {https://doi.org/10.18653/v1/2026.acl-long.629}
}

References

75 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jens Allwood, Joakim Nivre, and Elisabeth Ahlsén. 1992. On the semantics and pragmatics of linguistic feedback. Journal of Semantics, 9(1):1–26. find
  2. Siddhant Arora, Kai-Wei Chang, Chung-Ming Chien, Yifan Peng, Haibin Wu, Yossi Adi, Emmanuel Dupoux, Hung-Yi Lee, Karen Livescu, and Shinji Watanabe. 2025. On the landscape of spoken language models: A comprehensive survey. arXiv preprint arXiv:2504.08528. — [arXiv:2504.08528](https://arxiv.org/abs/2504.08528) find
  3. Agnes Axelsson, Hendrik Buschmeier, and Gabriel Skantze. 2022. Modeling feedback in interaction with conversational agents: A review. Frontiers in Computer Science, 4:744574. find
  4. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework https://github.com/qianlivia/Aligning-Backc hannel-and-Dialogue-Context-Representations for self-supervised learning of speech representations. Advances in Neural Information Processing Systems 33 (Neur… ↗
  5. Štefan Beňuš. 2016. The prosody of backchannels in Slovak. In Proceedings of the 8th International Conference on Speech Prosody, pages 75–79. find
  6. Štefan Beňuš, Agustín Gravano, and Julia Bell Hirschberg. 2007. The prosody of backchannels in American English. Proceedings of the 16th International Congress of Phonetic Sciences (ICPhS 2007), pages 1065–1068. find
  7. Roxane Bertrand, Gaëlle Ferré, Philippe Blache, Robert Espesser, and Stéphane Rauzy. 2007. Backchannels revisited from a multimodal perspective. In Auditory-Visual Speech Processing, pages 1–5. find
  8. Peter Blomsma, Julija Vaitonyté, Gabriel Skantze, and Marc Swerts. 2024. Backchannel behavior is idiosyncratic. Language and Cognition, 16(4):1158–1181. find
  9. Auriane Boudin, Roxane Bertrand, Stéphane Rauzy, Magalie Ochs, and Philippe Blache. 2021. A multimodal model for predicting conversational feedbacks. In Text, Speech, and Dialogue, pages 537–549. Springer International Publishing. find
  10. Carol Figueroa. 2024. FiCa speech dataset. Accessed: 2026-01-02. find
  11. Julio Cesar Cavalcanti and Gabriel Skantze. 2025. “Dyadosyncrasy”, idiosyncrasy and demographic factors in turn-taking. arXiv preprint arXiv:2505.24736. — [arXiv:2505.24736](https://arxiv.org/abs/2505.24736) find
  12. Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. 2022. WavLM: Large-scale self-supervised pre-training for full stack… find
  13. Christopher Cieri, David Miller, and Kevin Walker. 2004. The Fisher corpus: A resource for the next generations of speech-to-text. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), volume 4, pages 69–71.
  14. Herbert H. Clark and Edward F. Schaefer. 1989. Contributing to discourse. Cognitive Science, 13(2):259– 294. find
  15. Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: A speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037.
  16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, volume 1, pages 4171–4186.
  17. Erik Ekstedt and Gabriel Skantze. 2020. TurnGPT: A transformer-based language model for predicting turn-taking in spoken dialog. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2981–2990. find
  18. Gaëlle Ferré and Suzanne Renaudier. 2017. Unimodal and bimodal backchannels in conversational English. In Proceedings of the 21st Workshop on the Semantics and Pragmatics of Dialogue (SemDial 2017), pages 27–37. find
  19. Carol Figueroa, Adaeze Adigwe, Magalie Ochs, and Gabriel Skantze. 2022. Annotation of communicative functions of short feedback tokens in Switchboard. In Proceedings of the Thirteenth Language Resources and Evaluation Conference (LREC), pages 1849–1859. find
  20. Carol Figueroa, Marcel de Korte, Magalie Ochs, and Gabriel Skantze. 2024. Mhm... Yeah? Okay! evaluating the naturalness and communicative function of synthesized feedback responses in spoken dialogue. In Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue (…
  21. Yoshinori Fukunaga, Ryota Nishimura, Kengo Ohta, and Norihide Kitaoka. 2025. Backchannel prediction for natural spoken dialog systems using general speaker and listener information. In Proceedings of Interspeech 2025, pages 1078–1082. find
  22. Riccardo Fusaroli, Kristian Tylén, Katrine Garly, Jakob Steensig, Morten H. Christiansen, and Mark Dingemanse. 2017. Measures and mechanisms of common ground: Backchannels, conversational repair, and interactive alignment in free and task-oriented social interactions. In Proceedings of the Annual Me… find
  23. Pino Cutrone. 2005. A case study examining backchannels in conversations between Japanese-British dyads. Multilingua, Journal of Cross-Cultural and Interlanguage Communication, 24(3):237–274. find
  24. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  25. Pino Cutrone. 2014. A cross-cultural examination of the backchannel behavior of Japanese and Americans: Considerations for Japanese EFL learners. Intercultural Pragmatics, 11(1):83–120. find
  26. J.J. Godfrey, E.C. Holliman, and J. McDaniel. 1992. SWITCHBOARD: Telephone speech corpus for research and development. In Proceedings of ICASSP, volume 1, pages 517–520. find
  27. Google Deepmind. 2025. Gemma 3 4B. Accessed: 2026-01-02. find
  28. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and 1 others. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  29. Agustín Gravano and Julia Hirschberg. 2011. Turn-taking cues in task-oriented dialogue. Computer Speech & Language, 25(3):601–634. find
  30. Hanae Koiso, Yasuo Horiuchi, Syun Tutiya, Akira Ichikawa, and Yasuharu Den. 1998. An analysis of turn-taking and backchannels based on prosodic and syntactic features in Japanese map task dialogs. Language and Speech, 41(3-4):295–321. find
  31. Michelle Kraaz and Tobias Bernaisch. 2022. Backchannels and the pragmatics of South Asian Englishes. World Englishes, 41(2):224–243. find
  32. Bettina Heinz. 2003. Backchannel responses as strategic responses in bilingual speakers’ conversations. Journal of Pragmatics, 35(7):1113–1142. find
  33. Peter Kuppens, Francis Tuerlinckx, James A. Russell, and Lisa Feldman Barrett. 2013. The relation between valence and arousal in subjective experience. Psychological Bulletin, 139(4):917. find
  34. Mattias Heldner, Jens Edlund, and Julia Hirschberg. 2010. Pitch similarity in the vicinity of backchannels. In Proceedings of Interspeech 2010, pages 3054–3057. find
  35. Kanghee Lee. 2020. Backchannels as a cooperative strategy in ELF communications. Korean Journal of English Language and Linguistics, 20:257–281. find
  36. Mattias Heldner, Anna Hjalmarsson, and Jens Edlund. 2013. Backchannel relevance spaces. In Nordic Prosody XI, pages 137–146. find
  37. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.
  38. Christine Howes and Arash Eshghi. 2021. Feedback relevance spaces: Interactional constraints on processing contexts in dynamic syntax. Journal of Logic, Language and Information, 30(2):331–362. find
  39. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460.
  40. Koji Inoue, Divesh Lala, Gabriel Skantze, and Tatsuya Kawahara. 2025. Yeah, Un, Oh: Continuous and real-time backchannel prediction with fine-tuning of voice activity projection. In Proceedings of NAACL (Volume 1: Long Papers), pages 7171–7181. find
  41. Ryo Ishii, Xutong Ren, Michal Muszynski, and Louis-Philippe Morency. 2021. Multimodal and multitask approach to listener’s backchannel prediction: Can prediction of turn-changing and turn-management willingness improve backchannel modeling? In IVA ’21: Proceedings of the 21st ACM International Confe… find
  42. Wencke Liermann, Yo-Han Park, Yong-Seok Choi, and Kong Lee. 2023. Dialogue act-aided backchannel prediction using multi-task learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15073–15079.
  43. Andreas Liesenfeld and Mark Dingemanse. 2022. Bottom-up discovery of structure and variation in response tokens (‘backchannels’) across diverse languages. In Proceedings of Interspeech 2022, pages 1126–1130. find
  44. Guan-Ting Lin, Jiachen Lian, Tingle Li, Qirui Wang, Gopala Anumanchipalli, Alexander H. Liu, and Hung-yi Lee. 2025a. Full-Duplex-Bench: A benchmark to evaluate full-duplex spoken dialogue models on turn-taking capabilities. arXiv preprint arXiv:2503.04721. — [arXiv:2503.04721](https://arxiv.org/abs/2503.04721) find
  45. Yuxin Lin, Yinglin Zheng, Ming Zeng, and Wangzheng Shi. 2025b. Predicting turn-taking and backchannel in human-machine conversations using linguistic, acoustic, and visual signals. In Proceedings of ACL (Volume 1: Long Papers), pages 15310–15322. find
  46. Jin Yea Jang, San Kim, Minyoung Jung, Saim Shin, and Gahgene Gweon. 2021. BPM_MT: Enhanced backchannel prediction model using multi-task learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3447–3452. find
  47. Long Mai and Julie Carson-Berndsen. 2025. Real-time textless dialogue generation. arXiv preprint arXiv:2501.04877. — [arXiv:2501.04877](https://arxiv.org/abs/2501.04877) find
  48. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, and 1 others. 2023. Mistral 7B. Preprint, arXiv:2310.06825. find
  49. Mistral AI. 2024. Mistral 7B v0.3. Accessed: 2026-01-02. find
  50. Dan Jurafsky, Elizabeth Shriberg, Barbara Fox, and Traci Curl. 1998. Lexical, prosodic, and syntactic cues for dialog acts. In Discourse Relations and Discourse Markers.
  51. Meta AI. 2024. LLaMA 3.1 8B. Accessed: 2026-01-02. find
  52. Ali Mollahosseini, Behzad Hasani, and Mohammad H. Mahoor. 2019. AffectNet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, 10(1):18– 31. find
  53. Philipp Müller, Michael Dietz, Dominik Schiller, Do-minike Thomas, Hali Lindsay, Patrick Gebhard, Elisabeth André, and Andreas Bulling. 2022. MultiMe-diate’22: Backchannel detection and agreement estimation in group interactions. In Proceedings of the 30th ACM International Conference on Multimedia,… find
  54. Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoît Sagot, Abdelrahman Mohamed, and Emmanuel Dupoux. 2023. Generative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics, 11:250–266.
  55. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
  56. Daniel Ortega, Chia-Yu Li, and Ngoc Thang Vu. 2020. Oh, Jeez! or uh-huh? a listener-aware backchannel predictor on ASR transcriptions. In Proceedings of ICASSP, pages 8064–8068. find
  57. Yo-Han Park, Wencke Liermann, Yong-Seok Choi, and Kong Joo Lee. 2024. Improving backchannel prediction leveraging sequential and attentive context awareness. In Findings of the Association for Computational Linguistics: EACL 2024, pages 1689–1694. find
  58. Ronald Poppe, Khiet P. Truong, and Dirk Heylen. 2013. Perceptual evaluation of backchannel strategies for artificial listeners. Autonomous Agents and Multi-Agent Systems, 27(2):235–253. find
  59. Ronald Poppe, Khiet P. Truong, Dennis Reidsma, and Dirk Heylen. 2010. Backchannel strategies for artificial listeners. In International Conference on Intelligent Virtual Agents, pages 146–158. Springer. find
  60. Livia Qian, Carol Figueroa, and Gabriel Skantze. 2025. Representation of perceived prosodic similarity of conversational feedback. In Proceedings of Interspeech 2025, pages 374–378. find
  61. Livia Qian and Gabriel Skantze. 2024. Joint learning of context and feedback embeddings in spoken dialogue. In Proceedings of Interspeech 2024, pages 2955–2959. find
  62. Qwen. 2024. Qwen2.5 7B. Accessed: 2026-01-02. find
  63. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th Internatio…
  64. Robin Ruede, Markus Müller, Sebastian Stüker, and Alex Waibel. 2019. Yeah, right, uh-huh: A deep learning backchannel predictor. In Advanced Social Interaction with Agents: 8th International Workshop on Spoken Dialog Systems, pages 247–258. Springer. find
  65. Leah Shelley and Fernanado Gonzalez. 2013. Back channeling: Function of back channeling and L1 effects on back channeling in L2. Linguistic Portfolios, 2(1):9. find
  66. Khiet Phuong Truong, Ronald Walter Poppe, I.A. de Kok, and Dirk K.J. Heylen. 2011. A multimodal analysis of vocal and visual backchannels in spontaneous dialogs. In Proceedings of Interspeech 2011, pages 2973–2976. find
  67. Jinhan Wang, Long Chen, Aparna Khare, Anirudh Raju, Pranav Dheram, Di He, Minhua Wu, Andreas Stolcke, and Venkatesh Ravichandran. 2024. Turn-taking and backchannel prediction with acoustic and large language model fusion. In Proceedings of ICASSP, pages 12121–12125. find
  68. Nigel G. Ward, Divette Marco, and Olac Fuentes. 2025. Which prosodic features matter most for pragmatics? In Proceedings of ICASSP, pages 1–5.
  69. Amy Beth Warriner, Victor Kuperman, and Marc Brysbaert. 2013. Norms of valence, arousal, and dominance for 13,915 English lemmas. Behavior Research Methods, 45(4):1191–1207. find
  70. Deanna Wong and Pam Peters. 2007. A study of backchannels in regional varieties of English, using corpus mark-up as the means of identification. International Journal of Corpus Linguistics, 12(4):479– 510. find
  71. An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, and 1 others. 2025. Qwen2.5-1M technical report. arXiv preprint arXiv:2501.15383.
  72. Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, and 1 others. 2021. SUPERB: Speech Processing Universal PERformance Benchmark. Proceedings of Interspeech 2021, pages 1194–1198. find
  73. Victor H. Yngve. 1970. On getting a word in edgewise. In CLS-70, pages 567–577. University of Chicago. find
  74. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DialoGPT: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Lingu…
  75. Robin Ruede, Markus Müller, Sebastian Stüker, and Alex Waibel. 2017. Enhancing backchannel prediction using word embeddings. In Proceedings of Interspeech 2017, pages 879–883. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qian-2026-aligning-backchannel-dialogue-context-representations/