Elamur Research⌕
☰
home / papers / 2026 / Investigating the Representation of Backchannels and Fillers in Fine-tuned Language Models

Investigating the Representation of Backchannels and Fillers in Fine-tuned Language Models

Paper · 2026 · Yu Wang, Leyi Lao, Langchu Huang et al.

Summary

Transformer language models are mostly trained on written text. NLP pipelines also tend to delete backchannels (e.g., uh-huh, yeah) and fillers (e.g., uh, um) as noise. As a result, models know little about these items, even though they shape turn-taking, grounding and the feel of natural conversation. The authors ask whether fine-tuning can teach models useful representations of these items, and which models and training objectives benefit most. They use three two-party dialogue corpora in which backchannels and fillers are kept and annotated: two English ones, Switchboard and MapTask, plus a Japanese one, the BTSJ corpus (full title: 1000 Person Japanese Natural Conversation Corpus). They fine-tune BERT, GPT-2, LLaMA-3 8B and Qwen-3 8B with three objectives: masking, next token prediction and turn-taking prediction. Clustering the learnt representations yields higher silhouette scores after fine-tuning, meaning the models separate different uses of these expressions more clearly. The fine-tuned models also produce dialogue closer to human transcripts. This points toward building conversational models that talk more like people.

Key points

  • Data: The data cover English and Japanese, about 150,000 utterances per language. Switchboard and MapTask together roughly match the size of the BTSJ corpus. The backchannel and filler inventories follow Todd (2019) and Pilan et al. (2024).
  • Fine-tuning tasks:
    • MASK: selectively masks backchannel and filler spans.
    • NTP: next token prediction.
    • TTP: turn-taking prediction.
  • Representation analysis: Representations are analysed with unsupervised clustering, mainly on the last hidden layer, and evaluated with silhouette scores. Other layers appear only in an appendix.
  • Generation results: Fine-tuned models generate backchannels and fillers more often and with more variety. They also show lower perplexity on these tokens and higher BLEU and BERTScore against ground-truth dialogue. Side effects on general language understanding are minor.
  • Availability and limitations: Code and data are released on GitHub (colalao/discourse_markers) and Zenodo. German corpora (Verbmobil, MUNDEX) were excluded, and no large-scale human evaluation was carried out.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Bielefeld UniversityBielefeld · Germany35 works · 1,196 citations🎓KTH Royal Institute of TechnologyStockholm · Sweden20 works · 1,723 citations🎓Southern University of Science and TechnologyShenzhen · China22 works · 1,337 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Yu Wang, Leyi Lao, Langchu Huang, Gabriel Skantze, Yang Xu and Hendrik Buschmeier. 2026. Investigating the Representation of Backchannels and Fillers in Fine-tuned Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.241

@inproceedings{wang2026investigating,
  title = {{Investigating the Representation of Backchannels and Fillers in Fine-tuned Language Models}},
  author = {Yu Wang, Leyi Lao, Langchu Huang, Gabriel Skantze, Yang Xu and Hendrik Buschmeier},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.241},
  url = {https://doi.org/10.18653/v1/2026.acl-long.241}
}

References

65 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hervé Abdi and Lynne J. Williams. 2010. Principal component analysis. Wiley Interdisciplinary Reviews: Computational Statistics, 2(4):433–459. find
  2. AI@Meta. 2024. LLaMA 3 Model Card. find
  3. Ahmed Amer, Chirag Bhuvaneshwara, Gowtham K. Addluri, Mohammed M. Shaik, Vedant Bonde, and Philipp Müller. 2023. Backchannel detection and agreement estimation from video with transformer networks. In 2023 International Joint Conference on Neural Networks (IJCNN), pages 1–8. find
  4. Anne H. Anderson, Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jacqueline Kowtko, Jan McAllister, Jim Miller, and 1 others. 1991. The HCRC map task corpus. Language and Speech, 34(4):351–366. find
  5. Peter Ball. 1975. Listeners’ responses to filled pauses in relation to floor apportionment. British Journal of Social & Clinical Psychology, 14(4):423–424. find
  6. Claire Augusta Bergey and Simon DeDeo. 2024. From “um” to “yeah”: Producing, predicting, and regulating information flow in human conversation. Preprint, arXiv:2403.08890. — [arXiv:2403.08890](https://arxiv.org/abs/2403.08890) find
  7. André Berthold and Anthony Jameson. 1999. Interpreting symptoms of cognitive load in speech input. In Proceedings of the 7th International Conference on User Modeling, pages 235–244, Banff, Canada. Springer. find
  8. Hendrik Buschmeier and Stefan Kopp. 2018. Communicative listener feedback in human–agent interaction: Artificial speakers need to be attentive and adaptive. In Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems, pages 1213–1221, Stockholm, Sweden. IFAAMAS. find
  9. Eugene Charniak and Mark Johnson. 2001. Edit detection and parsing for transcribed speech. In Proceedings of the 2nd Meeting of the North American Chapter of the Association for Computational Linguistics, Pittsburgh, PA, USA.
  10. Grzegorz Chrupała. 2023. Putting natural in natural language processing. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7820–7827, Toronto, Canada. ACL. find
  11. Herbert H. Clark. 1996. Using Language. Cambridge University Press, Cambridge, UK. find
  12. Herbert H. Clark and Jean E. Fox Tree. 2002. Using uh and um in spontaneous speaking. Cognition, 84(1):73–111. find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  14. Mark Dingemanse and Andreas Liesenfeld. 2022. From text to talk: Harnessing conversational corpora for humane and diversity-aware language technology. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 5614– 5633, Dublin, Ireland. ACL. find
  15. Jens Edlund, Joakim Gustafson, Mattias Heldner, and Anna Hjalmarsson. 2008. Towards human-like spoken dialogue systems. Speech Communication, 50:630–645. find
  16. Bradley Efron. 1992. Bootstrap methods: Another look at the jackknife. In Samuel Kotz and Norman L. Johnson, editors, Breakthroughs in Statistics: Methodology and Distribution, pages 569–593. Springer, New York, NY, USA. find
  17. Erik Ekstedt and Gabriel Skantze. 2020. TurnGPT: a transformer-based language model for predicting turn-taking in spoken dialog. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2981–2990, Online. ACL. find
  18. Carol Figueroa, Adaeze Adigwe, Magalie Ochs, and Gabriel Skantze. 2022. Annotation of communicative functions of short feedback tokens in Switchboard. In Proceedings of the 13th Language Resources and Evaluation Conference, pages 1849–1859, Marseille, France. ELRA. find
  19. John Rupert Firth. 1957. A synopsis of linguistic theory 1930–1955. In Philological Society (Great Britain), editor, Studies in Linguistic Analysis, pages 1–32. Blackwell, Oxford, UK. find
  20. Jean E. Fox Tree. 2010. Discourse markers across speakers and settings. Language and Linguistics Compass, 4(5):269–281. find
  21. Janet M. Fuller. 2003. The influence of speaker roles on discourse marker use. Journal of Pragmatics, 35(1):23–45. find
  22. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. ACL.
  23. John J. Godfrey and Edward Holliman. 1993. Switchboard-1 Release 2 LDC97S62. Linguistic Data Consortium. find
  24. Christine Howes and Arash Eshghi. 2021. Feedback relevance spaces: Interactional constraints on processing contexts in dynamic syntax. Journal of Logic, Language and Information, 30:331–362. find
  25. Tanvi Dinkar, Chloé Clavel, and Ioana Vasilescu. 2022. Fillers in spoken language understanding: Computational and psycholinguistic perspectives. Traitement Automatique des Langues, 63(3):37–62. find
  26. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460.
  27. Kaja Dobrovoljc and Matej Martinc. 2018. Er . . . well, it matters, right? On the role of data representations in spoken language dependency parsing. In Proceedings of the 2nd Workshop on Universal Dependencies, pages 37–46, Brussels, Belgium. ACL. find
  28. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  29. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651–3657, Florence, Italy. ACL.
  30. Fredrik Jørgensen. 2007. The effects of disfluency detection in parsing spoken language. In Proceedings of the 16th Nordic Conference of Computational Linguistics, pages 240–244, Tartu, Estonia. find
  31. Andreas H. Jucker and Sara W. Smith. 1998. “And people just you know like ‘wow’”. Discourse markers as negotiating strategies. In Andreas H. Jucker and Yael Ziv, editors, Discourse Markers: Description and Theory, pages 171–201. Benjamins, Amsterdam, The Netherlands. find
  32. Masahito Kawamori, Akira Shimazu, and Takeshi Kawabata. 1996. A phonological study on Japanese discourse markers. In Proceedings of the Korean Society for Language and Information Conference, pages 297– 306. find
  33. Martin Kay, Peter Norvig, and Mark Gawron. 1992. Verbmobil: A Translation System for Face-to-Face Dialog. University of Chicago Press, Chicago, IL, USA. find
  34. Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. 2023. Surgical fine-tuning improves adaptation to distribution shifts. In The Eleventh International Conference on Learning Representations. find
  35. Andreas Liesenfeld and Mark Dingemanse. 2022. Building and curating conversational corpora for diversity-aware language science and technology. In Proceedings of the 13th Language Resources and Evaluation Conference, pages 1178–1192, Marseille, France. ELRA. find
  36. Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. 2020. What happens to BERT embeddings during fine-tuning? In Proceedings of the 3rd BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 33–44, Online. ACL.
  37. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, volume 26. Curran.
  38. Marius Mosbach, Anna Khokhlova, Michael A. Hedderich, and Dietrich Klakow. 2020. On the interplay between fine-tuning and sentence-level probing for linguistic knowledge in pre-trained transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2502–2516, Online. AC… find
  39. Bill Noble and Vladislav Maraev. 2021. Large-scale text pre-training helps with dialogue act recognition, but not without fine-tuning. In Proceedings of the 14th International Conference on Computational Semantics, pages 166–172, Groningen, The Netherlands (online). ACL.
  40. Toshiki Onishi, Naoki Azuma, Shunichi Kinoshita, Ryo Ishii, Atsushi Fukayama, Takao Nakamura, and Akihiro Miyata. 2023. Prediction of various backchannel utterances based on multimodal information. In Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents, pages 47:1–4, W… find
  41. Volha Petukhova and Harry Bunt. 2009. Towards a multidimensional semantics of discourse markers in spoken dialogue. In Proceedings of the 8th International Conference on Computational Semantics, pages 157–168, Tilburg, The Netherlands. ACL.
  42. Ildiko Pilan, Laurent Prévot, Hendrik Buschmeier, and Pierre Lison. 2024. Conversational feedback in scripted versus spontaneous dialogues: A comparative analysis. In Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 440–457, Kyoto, Japan. ACL. find
  43. Livia Qian and Gabriel Skantze. 2024. Joint learning of context and feedback embeddings in spoken dialogue. In Proceedings of Interspeech 2024, pages 2955–2959, Kos, Greece. ISCA. find
  44. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 28492–2…
  45. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI. find
  46. Ralph L. Rose. 2015. Um and uh as differential delay markers: The role of contextual factors. In Proceedings of the 7th Workshop on Disfluency in Spontaneous Speech, pages 73–76, Edinburgh, UK. find
  47. Peter J. Rousseeuw. 1987. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Computational and Applied Mathematics, 20:53–65. find
  48. Robin Ruede, Markus Müller, Sebastian Stüker, and Alex Waibel. 2017. Enhancing backchannel prediction using word embeddings. In Proceedings of Interspeech 2017, pages 879–883, Stockholm, Sweden. ISCA. find
  49. Serhad Sarica and Jianxi Luo. 2021. Stopwords in technical language processing. PLOS ONE, 16(8):1– 13. find
  50. Kei Sawada, Tianyu Zhao, Makoto Shing, Kentaro Mitsui, Akio Kaga, Yukiya Hono, Toshiaki Wakatsuki, and Koh Mitsuda. 2024. Release of pre-trained models for the Japanese language. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation… find
  51. Lawrence Schourup. 1999. Discourse markers. Lingua, 107(3):227–265. find
  52. Gabriel Skantze. 2017. Towards a general, continuous model of turn-taking in spoken dialogue using LSTM recurrent neural networks. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 220–230, Saarbrücken, Germany. ACL. find
  53. Gabriel Skantze. 2021. Turn-taking in conversational systems and human-robot interaction: a review. Computer Speech & Language, 67:101178. find
  54. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26:339–373.
  55. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, and 1 others. 2025. Qwen3 Technical Report. Preprint, arXiv:2505.09388.
  56. Tianyu Zhao and Kei Sawada. 2021. rinna/japanese-gpt2-medium. Huggingface. find
  57. Zheng Zhao, Yftah Ziser, and Shay B Cohen. 2024. Layer by layer: Uncovering where multi-task learning happens in instruction-tuned large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15195–15214, Miami, FL, USA. ACL. find
  58. Yichu Zhou and Vivek Srikumar. 2021. DirectProbe: Studying representations without classifiers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5070–5083, Online. ACL. find
  59. James Allen Todd. 2019. “It has the ability to make the other person feel comfortable”: L1 Japanese speakers’ folk descriptions of aizuchi. Lingua, 230:102737. find
  60. Olcay Türk, Petra Wagner, Hendrik Buschmeier, Angela Grimminger, Yu Wang, and Stefan Lazarov. 2023. MUNDEX: A multimodal corpus for the study of the understanding of explanations. In Book of Abstracts of the 1st International Multimodal Communication Symposium. find
  61. Mayumi Usami, editor. 2023. Building of a Japanese 1000 Person Natural Conversation Corpus for Pragmatic Analyses and Its Multilateral Studies, and NIN-JAL Institute-Based Projects: Multiple Approaches to Analyzing the Communication of Japanese Language Learners. NINJAL. find
  62. Chris van der Lee, Albert Gatt, Emiel van Miltenburg, and Emiel Krahmer. 2021. Human evaluation of automatically generated text: Current trends and best practice guidelines. Computer Speech & Language, 67:101151. find
  63. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research, 86:2579–2605. find
  64. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353– 355…
  65. Siyang Wang, Joakim Gustafson, and Éva Székely. 2022. Evaluating sampling-based filler insertion with spontaneous TTS. In Proceedings of the 13th Language Resources and Evaluation Conference, pages 1960– 1969, Marseille, France. ELRA. *Table 1: The top 15 selected English backchannels/fillers and th… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-investigating-representation-backchannels-fillers-fine/