Elamur Research⌕
☰
home / papers / 2023 / Dialogue Act-Aided Backchannel Prediction Using Multi-Task Learning

Dialogue Act-Aided Backchannel Prediction Using Multi-Task Learning

Paper · 2023 · Wencke Liermann, Yo-Han Park, Yong-Seok Choi et al.

Summary

Listeners keep a conversation flowing with short feedback such as "uh-huh" or "really?", known as backchannels. This paper asks whether knowing what kind of utterance the speaker just made helps predict when a listener will give such feedback, and what kind. The authors built BCDA, a model that combines audio and text. The audio part is an encoder pre-trained with self-supervised Voice Activity Projection (VAP). The text part runs BERT over recent speech segments and uses Time-Aware Attention. During training, a second output head learns dialogue act (DA) classification alongside the main backchannel task. DA labels are only needed for training, not at inference.

On the Switchboard Corpus, BCDA reaches 66.0 weighted F1, about 3 points above earlier baselines. The DA auxiliary task mainly helps on content-dependent "Assessment" backchannels. VAP pre-training adds about 1.4 weighted F1. In casual talk, sentiment carries little signal, so utterance type is a more useful auxiliary cue than sentiment.

Key points

  • Data and labels: Switchboard yields about 122k instances, split into three classes: NO-BC (50%), Continuer (22.5%) and Assessment (27.5%).
    • DA tags come from the Switchboard DA annotation, which covers 48.2% of conversations and uses 42 tags.
    • The authors aligned these tags to the newer transcript with a Levenshtein-based method. This gave labels for 41.9% of utterances that precede an instance.
  • Results:
    • BCDA_MT scores 66.0 weighted F1, compared with 63.0 for the BPM baseline (Jang et al.) and 58.4 for Ortega et al.
    • Adding the DA task raises Assessment F1 by 2.0 and NO-BC by 0.3. It has no significant effect on Continuer.
  • Pre-training: VAP pre-training lifts an audio-only model from 61.36 to 62.71 weighted F1, even with plenty of labeled data available.
  • Modalities: Audio alone beats text alone by more than 3.3 weighted F1. However, text is better at Assessment and audio is better at Continuer, so combining them helps.
  • Setup and other findings:
    • The loss weight for the auxiliary task is λ = 0.3.
    • The model classifies dialogue acts with 75.12% accuracy and separates statement-opinion (sv) from statement-non-opinion (sd) well.
    • 72.6% of backchannels in the data follow neutral utterances.
    • Code is published at github.com/wencke-lm/BCDA.

Interesting ideas

  • I1. Use an auxiliary head whose loss is only computed when a label exists, so a partially annotated corpus can still shape the shared representation without needing labels at test time.
  • I2. Recover annotations made on an older version of a transcript by aligning it to the corrected version with edit distance, accepting imperfect but useful coverage.
  • I3. Attention over earlier segments is scaled down by a function that decreases with temporal distance, so recent context counts more while older context can still fill in omitted material like pronoun referents.
  • I4. To prevent leakage, fine-tuning and evaluation use only the speaker's audio, so the model cannot simply detect the listener's actual backchannel when timestamps are slightly off.
  • I5. Separating disputable opinions (sv) from non-disputable narrative statements (sd) predicts the kind of response a listener gives, such as agreement or encouragement.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver data with uneven coverage, such as frames on some sentences only, could train extra heads on a shared encoder for the tagger or gparse.
  • I2 (tree): The same trick could move UD or other annotations between treebank releases or differing tokenizations during dialect conversion.
  • I3 (frames): Coreference candidate scoring could include an explicit, inspectable distance-decay term rather than relying on learned weights alone.
  • I4 (learn): Evaluations should systematically strip inputs that could expose the gold answer, as a built-in negative control against shortcut learning.
  • I5 (frames): The speech-act layer in prag could tag claims as opinion versus description to anticipate expected replies or flag statements open to dispute in QA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Chungnam National UniversityDaejeon · South Korea6 works · 14 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 654 worksSpeech act theory · 229 works

Cite

Cite as Wencke Liermann, Yo-Han Park, Yong-Seok Choi and Kong Lee. 2023. Dialogue Act-Aided Backchannel Prediction Using Multi-Task Learning. Findings of the Association for Computational Linguistics: EMNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.1006

@inproceedings{liermann2023dialogue,
  title = {{Dialogue Act-Aided Backchannel Prediction Using Multi-Task Learning}},
  author = {Wencke Liermann, Yo-Han Park, Yong-Seok Choi and Kong Lee},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.1006},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.1006}
}

References

20 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amalia Istiqlali Adiba, Takeshi Homma, and Toshinori Miyoshi. 2021. Towards immediate backchannel generation using attention-based early prediction model. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7408–7412. IEEE. find
  2. Mukesh Barange, Sandratra Rasendrasoa, Maël Bouab-delli, Julien Saunier, and Alexandre Pauchet. 2022. Impact of adaptive multimodal empathic behavior on the user interaction. In Proceedings of the 22nd ACM International Conference on Intelligent Virtual Agents, pages 1–8. Association for Computing Machinery. find
  3. Janet B Bavelas, Linda Coates, and Trudy Johnson. 2000. Listeners as co-narrators. Journal of Personality and Social Psychology, 79(6):941–952. ↗
  4. Neeraj Deshmukh, Aravind Ganapathiraju, Andi Glee-son, Jonathan Hamaker, and Joseph Picone. 1998. Resegmentation of SWITCHBOARD. In 5th International Conference on Spoken Language Processing (ICSLP 1998). ISCA. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  6. Erik Ekstedt and Gabriel Skantze. 2022. Voice activity projection: Self-supervised learning of turn-taking events. In Interspeech 2022, pages 5190–5194. ISCA. find
  7. John Godfrey and Edward Holliman. 1993. Switchboard-1 Release 2 LDC97S62. Linguistic Data Consortium. prediction of turn-changing and turn-management willingness improve backchannel modeling? In Proceedings of the 21st ACM International Conference on Intelligent Virtual Agents, pages 131–138. Association for Computing Machinery. find
  8. Shoichi Iwasaki. 1997. The northridge earthquake conversations: The floor structure and the ‘loop’ sequence in japanese conversation. Journal of Pragmatics, 28(6):661–693. ↗
  9. Jin Yea Jang, San Kim, Minyoung Jung, Saim Shin, and Gahgene Gweon. 2021. BPM_MT: Enhanced backchannel prediction model using multi-task learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3447–3452, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  10. Dan Jurafsky, Elizabeth Shriberg, and Debra Bi-asca. 1997. Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual. Technical Report Draft 13, University of Colorado, Institute of Cognitive Science. find
  11. Ganeshan Malhotra, Abdul Waheed, Aseem Srivastava, Md Shad Akhtar, and Tanmoy Chakraborty. 2022. Speaker and time-aware joint contextual learning for dialogue-act classification in counselling conversations. In Proceedings of the 15th ACM International Conference on Web Search and Data Mining, pages 735–745. Association for Computing Machinery.
  12. Akira Morikawa, Ryo Ishii, Hajime Noto, Atsushi Fukayama, and Takao Nakamura. 2022. Determining most suitable listener backchannel type for speaker’s utterance. In Proceedings of the 22nd ACM International Conference on Intelligent Virtual Agents, pages 1–3. Association for Computing Machinery. find
  13. OpenAI. 2023. Gpt-4 technical report. Computing Research Repository, arXiv:2303.08774. Version 3. ↗
  14. Kohei Hara, Koji Inoue, Katsuya Takanashi, and Tatsuya Kawahara. 2018. Prediction of turn-taking using multitask learning with prediction of backchannels and fillers. In Interspeech 2018, pages 991–995. ISCA. find
  15. Daniel Ortega, Chia-Yu Li, and Ngoc Thang Vu. 2020. OH, JEEZ! or UH-HUH? a listener-aware backchannel predictor on ASR transcriptions. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8064– 8068. IEEE. find
  16. Zihao He, Leili Tavabi, Kristina Lerman, and Mohammad Soleymani. 2021. Speaker turn modeling for dialogue act classification. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2150–2157, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  17. Morgane Riviere, Armand Joulin, Pierre-Emmanuel Mazaré, and Emmanuel Dupoux. 2020. Unsupervised pretraining transfers well across languages. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7414–7418. IEEE. find
  18. Lixing Huang, Louis-Philippe Morency, and Jonathan Gratch. 2011. Virtual rapport 2.0. In Intelligent Virtual Agents, pages 68–79. Springer Berlin Heidelberg. find
  19. Robin Ruede, Markus Müller, Sebastian Stüker, and Alex Waibel. 2017. Enhancing Backchannel Prediction Using Word Embeddings. In Interspeech 2017, pages 879–883. ISCA. ↗
  20. Ryo Ishii, Xutong Ren, Michal Muszynski, and Louis-Philippe Morency. 2021. Multimodal and multitask approach to listener’s backchannel prediction: Can find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liermann-2023-dialogue-act-aided-backchannel-prediction/