Elamur Research⌕
☰
home / papers / 2024 / Mhm... Yeah? Okay! Evaluating the Naturalness and Communicative Function of Synthesized Feedback Responses in Spoken Dialogue

Mhm... Yeah? Okay! Evaluating the Naturalness and Communicative Function of Synthesized Feedback Responses in Spoken Dialogue

Paper · 2024 · Carol Figueroa, Marcel de Korte, Magalie Ochs et al.

Summary

Conversational agents that listen like humans need short feedback responses such as "mhm", "yeah" or "wow". These tokens carry their meaning through both their words and their prosody. The authors test whether existing synthesis methods can reproduce such responses so that they sound natural, fit their context, and keep their intended communicative function. They do not try to predict prosody. Instead, they assume prosody is known and transplant it from listeners in U.S. English Switchboard telephone calls onto one female voice talent, using two routes:

  • Frame-level signal processing: TD-PSOLA, WORLD and the Praat Vocal Toolkit.
  • Phone-level TTS: a fine-tuned FastPitch 1.1 with a HiFiGAN vocoder.

In online listening tests, 86 raters judged original, TTS, signal-processed and pitch-flattened versions in dialogue context. They gave naturalness and appropriateness scores and chose a function label for each response. TTS sounded more natural than signal processing, and the two were rated about equally appropriate. However, TTS preserved the original function worse, even worse than the monotone control. The result suggests that future prosody predictors for feedback should work at the frame level.

Key points

  • Switchboard listener feedback (47 responses in total) was re-synthesized and placed back into 12 conversation clips. Only the listener channel was replaced.
  • The TTS voice was pretrained on LJ-Speech and fine-tuned on a mix of target-voice recordings:
    • read CMU ARCTIC sentences (43 min);
    • Taskmaster-2 role-play (4 min);
    • 724 imitated Switchboard feedback tokens (11 min);
    • 34 min of free conversation containing 981 feedback instances.
  • Mean naturalness scores (1–5) were Monotone 1.73, SignalP 2.61, TTS 3.0 and Original 4.37. All pairwise differences were significant in a CLMM analysis with Bonferroni-corrected post-hoc tests.
  • Raters labeled each response with one of 10 feedback functions from Figueroa et al. (2022), such as Continue, Agree, Sympathy, Mild Surprise and Strong Surprise.
  • Agreement between each rater's label for the original and for the synthesized version (Cohen's kappa) was 0.79 for SignalP, 0.74 for Monotone and 0.71 for TTS. TTS mainly lost attitudinal functions such as Sympathy and the two Surprise levels.

Interesting ideas

  • I1. Naturalness, contextual fit and preserved function are measured separately, which shows that output can sound fine and fit the context while still meaning something else.
  • I2. A deliberately degraded condition (pitch flattened) serves as a floor, and the TTS scoring below it exposed that TTS carried almost no extra functional signal.
  • I3. In the within-participant design, each rater's reading of the original response serves as that rater's own gold label, which absorbs individual variation in how functions are perceived.
  • I4. The authors note that two labels can both fit the context yet be different functions, such as Mild versus Strong Surprise, so appropriateness does not equal correctness.
  • I5. A fine-grained, ten-way taxonomy of short feedback responses (non-understanding, continue, agree, disagree, yes, no, sympathy, disapproval, two levels of surprise) covers functions carried by very short responses.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): Evaluate the text → graph → text compressor on whether regenerated text keeps the same speech act and stance, not only on fluency.
  • I2 (learn): Add stripped-information baselines (e.g., lemmas only, no tree) as negative controls, so any component that fails to beat them is flagged.
  • I3 (learn): When LLM teachers or witness parsers label pragmatic phenomena, compare each witness against its own judgment on the original input, so systematic per-witness bias is separated from real loss.
  • I4 (frames): Let prag keep graded or neighboring speech-act labels and score near-misses differently from outright errors.
  • I5 (frames): Use it as a speech-act inventory for interjections and short replies in dialogue texts, for example in tales, inferring the function from context and punctuation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓KTH Royal Institute of TechnologyStockholm · Sweden20 works · 1,723 citations🎓Aix-Marseille UniversitéMarseille · France25 works · 39,241 citations🔬Laboratoire d’Informatique et SystèmesMarseille · France10 works · 187 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 worksDialogue systems · 654 works

Cite

Cite as Carol Figueroa, Marcel de Korte, Magalie Ochs and Gabriel Skantze. 2024. Mhm... Yeah? Okay! Evaluating the Naturalness and Communicative Function of Synthesized Feedback Responses in Spoken Dialogue. Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.sigdial-1.46

@inproceedings{figueroa2024mhm,
  title = {{Mhm... Yeah? Okay! Evaluating the Naturalness and Communicative Function of Synthesized Feedback Responses in Spoken Dialogue}},
  author = {Carol Figueroa, Marcel {de Korte}, Magalie Ochs and Gabriel Skantze},
  booktitle = {Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2024},
  doi = {10.18653/v1/2024.sigdial-1.46},
  url = {https://doi.org/10.18653/v1/2024.sigdial-1.46}
}

References

22 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2002. Qualtrics. https://www.qualtrics.com/.2014. Prolific. https://www.prolific.com/. ↗
  2. Tuan Dinh, Alexander Kain, and Kris Tjaden. 2019. Using a manifold vocoder for spectral voice and style conversion. In Proc. Interspeech 2019, pages 1388– 1392. find
  3. Carol Figueroa, Adaeze Adigwe, Magalie Ochs, and Gabriel Skantze. 2022. Annotation of communicative functions of short feedback tokens in Switchboard. In Proc. LREC, pages 1849–1859. find
  4. John J Godfrey, Edward C Holliman, and Jane Mc-Daniel. 1992. SWITCHBOARD: Telephone speech corpus for research and development. In IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 517–520. IEEE Computer Society. find
  5. Catharine Oertel, Joakim Gustafson, and Alan W Black. 2016. On data driven parametric backchannel synthesis for expressing attentiveness in conversational agents. In Proceedings of the Workshop on Multimodal Analyses Enabling Artificial Agents in Human-Machine Interaction, pages 43–47. find
  6. Sathish Pammi, Marc Schröder, Marcela Charfuelan, Oytun Türk, and Ingmar Steiner. 2010. Synthesis of listener vocalisations with imposed intonation contours. In Proc. 7th ISCA Workshop on Speech Synthesis (SSW 7), pages 240–245. find
  7. Charles Goodwin. 1986. Between and within: Alternative sequential treatments of continuers and assessments. Human Studies, 9(2-3):205–217. ↗
  8. R Core Team. 2023. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. find
  9. Keith Ito and Linda Johnson. 2017. The LJ speech dataset. https://keithito.com/ LJ-Speech-Dataset/. ↗
  10. Emanuel A Schegloff. 1982. Discourse as an interactional achievement: Some uses of ‘uh huh’ and other things that come between sentences. Analyzing Discourse: Text and Talk, 71:71–93. find
  11. John Kominek and Alan W Black. 2004. The CMU Arctic speech databases. In Fifth ISCA Workshop on Speech Synthesis. find
  12. Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022– 17033.
  13. Divesh Lala, Koji Inoue, Tatsuya Kawahara, and Kei Sawada. 2022. Backchannel generation model for a third party listener agent. In Proceedings of the 10th International Conference on Human-Agent Interaction, pages 114–122. find
  14. Adrian Łańcucki. 2021. Fastpitch: Parallel text-to-speech with pitch prediction. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6588–6592. IEEE. ↗
  15. Thorsten Stocksmeier, Stefan Kopp, and Dafydd Gibbon. 2007. Synthesis of prosodic attitudinal variants in German backchannel ja. In Proc. Interspeech 2007, pages 1290–1293. find
  16. Jinhan Wang, Long Chen, Aparna Khare, Anirudh Raju, Pranav Dheram, Di He, Minhua Wu, Andreas Stolcke, and Venkatesh Ravichandran. 2024. Turn-taking and backchannel prediction with acoustic and large language model fusion. arXiv preprint arXiv:2401.14717. ↗
  17. Victor H Yngve. 1970. On getting a word in edgewise. In Papers from the 6th Regional Meeting Chicago Linguistic Society, pages 567–578. find
  18. Åsa Wallers. 2006. Minor sounds of major importance-prosodic manipulation of synthetic backchannels in Swedish. find
  19. Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017. Montreal Forced Aligner: Trainable text-speech alignment using Kaldi. In Proc. Interspeech 2017, pages 498–502. find
  20. Kentaro Mitsui, Yukiya Hono, and Kei Sawada. 2023. Towards human-like spoken dialogue generation between ai agents from written dialogue. Preprint, arXiv:2310.01088. ↗
  21. Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. 2016. WORLD: a vocoder-based high-quality speech synthesis system for real-time applications. IE-ICE TRANSACTIONS on Information and Systems, 99(7):1877–1884. ↗
  22. Anindita Nath and Nigel Ward. 2022. On the predictability of the prosody of dialog markers from the prosody of the local context. In Proc. Speech Prosody, pages 664–668. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/figueroa-2024-mhm-yeah-okay-evaluating-naturalness/