Elamur Research⌕
☰
home / papers / 2024 / Which Prosodic Features Matter Most for Pragmatics?

Which Prosodic Features Matter Most for Pragmatics?

Paper · 2024 · Nigel G. Ward, Divette Marco and Olac Fuentes

Summary

The paper asks which kinds of prosodic features carry the most pragmatic information in spoken dialog, where pragmatics covers things like enthusiasm, topic shifts, turn yielding and criticism. Instead of studying one narrow task, the authors use a general proxy: predicting how pragmatically similar humans judge two utterances to be. The data come from an existing collection of pragmatic-similarity judgments. It has 458 American English pairs and 235 Northern Mexican Spanish pairs, each made of a dialog utterance and a re-enactment of it. Each utterance is described by 10 Midlevel Toolkit-derived features averaged over 10 fixed-percentage windows, which gives 100 features. These feed several regressors, and feature importance is then measured three ways. Duration-related cues, especially speaking rate and lengthening, turn out to matter far more than pitch, and the start of an utterance matters more than its end. Failure analysis points to overlooked cues such as nasality and vibrato. The findings matter for speech-synthesis evaluation, speech-to-speech translation, codecs and basic prosody research, where pitch has usually dominated attention.

Key points

  • Model correlations with human judgments were: Euclidean Distance –0.33, Linear Regression 0.44, KNN (k=50) 0.58, Random Forest (100 trees) 0.70, and cosine over 103 selected HuBert layer-24 features 0.74.
  • In Random Forest importance, speaking rate scores 43.7% and lengthening 20.9%. The four pitch features together reach only 15.2%, and a pitch-only model correlates at just 0.16.
  • Used alone, speaking rate reaches 0.64 correlation. CPPS and intensity show near-zero solo correlation yet still have some importance, which suggests that features act in combinations rather than independently.
  • On lexically identical pairs the Random Forest reaches 0.80, matching pretrained features. On lexically different pairs it performs near chance.
  • A survey of Interspeech 2023 synthesis papers found that most focus on pitch and duration, only 10% mention intensity, and just one mentions voicing.

Interesting ideas

  • I1. The authors judge feature usefulness against a broad, task-agnostic target (human pragmatic similarity of pairs) instead of any single classification task, so the conclusions generalize better.
  • I2. Utterances are tiled into fixed-percentage windows that are finer near the edges, which revealed that the beginning of an utterance matters more than the end.
  • I3. Importance is measured three ways: solo correlation, model impurity importance, and subset/ablation. The authors treat disagreement among these as evidence that features interact.
  • I4. Signed seed-minus-copy deltas outperformed absolute differences, because the direction of change carries information.
  • I5. Splitting results by lexically identical versus lexically different pairs exposed a model that only works in one regime.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): A pair-similarity set for speech-act and implicature readings could rank which prag frame features truly drive pragmatic interpretation, without having to commit to one downstream task.
  • I2 (frames): A position-aware representation could be tested for speech-act cues to check whether sentence-initial material deserves extra weight in prag.
  • I3 (learn): Evaluations of repair rules or parser features could combine solo-use and leave-one-out runs, flagging items that look useless alone but hurt when removed.
  • I4 (back): When scoring the text → graph → text compressor, signed differences between source and regenerated text features could reveal systematic drift, such as hedges being consistently dropped.
  • I5 (answer): Test results could be stratified by surface overlap with training material, as a negative control showing whether a reader understands the meaning or just matches wording.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at El PasoEl Paso · United States4 works · 17 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 works

🧭 Topics 1Research subjects and languages the work is about.Speech & spoken language · 519 works

Cite

Cite as Nigel G. Ward, Divette Marco and Olac Fuentes. 2024. Which Prosodic Features Matter Most for Pragmatics?. arXiv:2408.13240.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2408.13240 · arXiv 2408.13240

@misc{ward2024which,
  title = {{Which Prosodic Features Matter Most for Pragmatics?}},
  author = {Nigel G. Ward, Divette Marco and Olac Fuentes},
  year = {2024},
  doi = {10.48550/arxiv.2408.13240},
  eprint = {2408.13240},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2408.13240}
}

References

34 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Matthew Marge, Carol Espy-Wilson, Nigel G. Ward, et al., “Spoken language interaction with robots: Research issues and recommendations,” Computer Speech and Language, vol. 71, 2022. find
  2. Dagmar Barth-Weingarten, Nicole Dehé, and Anne Wichmann, Where Prosody Meets Pragmatics, Brill, 2009. find
  3. Harm Lameris, Joakim Gustafsson, and Éva Székely, “Beyond style: Synthesizing speech with pragmatic functions,” in Interspeech, 2023, pp. 3382–3386. find
  4. Weiqin Li, Peiji Yang, et al., “Spontaneous style text-to-speech synthesis with controllable spontaneous behaviors based on language models,” in Interspeech, 2024. find
  5. Guan-Ting Lin, Chi-Luen Feng, et al., “On the utility of self-supervised models for prosody-related tasks,” in IEEE Workshop on Spoken Language Technology (SLT), 2022, pp. 1104– 1111. find
  6. Dan Andrei Iliescu, Devang S Ram Mohan, Tian Huey Teh, and Zack Hodari, “Controllable prosody generation with partial inputs,” in IEEE ICASSP, 2024, pp. 11916–11920. find
  7. Petra Wagner, Jonas Beskow, et al., “Speech synthesis evaluation: State-of-the-art assessment and suggestion for a novel research program,” in Proceedings of the 10th Speech Synthesis Workshop (SSW10), 2019. find
  8. Wen-Chin Huang, Benjamin Peloquin, et al., “A Holistic Cascade System, Benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translation,” in ICASSP, 2023. find
  9. Yayue Deng, Jinlong Xue, et al., “ConCSS: Contrastive-based context comprehension for dialogue-appropriate prosody in conversational speech synthesis,” in IEEE ICASSP, 2024, pp. 10706–10710. find
  10. Kevin Heffernan, Artyom Kozhevnikov, et al., “Aligning speech segments beyond pure semantics,” in Findings of the Association for Computational Linguistics, 2024, pp. 3626– 3635. find
  11. Loı̈c Barrault, Yu-An Chung, et al., “Seamless: Multilingual expressive and streaming speech translation,” arXiv preprint arXiv:2312.05187, 2023. ↗
  12. Eliya Nachmani, Alon Levkovitch, Yifan Ding, Chulayuth Asawaroengchai, Heiga Zen, and Michelle Tadmor Ramanovich, “Translatotron 3: Speech to speech translation with monolingual data,” in IEEE ICASSP, 2024, pp. 10686–10690. find
  13. Leyuan Qu, Taihao Li, et al., “Disentangling prosody representations with unsupervised speech reconstruction,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023. ↗
  14. Anton Batliner, Stefan Steidl, Bjorn Schuller, et al., “Whodunnit: Searching for the most important feature types signalling emotion-related user states in speech,” Computer Speech and Language, vol. 25, pp. 4–28, 2011. ↗
  15. Florian Eyben, Klaus R. Scherer, et al., “The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing,” IEEE Transactions on Affective Computing, vol. 7, pp. 190–202, 2016. ↗
  16. Bogdan Vlasenko, Sargam Vyas, et al., “Comparing data-driven and handcrafted features for dimensional emotion recognition,” in IEEE ICASSP, 2024, pp. 11841–11845. find
  17. Neville Ryant, Malcolm Slaney, Mark Liberman, Elizabeth Shriberg, and Jiahong Yuan, “Highly accurate Mandarin tone classification in the absence of pitch information,” in Proceedings of Speech Prosody, 2014. find
  18. Anton Batliner, Jan Buckow, Richard Huber, Volker Warnke, Elmar Nöth, and Heinrich Niemann, “Prosodic feature evaluation: Brute force or well designed,” in Proc. 14th Int. Congress of Phonetic Sciences, 1999, vol. 3, pp. 2315–2318. find
  19. Anton Batliner, Jan Buckow, Richard Huber, Volker Warnke, Elmar Nöth, and Heinrich Niemann, “Boiling down prosody for the classification of boundaries and accents in German and English,” in Eurospeech, 2001, pp. 2781–2784. find
  20. Eduardo Coutinho, Florian Hönig, et al., “Assessing the prosody of non-native speakers of English: Measures and feature sets,” in Conference on Language Resources and Evaluation (LREC 2016), 2016, pp. 1328–1332.
  21. Gabriel Skantze, “Turn-taking in conversational systems and human-robot interaction: A review,” Computer Speech & Language, vol. 67, pp. 101178, 2021. ↗
  22. Nigel G. Ward, Alejandro Vega, and Timo Baumann, “Prosodic and temporal features for language modeling for dialog,” Speech Communication, vol. 54, pp. 161–174, 2011. ↗
  23. Luciana Ferrer, Nicolas Scheffer, and Elizabeth Shriberg, “A comparison of approaches for modeling prosodic features in speaker recognition,” in IEEE ICASSP, 2010, pp. 4414–4417. find
  24. Raymond W. M. Ng, Tan Lee, Cheung-Chi Leung, Bin Ma, and Haizhou Li, “Analysis and selection of prosodic features for language identification,” in IEEE International Conf. on Asian Language Processing, 2009, pp. 123–128. find
  25. Ethan Weed and Riccardo Fusaroli, “Acoustic measures of prosody in right-hemisphere damage: A systematic review and meta-analysis,” Journal of Speech, Language, and Hearing Research, vol. 63, no. 6, pp. 1762–1775, 2020. ↗
  26. Nigel G. Ward, “Midlevel prosodic features toolkit (2016-2023),” https://github.com/nigelgward/midlevel, 2023. ↗
  27. Dik J. Hermes, “Auditory and visual similarity of pitch contours,” Journal of Speech, Language, and Hearing Research, vol. 41, pp. 63–72, 1998. ↗
  28. Uwe D. Reichel, Felicitas Kleber, and Raphael Winkelmann, “Modelling similarity perception of intonation,” in Interspeech, 2009, pp. 1711–1714. find
  29. Olivier Nocaudie and Corine Astésano, “Evaluating prosodic similarity as a means towards L2 teacher’s prosodic control training,” Speech Prosody 2016, pp. 26–30, 2016. find
  30. Nigel G. Ward and Divette Marco, “A collection of pragmatic-similarity judgments over spoken dialog utterances,” in Linguistic Resources and Evaluation Conference, 2024. find
  31. Nigel G. Ward, Jonathan E. Avila, Emilia Rivas, and Divette Marco, “Dialogs re-enacted across languages, version 2,” Tech. Rep. UTEP-CS-23-27, University of Texas at El Paso, Department of Computer Science, 2023. find
  32. Jonathan E. Avila and Nigel G. Ward, “Towards cross-language prosody transfer for dialog,” in Interspeech, 2023. find
  33. Nigel G. Ward, Prosodic Patterns in English Conversation, Cambridge University Press, 2019. find
  34. Nigel G. Ward, Andres Segura, Alejandro Ceballos, and Divette Marco, “Towards a general-purpose model of perceived pragmatic similarity,” in Interspeech, 2024. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ward-2024-which-prosodic-features-matter-most/